Open NLI-based zero-shot classifiers (the transformers zero-shot-classification pipeline: the label with the
highest entailment score wins), all evaluated with the same script, templates and labels. Hypotheses and labels are in
English for every language, except the two "native" columns, where the label set and template were translated into the text's
language. Accuracy unless noted. Click a column to sort; the best value per column is highlighted.
Disclosure: this page is maintained by Horizon Labs, which makes the three multilingual-zeroshot models.
§ = label names that occur in Horizon Labs' synthetic training data (generic topic, emotion and sentiment taxonomies; for
CLINC150, 31 of 150 intent names), so those columns are not zero-shot for our models; "intent macro" averages MASSIVE, MTOP,
Banking77 and CLINC150. We do not know which label sets other models saw in training. † XNLI is included for reference only:
models trained on XNLI (xlm-roberta-large-xnli, mDeBERTa-xnli) score very high on it. "Commercial data" says whether a model's
training data allows commercial use (several strong models are MIT-licensed but trained partly on non-commercial data).
English-only models are included for comparison on the English columns. "–" = not evaluated. Eval code:
code/zeroshot/evaluate_zs.py, build_evals.py, build_evals2.py in
the model repo. Want a model added or
found a problem? Open a discussion on this Space.