Zero-Shot Intent Classification Benchmark

Choosing an open-weight language model for a zero-shot classification task is normally settled by one accuracy number on one benchmark. That number does not say whether the gap between the top two models is real, whether the model's confidence is trustworthy, whether performance survives a typo, or whether the benchmark still has headroom to measure with.

The practical question — which model should I actually deploy? — needs evidence that a leaderboard does not supply. Selecting a 7B model when a 3B instruction-tuned model performs comparably is a recurring and expensive default.

• 41 open-weight models, spanning 15 model families and the 135M–9B parameter range.
• Eight English single-label intent-classification datasets, covering standard academic benchmarks, a large-scale voice-assistant corpus, and production-derived e-commerce datasets.
• Evaluation is zero-shot throughout — no fine-tuning on the target datasets.
• Measured beyond exact-match accuracy: confidence calibration, robustness to realistic input perturbations, statistical reliability of rankings, deployment efficiency, and benchmark saturation.

Three findings from the paper, each scoped to this benchmark suite:

1. Parameter count is a poor selection criterion. Instruction-tuned 3B models outperform several of the evaluated 7B base models.
2. Leading models on MASSIVE are statistically indistinguishable. Under pairwise McNemar tests the differences between the top models do not reach significance — ranking them by raw accuracy overstates the gap.
3. SNIPS has saturated. It no longer meaningfully discriminates among current open-weight models, which limits what a result reported on it alone can support.

Scoped to English, single-label intent classification, and to models in the 135M–9B range under a fixed zero-shot prompting protocol. Conclusions about benchmark saturation are statements about these datasets and this model pool, not about intent classification in general.

Extending the protocol to tasks where label ambiguity is intrinsic, and treating benchmark saturation as a quantity to monitor rather than something discovered retrospectively.