A statistical test, AltRA-Test, determines when large language models can replace human annotators by comparing inter-annotator agreement in resampled mixed human-LLM panels, and evaluations show it often requires about 100 items and makes better replacement decisions than baselines.
Large language models (LLMs) are increasingly used as judges and graders to replace human annotators, raising the question of when such replacement is statistically justified. To support such decisions, the Alternative Annotator Test (Alt-Test) was introduced for dense settings where all items are annotated by the same annotators. However, in many large-scale annotation settings, items are annotated by different subsets of annotators. We therefore propose the Alternative Annotator Test with Resampled Annotations (AltRA-Test), a statistical test that does not require that all items are annotated by the same annotators. AltRA-Test uses a bootstrap that resamples items, draws k human…
large language models (LLMs)LLM judgesLLM gradersprompting techniquesAlternative Annotator Test (Alt-Test)Alternative Annotator Test with Resampled Annotations (AltRA-Test)bootstrap resamplinginter-annotator agreement (IAA)asymptotic level alphaconsistent test
- Upvotes
- 10
- Authors
- 6
- Published
- Nov 19, 2026
Summary and abstract excerpt via Hugging Face Papers ↗ · arXiv.