Trendoday

AltRA-Test: An Alternative Annotator Test for LLM Judges Without Shared Annotators

Large language models (LLMs) are increasingly used as judges and graders to replace human annotators, raising the question of when such replacement is statistically justified. To support such decision

  • Upvotes10
#11Rank now –
#11Peak rank
3Days on list
359Score · 7d
Nov 20, 2026First seen
Rank · 7 days

About this paper

A statistical test, AltRA-Test, determines when large language models can replace human annotators by comparing inter-annotator agreement in resampled mixed human-LLM panels, and evaluations show it often requires about 100 items and makes better replacement decisions than baselines.

Large language models (LLMs) are increasingly used as judges and graders to replace human annotators, raising the question of when such replacement is statistically justified. To support such decisions, the Alternative Annotator Test (Alt-Test) was introduced for dense settings where all items are annotated by the same annotators. However, in many large-scale annotation settings, items are annotated by different subsets of annotators. We therefore propose the Alternative Annotator Test with Resampled Annotations (AltRA-Test), a statistical test that does not require that all items are annotated by the same annotators. AltRA-Test uses a bootstrap that resamples items, draws k human…

large language models (LLMs)LLM judgesLLM gradersprompting techniquesAlternative Annotator Test (Alt-Test)Alternative Annotator Test with Resampled Annotations (AltRA-Test)bootstrap resamplinginter-annotator agreement (IAA)asymptotic level alphaconsistent test
Upvotes
10
Authors
6
Published
Nov 19, 2026

Summary and abstract excerpt via Hugging Face Papers ↗ · arXiv.

Trending history

AltRA-Test: An Alternative Annotator Test for LLM Judges Without Shared Annotators was first seen on Hugging Face · Daily papers on Nov 20, 2026. Since then it has appeared in 27 checks on 3 different days. Its best position was #11 of 30, reached on Nov 21, 13:29 UTC. It is #11 right now, holding steady at #11 over the last 6 checks.

Rank history

Position on Hugging Face · Daily papers at every check (lower is better).

#1#11#20#30 Nov 20Nov 21Nov 21Nov 22Nov 22 Nov 20, 11:29 UTC · #17Nov 20, 13:29 UTC · #14Nov 20, 15:29 UTC · #13Nov 20, 17:29 UTC · #13Nov 20, 19:29 UTC · #13Nov 20, 21:29 UTC · #13Nov 20, 23:29 UTC · #13Nov 21, 01:29 UTC · #13Nov 21, 03:29 UTC · #13Nov 21, 05:29 UTC · #12Nov 21, 07:29 UTC · #12Nov 21, 09:29 UTC · #12Nov 21, 11:29 UTC · #12Nov 21, 13:29 UTC · #11Nov 21, 15:29 UTC · #11Nov 21, 17:29 UTC · #11Nov 21, 19:29 UTC · #11Nov 21, 21:29 UTC · #11Nov 21, 23:29 UTC · #11Nov 22, 01:29 UTC · #11Nov 22, 03:29 UTC · #11Nov 22, 05:29 UTC · #11Nov 22, 07:29 UTC · #11Nov 22, 09:29 UTC · #11Nov 22, 11:29 UTC · #11Nov 22, 13:29 UTC · #11Nov 22, 15:29 UTC · #11

Last 14 days

Best rank per day (UTC).

Nov 9today

Metric history

▲ as reported by Hugging Face.

081523 Nov 20Nov 21Nov 21Nov 22Nov 22 Nov 20, 11:29 UTC · 10Nov 20, 13:29 UTC · 14Nov 20, 15:29 UTC · 14Nov 20, 17:29 UTC · 18Nov 20, 19:29 UTC · 19Nov 20, 21:29 UTC · 20Nov 20, 23:29 UTC · 20Nov 21, 01:29 UTC · 20Nov 21, 03:29 UTC · 20Nov 21, 05:29 UTC · 21Nov 21, 07:29 UTC · 21Nov 21, 09:29 UTC · 22Nov 21, 11:29 UTC · 22Nov 21, 13:29 UTC · 23Nov 21, 15:29 UTC · 23Nov 21, 17:29 UTC · 23Nov 21, 19:29 UTC · 23Nov 21, 21:29 UTC · 23Nov 21, 23:29 UTC · 23Nov 22, 01:29 UTC · 23Nov 22, 03:29 UTC · 23Nov 22, 05:29 UTC · 23Nov 22, 07:29 UTC · 23Nov 22, 09:29 UTC · 23Nov 22, 11:29 UTC · 23Nov 22, 13:29 UTC · 23Nov 22, 15:29 UTC · 23

Questions

Is AltRA-Test: An Alternative Annotator Test for LLM Judges Without Shared Annotators trending on HF Daily Papers right now?

Yes. As of Nov 22, 15:29 UTC it is #11, holding steady at #11 over the last 6 checks.

What is the highest rank AltRA-Test: An Alternative Annotator Test for LLM Judges Without Shared Annotators reached?

#11 on Hugging Face · Daily papers, on Nov 21, 13:29 UTC. Trendoday checks this list every 120 minutes.

How long has AltRA-Test: An Alternative Annotator Test for LLM Judges Without Shared Annotators been trending?

It was first seen on Nov 20, 2026 and has been on the list on 3 days (27 checks) since.