How much of zero-shot benchmark accuracy is captured by word-level alignment between a pre-training corpus and the benchmark target? We study this question through word-level unigram cross-entropy (UC
Pre-training corpora leave a lexical imprint on zero-shot accuracy that is consistently tracked by word-level unigram cross-entropy across benchmarks, document selection, and prompt routing.
How much of zero-shot benchmark accuracy is captured by word-level alignment between a pre-training corpus and the benchmark target? We study this question through word-level unigram cross-entropy (UCE), a tokenizer-agnostic, model-free measure of lexical alignment between a reference corpus and a target benchmark. Likelihood training shapes the model's predictive distribution from the pre-training corpus, and zero-shot scores under likelihood-based evaluation depend in part on the lexical alignment between that corpus and the benchmark target. Cleanly tracing how the lexical imprint of pre-training data scales with corpus--benchmark alignment requires controlled from-scratch pre-training…
Summary and abstract excerpt via Hugging Face Papers ↗ · arXiv.
The Lexical Imprint of Pre-training Data was first seen on Hugging Face · Daily papers on Oct 28, 2026. Since then it has appeared in 22 checks on 3 different days. Its best position was #18 of 30, reached on Oct 30, 03:28 UTC. It dropped off the list after Oct 30, 05:28 UTC.
Position on Hugging Face · Daily papers at every check (lower is better).
Best rank per day (UTC).
Platform metric as reported by Hugging Face.
Archived daily, weekly and monthly lists that include this entry.
Not at the moment. It was last on the list on Oct 30, 05:28 UTC, after 3 days of trending.
#18 on Hugging Face · Daily papers, on Oct 30, 03:28 UTC. Trendoday checks this list every 120 minutes.
It was first seen on Oct 28, 2026 and has been on the list on 3 days (22 checks) since.