About this paper
A dynamic weight precision allocation strategy is proposed, utilizing low-bit quantization and uncertainty-based gating to selectively upgrade weight precision during LLM generation, thereby optimizing computational cost.
Local decoding usually fixes one weight precision for the whole generation. We study a different allocation: store 8-bit weights, decode with their top 4 bits, and redo a step at 8 bits only when the low-precision output distribution looks uncertain (entropy above a threshold or a small top-two margin). Quantization is simulated, the model is Qwen3-0.6B, and the benchmark is GSM8K. Cost is counted in weight bytes read per token. On 36724 replayed steps the entropy of the low pass ranks argmax flips with AUC 0.93. At a 15% escalation budget it catches 78% of flips and the margin gate 94%, against the 15% expected for a random gate of equal rate (a computed expectation, not a run). In 150…
weight precisionquantization8-bit weightstop 4 bitsentropymargin gateescalation budgetKL divergenceQwen3-0.6BGSM8K
- Upvotes
- 5
- Authors
- 3
- Published
- Nov 10, 2026
Summary and abstract excerpt via Hugging Face Papers ↗ · arXiv.
Trending history
Precision on Demand: Escalating Weight Bits at Uncertain Decoding Steps in a Small Local Model was first seen on Hugging Face · Daily papers on Nov 12, 2026. Since then it has appeared in 26 checks on 5 different days. Its best position was #25 of 30, reached on Nov 13, 23:28 UTC. It dropped off the list after Nov 16, 03:28 UTC.