BYOD converts autoregressive language models into diffusion models via parameter-efficient LoRA adapters, enabling parallel generation with minimal training cost while preserving base-model knowledge.
Unlike autoregressive (AR) models, diffusion language models (DLMs) permit parallel token updates, flexible generation, and bidirectional attention, but leading systems rely on costly pretraining or full-model fine-tuning. We present Build Your Own DLM (BYOD), a generalizable LoRA-only recipe for parameter-efficient AR-to-DLM conversion. Without architecture-specific tuning, BYOD induces diffusion generation in Gemma 2 9B, Llama 3.1 8B, Qwen2.5 7B, and Ministral 8B; to our knowledge, this is the first reported AR-to-DLM conversion of a Mistral model. The adapters comprise 4.1-5.8% of final parameters while every base-model parameter remains frozen; each 25,000-iteration conversion needs at…
diffusion language modelsLoRAparameter-efficient fine-tuningautoregressive modelsbidirectional attentionmasked sequencesnetwork function evaluationperplexity
- Upvotes
- 5
- Authors
- 2
- Published
- Oct 30, 2026
Summary and abstract excerpt via Hugging Face Papers ↗ · arXiv.