TIDEWAY enables efficient inference of large Mixture-of-Experts models on memory-constrained consumer devices by treating host RAM and GPU VRAM as a unified memory pool with phase-aware layouts and optimized CUDA Graph execution under expert paging.
Large language model (LLM) agents are rapidly becoming a dominant paradigm for AI applications. Recently, there is a growing demand to deploy large-scale Mixture-of-Experts (MoE) models locally on consumer PCs to balance data privacy, API costs, and inference efficiency. However, unlike datacenter servers with abundant DRAM, consumer devices are tightly constrained in both GPU VRAM and host RAM, forcing massive MoE weights to spill across VRAM, host RAM, and SSD. Under such spilling, we observe that existing systems fail to fully utilize this memory hierarchy, treating the tiers as independent caches and thrashing in inference. We present TIDEWAY, a system for efficient inference of large…
large language model (LLM) agentsMixture-of-Experts (MoE) modelsinferenceGPU VRAMhost RAMprefilldecodephase-aware memory layoutsCUDA Graph executionexpert paging
- Upvotes
- 4
- Authors
- 10
- Published
- Nov 10, 2026
Summary and abstract excerpt via Hugging Face Papers ↗ · arXiv.