FlashMoE Enables SSD Offloading for Efficient On-Device Mixture-of-Experts Inference

Global: FlashMoE Enables SSD Offloading for Efficient On-Device Mixture-of-Experts Inference

A team of computer scientists has introduced FlashMoE, a system that moves inactive experts of large mixture-of-experts (MoE) language models to solid-state drives (SSD) to facilitate inference on devices with limited RAM. The approach reportedly raises cache hit rates by as much as 51% and delivers speed improvements of up to 2.6 times compared with earlier MoE inference solutions.

Background on Mixture-of-Experts Models

Mixture-of-Experts architectures achieve scalability by activating only a small subset of model components—called experts—during each inference step. This sparse activation reduces computational load, yet the overall model size can reach hundreds of gigabytes, creating storage challenges for edge devices.

Challenges with Existing Inference Systems

Prior systems such as Fiddler and DAOP rely on offloading data to DRAM, a strategy that becomes impractical when model footprints exceed the memory capacity of typical on-device environments. As MoE models continue to grow, DRAM‑based solutions risk excessive latency and energy consumption.

FlashMoE Architecture

FlashMoE addresses the memory bottleneck by storing inactive experts on an SSD and loading them on demand. The design incorporates a lightweight machine‑learning‑driven cache that evaluates both recency and frequency of expert usage, aiming to maximize reuse while minimizing storage I/O.

Adaptive Caching Strategy

The caching algorithm combines recent access patterns with long‑term usage frequencies, allowing the system to predict which experts are likely to be needed soon. By dynamically adjusting cache contents, FlashMoE reduces the number of SSD reads required during inference.

Experimental Evaluation

Researchers built a user‑grade desktop platform to test FlashMoE under realistic hardware conditions. Benchmarks showed that the new caching policy outperformed traditional policies such as Least Recently Used (LRU) and Least Frequently Used (LFU), achieving the cited 51% increase in cache hit rate and up to 2.6× faster inference compared with existing MoE systems.

Implications and Future Work

The results suggest that SSD‑based offloading can make large‑scale MoE models viable for edge applications, potentially expanding the range of on‑device AI capabilities. Ongoing work may explore further optimization of the caching model, integration with other storage technologies, and evaluation on a broader set of hardware configurations.

This report is based on information from arXiv, licensed under Academic Preprint / Open Access. Based on the abstract of the research paper. Full text available via ArXiv.