Janas: Run LLMs larger than RAM by streaming experts from SSD
Janas-LLM reports streaming 48 GB MoE models from NVMe on a 32 GB laptop at 24 tok/s, far above llama.cpp in its tests, but the project is an untested experiment and the supplied discussion questions agentic and virtual-
Janas-LLM is a C inference engine that runs mixture-of-experts language models larger than RAM by keeping weights on an NVMe SSD and streaming only the experts each token needs, caching frequently reused experts in memory.
On a single 32 GB Intel laptop, its author reports Qwen3-Next-80B-A3B (48.4 GB) at 62 tok/s prompt reading and 24 tok/s generation, versus llama.cpp at 16 and 2.8 tok/s, and faster results across 13 models. The project is explicitly an early experiment: the API changes, hardware support is untested beyond one machine, and aggressive caching/O_DIRECT disk reads can push a desktop into swap.
Key takeaways
- Streams MoE experts from NVMe with O_DIRECT; frequently used experts stay in a RAM cache, allowing models larger than system memory to run.
- Benchmarks claim Qwen3-Next-80B-A3B at 62 tok/s prompt and 24 tok/s generation on a 32 GB laptop; llama.cpp needed special build flags and managed 16 and 2.8 tok/s.
- Reports faster prompt and generation throughput than llama.cpp on all 13 compared models, including dense and MoE Qwen/Gemma variants, with one 3-bit Qwen exception on prompt reading.
- Supports optional Vulkan GPU offload, speculative/draft decoding, OpenAI-compatible HTTP serving, MCP tools, and code completion via janas-server.
- Caveats: early development, unstable API, only tested on one Intel Core Ultra 9 185H laptop; may cause swap, driver resets, or refuse models on other hardware.
- Measurements are means of three rounds where noted, but comparison with llama.cpp is not exactly apples-to-apples and is from one machine.
What people think
Analysis details
8 comments analyzed. Analyzed .
The sample selects active discussion branches and includes available parent comments. It may omit parts of the full thread. The analysis included 8 of 11 usable stored comments because of the input limit. Selected themes do not measure community opinion or how common a view is.
Open-weight AI as non-monopoly normalization
A commenter sees open weights and open-source tooling as a source of hope, arguing that the proliferation of such projects signals the start of normalizing openly available, non-monopoly AI tools.
Hardware as the current choke-point
Replies argue hardware is becoming the limiting factor; if resolved, progress could accelerate. One commenter suggests current hardware has headroom that shortages will force optimization into, raising the achievable upper bound once the bottleneck eases.
62 tokens/sec for local chat vs agentic workloads
A link illustrates 62 tokens/sec as sufficient for ordinary local chat, but a reply qualifies that 60 tokens/sec may be acceptable for pure chat or token generation while being inadequate for token preprocessing or agentic work.
Reimplementations compared with llama.cpp
One commenter questions repeated reimplementations of what llama.cpp has offered for years. A reply explains the motivation: up to 2x performance over llama.cpp on the same model weights by optimizing for serving a single model well.
NVMe-integrated accelerators for bandwidth and model switching
A commenter speculates about small commercial accelerators integrated with NVMe drives, suggesting they could use more bandwidth than typical four-lane NVMe and, with greater capacity, allow very fast switching between models because all would be effectively loaded.