Bonsai 2 27B: Near-Lossless Compression in a 9x Smaller Footprint
Bonsai 2 27B's headline is a 5.9GB ternary 27B-class multimodal model with claimed near-lossless benchmarks, but the discussion suggests the real test is whether it remains useful in long agentic tasks and how it compares with other low-bit quants and upstream tooling.
The brief
PrismML announces Ternary Bonsai 2 27B, a ternary-weight version of Qwen3.8 27B that uses {-1, 0, +1} weights with FP16 group-wise scaling, a 5.9GB footprint, 262K context, multimodal input, and Apache 2.0 licensing. PrismML claims it is more than 9x smaller than the full-precision model while retaining 98.2% of aggregate benchmark performance, with strong throughput and energy-efficiency claims.
- Ternary Bonsai 2 27B uses ternary {-1, 0, +1} weights with FP16 group-wise scaling, 1.76 effective bits per weight, and a 5.9GB total footprint; it supports 262K context, multimodal text-and-image input, and Apache 2.0 licensing.
- PrismML claims it is more than 9x smaller than its full-precision counterpart while retaining 98.2% of aggregate benchmark performance, scoring 83.9 overall versus 85.4 for Qwen3.8 27B and 83.6 for Qwen3.6 27B.
- Benchmark retention varies: math 96.57 vs 97.06, coding 81.58 vs 82.17, instruction following 82.66 vs 81.25, but knowledge/reasoning 83.95 vs 86.66, vision 78.59 vs 81.64, and agentic/tool calling 77.57 vs 79.74.
- PrismML reports up to 143 tokens/second on RTX 5090 and 46.8 tokens/second on M5 Max; on RTX 4090 it claims 0.714 mWh/token, 40% more energy-efficient than a full-precision 8B model.
- The model runs on NVIDIA GPUs via CUDA and Apple devices via MLX through custom low-bit kernels; PrismML says retention improved from 95% in the first Bonsai 27B to over 98%, making the release practically lossless.
In the discussion
Commenters focus on practical local deployment, quality caveats, comparisons with other low-bit quants, throughput and hardware limits, tooling friction, and a terminology dispute over '9x smaller'.
Practical local deployment is the main draw
kamranjon says Qwen3.8 27B was not runnable under 32GB and plans to try this on a 16GB Intel B50; pizza234 calls it interesting for 16GB GPUs but questions the 5090 framing because Q6 fits 32GB with better quality; wombat23 reports running the GGUF on an RTX 3070 8GB with q8 KV cache and getting 40.6 tok/s generation.
Quality caveats remain for longer tasks
Aurornis says the browser demo is impressive but longer tasks fall apart spectacularly; huseyinkeles reports a simple Swift app task took 25 minutes and the model could not decide, versus 3 minutes for GPT 5.6 sol-medium; adrian17 notes a recent post found Q2 quants of the same base Qwen at the edge of 'noticeably worse' and says Bonsai does not compare itself to typical quants or explain its special sauce; edflsafoiewq attributes low-bit viability to sophisticated QAT-adjacent methods and says Bonsai's quantization is proprietary.
Comparisons with other quantizations and speculative implications
danbrooks asks how it compares to Unsloth quantizations; nulld3v points to an HF table comparing against Unsloth UD-Q4_K_XL and IQ2_XXS; WithinReason says Unsloth has been 'dethroned' by ISTA's 3-bit quant, described as lossless based on benchmarks; Chance-Device extrapolates that applying similar 1/9 compression to GLM-5.3-Flash would yield a ~72GB model comparable to GPT-5.6 Sol high, within high-end consumer hardware with offloading, and questions labs' moat.
Throughput and hardware limits vary widely
flutetornado cites a DGX Spark benchmark of 34.38 tok/s generation, no drafter for speculative decoding, and ngram speculative decoding not helping much, suggesting memory bandwidth may limit performance; simonw reports ~20 tok/s on M5 Pro (44 after restart) and a startup warning that the tensor API is unsupported; huseyinkeles sees ~15 tok/s dropping to ~10 tok/s at 64k context on M4 Pro 24GB; SXX says the browser demo crashes on Pixel 9 Pro.
Tooling and ecosystem friction
simonw says the GGUFs require Prism's llama.cpp fork and provides commands; refibrillator questions those instructions, pointing to a demo repo setup.sh and saying the release tag/weight file do not match; nikwen and wombat23 want upstream llama.cpp support.
Terminology dispute over '9x smaller'
miffy900 argues it is mathematically nonsensical and should be '1/9th the size'; replies debate whether '9x faster' is analogous, whether the phrase is a useful linguistic convention, and whether '11% as large' is clearer.