What happened
A startup called PrismML has released Bonsai 27B, a pair of extreme-compression versions of the open-weight Qwen3.6-27B model, and one of them is small enough to run entirely on a smartphone. The weights are published on Hugging Face under the Apache 2.0 license, and PrismML's own demos show the model reasoning, calling tools, and reading screenshots directly on an iPhone 17 Pro Max — with zero requests sent to the cloud.
The achievement is about size, not just speed. Qwen3.6-27B normally needs about 54 GB of memory at standard 16-bit precision. Even a solid 4-bit build still weighs in around 18 GB — too big for a phone and for most laptops too. PrismML shipped two much smaller alternatives instead. The ternary version stores each weight as one of three values (-1, 0, or +1) and shrinks the model to 5.9 GB, aimed at laptops and budget GPUs. The binary version goes further, using only {-1, +1}, landing at 3.9 GB and roughly 1.125 bits per weight — built specifically for phones.
Accuracy is preserved mainly through shared FP16 scaling factors, one per group of 128 weights, and the compression is applied uniformly across the entire network — embeddings, attention layers, MLP blocks, and the output head — with no pockets of higher precision left untouched. That uniformity matters: models advertised as "2-bit" often hide such pockets and end up averaging closer to 2.8 bits per weight in practice. Bonsai's binary version doesn't cheat that way, which is how it gets under 4 GB.
Why it matters
Memory limits, not raw processing power, are usually what keeps large language models off phones. The iPhone 17 Pro Max has 12 GB of RAM, but iOS caps any single app at around 6 GB — a budget that has to cover the model itself plus the KV cache and activations that grow as the conversation gets longer. No standard build of a 27-billion-parameter model comes close to fitting that ceiling. Bonsai's 3.9 GB binary version does, with room left over to actually run.
PrismML tested both versions across 15 benchmarks in six categories, in "thinking" mode where the model reasons at full depth. The company reports the ternary version retains 95% of the original model's performance, and the binary version retains 90%. The drop isn't even across tasks. Math (GSM8K, MATH-500, AIME) barely moved — 95.3 for the original versus 93.4 (ternary) and 91.7 (binary). Coding slipped moderately, from 88.7 to 86.0 and 81.9. Agentic tasks and tool-calling (BFCL v3, TauBench) took the biggest hit, falling from 80.0 to 74.0 and 66.0 — notable because agentic use is exactly the scenario PrismML is marketing the model for.
For comparison, PrismML also quantized a standard 2-bit version of the same Qwen model without its technique. That build held up on short knowledge tests like MMLU but collapsed on tasks requiring long reasoning chains, scoring just 57.5 on AIME26 and 56.4 on LiveCodeBench. A surface-level test would miss that weakness entirely — it only shows up once a model has to reason for many steps, which is precisely where Bonsai's approach holds its ground while using 3.9 GB instead of 9.4 GB, less than half the memory.
How to use it today
Both Bonsai versions keep the original model's full feature set: multimodal input via a separate 4-bit vision tower, a 262,000-token context window, and support for speculative decoding. Developers can pull the weights from Hugging Face right now under Apache 2.0, meaning commercial use is permitted.
The catch is speed. On the iPhone 17 Pro Max, the binary version tops out at around 11 tokens per second — below the threshold most people would call a comfortable, real-time chat experience. Tooling is also rough at launch: LM Studio and Unsloth didn't support the format on day one, the ternary version still partly depends on PrismML's own llama.cpp fork, and Apple devices need the company's custom MLX fork rather than the stock one. If you want to try running compressed open models on your own hardware without wrestling with any of that setup, a service like [mykreatool.com](https://mykreatool.com) offers free AI tools you can use directly in the browser while the on-device tooling matures.
Who benefits
Developers building offline-first apps — field tools, note-taking assistants, translation apps for areas with poor connectivity — get the clearest win, since Bonsai removes the API bill and the network dependency entirely. Privacy-sensitive use cases benefit too: screenshots, documents, or conversations never leave the device, which matters for legal, medical, or internal-business applications where sending data to a third-party API isn't an option.
Hardware makers and OS vendors have an obvious interest as well, since a genuinely capable 27B model running locally is a strong argument for premium phone memory tiers. Researchers studying quantization get a useful data point too: proof that 1-bit weights can preserve 90% of a 27B model's benchmark performance when applied consistently, rather than patched with high-precision exceptions.
Risks
The agentic and tool-calling drop — from 80.0 down to 66.0 in the binary version — is the biggest practical concern, since it directly undercuts the "autonomous on-device agent" pitch PrismML is making. A phone assistant that reasons well about math but stumbles on multi-step tool use is a mismatch for that use case.
The 11 tokens-per-second speed on binary weights will feel sluggish for anything beyond short exchanges, and platform support is genuinely immature: Android isn't officially supported yet, and at least one Hacker News user reported the model on Android returning a string of exclamation marks instead of a real answer. Apple-specific tooling requirements and the reliance on forked inference engines mean this is still closer to a research preview than a polished consumer feature, despite the impressive compression numbers.
Conclusion
Bonsai 27B is a genuine milestone: a 27-billion-parameter model, quantized down to roughly 1 bit per weight, running fully offline on a phone with about 90% of its original benchmark performance intact. The math and coding scores hold up well, the agentic and speed numbers still need work, and the tooling around the release is rough. But as a proof of concept for cloud-free, privacy-preserving AI on everyday hardware, it's a clear signal of where on-device models are headed next.
Comments 0