6 ms·
Bonsai 2 27B: Near-Lossless Compression in a 9x Smaller Footprint
- kamranjon 2mo agoLove this for the folks with 16gb graphics cards - 3.8 27b has been incredible but not quite runnable on anything less than 32gb - will try loading this up on my 16gb intel b50 and see how it goes - not sure these quants can be accelerated by the XPU cores yet but maybe in time!
- kadoban 2mo agoYou can run the ~4 bit quant(s) on 24gb, if you're not _too_ picky on context size. This will hopefully be better, though it'd be a _very_ surprising increase in performace at the size they say. Would love to see more about how it benchmarks.
- spijdar 2mo agoI run Unsloth's UD-Q4_K_S on 20 GB of VRAM (RX 7900 XT) and I get ~90k tokens of context without quantizing KV cache. With 8-bit quantization, I get about a 134k token context window. That's with only one slot, but for me, it works pretty darn well, with 20-35 tok/s depending on how full that window is.
- orsorna 2mo ago7900 XT is a sleeper card. When I initially bought it, it was priced at the lowest wattage per $ per GB VRAM (not normalized for token speeds...) Although I ended up swapping for the XTX because that 4GB means everything in just increasing the context window. At 8bit KV my window is over 200k, and although qwen3.8 loves vomiting out tokens as part of its reasoning chain I trust it enough to get assigned tasks done eventually, which I could not say of any model before its release.
- kadoban 2mo agoHow has software/driver support been? I got burned hard by AMD last generation or the one before. Things smoother now, or do you have to baby it like hell and pick and choose software that works?
- orsorna 2mo agoI don't do anything fancier than inference, and I only use llama.cpp, which supports rOCM. I've had few issues; most GGUFs I download work right out of the box. Nearly any popular model has a quant that just works. But as you can see I don't use my GPU for anything weird or nonstandard.
- lta 2mo agoI'm doing the same with a context of about 128-150k Surprisingly, I get subjectively better results with Unsloth's 3 bit quants (UD-Q3-XL something), than their 4 bit quants (S or M)
- redox99 2mo agoYou can trivially run 131k on 24GB 4bit, and there are repos with tweaks that allow you to get the full 262k but idk if there's degradation with their approach.
- Zambyte 2mo agoHow? I'm running 4bit with a q8 kv on a 24gb card, and I'm not able to get 100k out of it. I use a context size of 90k.
- deleted 2mo ago[deleted]
- djkoolaide 2mo agoTried it today on a B70 and couldn't get anything usable out of it. Prism's llama.cpp fork only has the kernels for CUDA, CPU and Vulkan. No SYCL at all :(
- kennywinker 2mo agoI run qwen 27b on an old-ass 16gb gpu. It’s very possible using unsloth 2bit and 3bit quants, tho there are a bunch of interesting quants that let you run closer to 4bit on 16gb. This article that’s currently also on the front page mentions a bunch of them while discussing their own quant https://byteshape.com/blogs/Qwen3.8-27B/ https://byteshape.com/blogs/Qwen3.8-27B/
- abraxas 2mo agoI'm not following the local mdoel scene too closely but this seems quite amazing. Is this able to be run on Apple silicon too?
- kamranjon 2mo ago"Ternary Bonsai 2 27B reaches up to 143 tokens/second on NVIDIA GeForce RTX 5090 and 46.8 tokens/second on M5 Max. On an RTX 4090, Ternary Bonsai 2 27B consumes just 0.714 mWh/token, making it 40% more energy-efficient than an 8B model running in full-precision."
- pizza234 2mo agoTheir mention of the 5090 is bit odd, since on 32 GB GPUs, Q6 fits while having better quality. Very interesting model for 16 GB GPUs though!
- sisve 2mo agoThey mention 5090 with regards to speed, Q6 will not have that speed? And speed matters a lot for many use cases
- selectodude 2mo ago150 tokens per second on a ternary model implies that it’s GPU bound, I’d bet a Q6 model is even faster because it’s existed longer and seen more optimization. You’d have to be insane to not run an NVFP4 quant over a ternary quant on Blackwell if they both fit.
- wincy 2mo agoWith Ninfer and Qwen 3.8 27b it uses a groupwise int mixed quant, and it gets 160 tokens/sec. The mixed quant is between 4 and 6 bits.
- Foobar8568 2mo ago
- Aurornis 2mo agoThese are small enough that you can run them entirely in the browser https://huggingface.co/spaces/webml-community/ternary-bonsai-2-webgpu-kernels https://huggingface.co/spaces/webml-community/ternary-bonsai... Remember to clear the downloaded weights afterward. Like the last model, it's amazing they work as well as they do. Use it for any longer task and they fall apart spectacularly and in interesting ways.
- outofpaper 2mo agoSo you have some fun examples?
- Aurornis 2mo agoMaybe I oversold the fun-ness of it. The most common failure mode is that it goes into loops and you come back to find it exhausted the output length without getting anywhere.
- SXX 2mo agoSadly crashing on Pixel 9 Pro, but I guess phone GPU with 16GB RAM total wouldnt be enough anyway.
- z2 2mo agoI'd love to see a Bonsai model start with a 100B+ parameter model and get that down to <30 GB. But maybe at that point we call it Topiary?
- JonSchneider 2mo agoI'm hoping they release an 8B v2 based on the Qwen 3.8 series in the near future - that would give us a really powerful model that could be run directly on users phones.
- simonw 2mo agoIf you want to try out out the GGUFs from https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#these-files-need-our-llamacpp-build https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#th... be aware that you need Prism's llama.cpp fork to get them to work, from https://github.com/PrismML-Eng/llama.cpp/releases/tag/prism-b10685-7dffb15 https://github.com/PrismML-Eng/llama.cpp/releases/tag/prism-... This should work: cd /tmp # Get the Prism macOS runtime curl -fL https://github.com/PrismML-Eng/llama.cpp/releases/download/prism-b10685-7dffb15/llama-prism-b10685-7dffb15-bin-macos-arm64.tar.gz -o bonsai-runtime.tar.gz tar -xzf bonsai-runtime.tar.gz # Get the ~5.95 GB GGUF model: curl -fL https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf/resolve/main/Ternary-Bonsai-2-27B-PTQ1_0.gguf -o Ternary-Bonsai-2-27B-PTQ1_0.gguf # Run the server, I used port 8331 ./llama-prism-b10685-7dffb15/llama-server \ -m Ternary-Bonsai-2-27B-PTQ1_0.gguf \ --port 8331 -ngl 99 -fa on -c 32768 Then open http://localhost:8331 for the (very good) baked in llama-server web UI... or run a prompt via the API like this: uvx llm openai endpoint http://127.0.0.1:8331/v1 \ --model bonsai-2-27b --responses hi That's running at ~20 token/second for me on an M5 Pro (after a server restart I got 44 token/second, not sure why), but I'm pretty sure something isn't working right, on startup the server said "ggml_metal_device_init: - the tensor API is not supported in this environment - disabling".
- deleted 2mo ago[deleted]
- simonw 2mo agoI used that to Generate an SVG of a pelican riding a bicycle: https://tools.simonwillison.net/markdown-svg-renderer?url=https%3A%2F%2Fgist.github.com%2Fsimonw%2Fba4cf3a88f4e7dc32994f2672150f770 https://tools.simonwillison.net/markdown-svg-renderer?url=ht... It took 18 minutes 20 seconds. Pretty decent for a 5.5GB model file.
- kadoban 2mo agoHonestly looks pretty good except whatever is going on with its booty. Is that an ass helmet? I cannot parse what's going on there.
- miffy900 2mo agoI really wish people would stop saying N times smaller than something when making a comparison; that makes no sense - it's 1/9th (11.11%) the size. You don't get a smaller quantity by multiplying by a number greater than 1.0. You could instead reverse the subjects being compared - "the original model is 9x bigger than this new smaller, efficient model" or some such. That makes sense. I keep seeing this being used when people talk about efficiency or performance gains and it's just very unintuitive language.
- hamandcheese 2mo agoIf we were talking about speed instead of size, i think it would be perfectly reasonable to say 9x faster. I'm not sure I agree that 9x smaller is unintuitive. It makes sense to me.
- simondotau 2mo ago"Nine times" literally means multiplied by nine, but here we're dividing by nine. It's not unintelligible (because the corrupted verbiage is so commonplace) but it is needlessly awkward. Like saying "resulted in a size reduction increase of 10 megabytes."
- D-Machine 2mo ago> "Nine times" literally means multiplied by nine Rather, "nine times larger" means multiplied by nine, and "nine times smaller" means divided by nine. This is basic and not particularly awkward, certainly not more so than e.g. positive/negative correlation, or many much more awkward and more common linguistic constructions, IMO. If you have to edit out words (i.e. context) to argue a phrase doesn't make sense... I am not sure what mental model you have for natural language, exactly, but it certainly isn't a very robust one.
- simondotau 2mo ago[flagged]
- 2mo ago
- 2001zhaozhao 2mo agoI think if they made this for Qwen3.8-Next it could fit in a single 5090?
- kennywinker 2mo ago180b * 1.76 bits per weight = 39.6 gigabytes. Best you could realistically run in 32gb is like 28gb, or a 127B param model
- jokethrowaway 2mo agoQwen3.8-Next, thanks to its new architecture, is quite fast even if part of it is streaming from disk
- kennywinker 2mo agoTotally. Any MoE model can have experts swapped in and out from disk or system ram. I only framed it this way because the question was about the model fitting in vram.
- danbrooks 2mo agoNice! Does anyone know how this compares to the Unsloth quantizations of this model? https://unsloth.ai/docs/models/qwen3.8#run-qwen3.8-guide https://unsloth.ai/docs/models/qwen3.8#run-qwen3.8-guide
- 0xbadcafebee 2mo agoCame to ask the same. From my really rough understanding, it seems like Unsloth's method allows a slightly higher precision at a higher file size, while PrismML's uses a different approach to achieve a smaller size (and presumably less precision).
- nulld3v 2mo agoThere's a table on the HF page that compares it against Unsloth's UD-Q4_K_XL and IQ2_XXS: https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#full-per-benchmark-results https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#fu...
- kadoban 2mo agoOh, wow, they think it's just a smidge below the q4? That's crazy good if true.
- anana_ 2mo agoThe benchmarks they chose are rather cherry picked to not include long context or difficult ones that involve long horizon work or many agent turns, as I suspect this is where the model shows more differences compared to the full fat one
- SillyUsername 2mo agoYep more hops from the lower Q is likely going to skew the vectors further over time. I wonder if there's a way to mitigate this by running it through an original Q8 draft model, attuned somehow for the PTQ1 quant, but giving it a higher threshold for the acceptance linear with the context length itself? The longer the context, the higher the multiplier on the threshold, and more likely the draft result is used. Not ideal but it may extend the usable max context. This model might, even without this, be amazing for short lived agents that work via generations / have changing tasks.
- adrian17 2mo ago> Ternary Bonsai 2 27B uses ternary {−1, 0, +1} weights with FP16 group-wise scaling, for 1.76 effective bits per weight If I recall correctly, a recent post [1] has shown that Q2 quants (with like 2.6 bpw) of the same base Qwen model sit at the edge between "noticeably worse" and Q1's "useless". I took a quick glance at Bonsai's blog posts, and don't really see them comparing themselves to "typical" quants or explaining what's the special sauce that makes them better? https://news.ycombinator.com/item?id=49611128 https://news.ycombinator.com/item?id=49611128
- edflsafoiewq 2mo agoI think the general idea is naive quantization falls apart below 4bpw but you can go lower with more sophisticated QAT-adjacent methods. Bonsai's quantization method is proprietary though.
- yowlingcat 2mo agoThat's correct. I think there can certainly be issues even with 4bpw with naive quantization (IE you'll notice far better results from a QAT 4bpw vs a naive 4bpw). One such method that I've been meaning to look into further is Tencent's AngelSlim QAT/PTQ approach. They did a Hy4 preview release thats an STQ_1_0 at 2.38 bpw: https://huggingface.co/AngelSlim/Hy4-preview-GGUF https://huggingface.co/AngelSlim/Hy4-preview-GGUF https://arxiv.org/abs/2602.21233 https://arxiv.org/abs/2602.21233 Of course, it's still 213g of VRAM I'd need so it's somewhat out of the range of what I can run locally. In contrast, this new Bonsai is nice because the original was already exciting for making use of low VRAM devices. Could breath new life into some of the older GPUs that were previously close to top of the line just quite VRAM constrained by modern standards and still quite cost effective for now.
- om8 2mo agoCould've been better if GGUF implemented QTIP format. GGUF representation is a major limitation for llama.cpp quantization performance
- edflsafoiewq 2mo ago
- logicallee 2mo ago(In case anyone remembers the compression post from yesterday[1], I checked and this one doesn't qualify for further compression - it's not zero-biased at all.) [1] https://news.ycombinator.com/item?id=49732931 https://news.ycombinator.com/item?id=49732931
- Havoc 2mo agoCautiously optimistic. The V1 was noticeably weak on world knowledge but here the 3.8 base model is geared more towards reasoning than world knowledge anyway so might not matter as much
- jedbrooke 2mo agoRunning at about 7-8 tok/s (~60 tok/s prefill) on a Mac Mini M2 16GB. So far feels smarter than Bonsai 1 27B, it’s slightly larger than the Q1_0 quant. Super exciting stuff :)
- flutetornado 2mo agoGPT Astra did some benchmarking on the DGX Spark. Speed: 34.38 tokens/sec for generation. Seems like we don't have a drafter model yet so it could not test with speculative decoding on. ngram speculative decoding did not help too much either - not enough accepted tokens. Smaller size I suppose does not mean better performance in this case - we maybe limited by Spark's low memory bandwidth.
- cmrdporcupine 2mo agoWhat are you getting for prefill?
- flutetornado 2mo ago450 with PTQ_01 and 900 with the other PQ2_0.
- mmastrac 2mo agoThat's a rough place to land on a spark. It seems unlikely to be memory bandwidth at this model size, but maybe just lack of tuned kernels? The chip is missing some CUDA features but with tuning you should be able to hit way more than that even without a drafter.
- flutetornado 2mo agoI was wondering if those ternary bits get expanded into full floats internally in the kernels - you’re probably right about lack of tuned kernels. I’m not sure you could just tune your way out of that easily though. Any suggestions on trying particular solutions?
- ipoole_dev0 2mo ago[flagged]
- circularfoyers 2mo agoI wonder how their talks with Apple went. Having this run on the TPU opposed to just the GPU, which drains a significant amount of battery life by comparison, is what I'm really interested in.
- deleted 2mo ago[deleted]
- cmrdporcupine 2mo agoWhat I'd love to see is this done for DS4.1 Flash. That would bring it down to the point where it can fit in 128GB on things like the Spark or Strix Halo.
- redlimetea 2mo ago[dead]
- hedora 2mo agoRAM requirements? My current rule of thumb is “a byte per parameter”, but I doubt this runs in 1/9th that (~ 3GiB). Also, perf speedup?
- jjcm 2mo agoI'm seeing around 7.9GB of ram, 120 tokens/s on a 6000 pro blackwell.
- kennywinker 2mo agoWithout context, it should be number of params * 1.76 (the “effective bits per weight”) / 8 So for this one, 27B * 1.76 / 8 = 5.94 GB For speed, far as I can tell it depends if your gpu is memory bandwidth bound or (mostly older gpus) processing bound. If it’s memory bandwidth bound, and your gpu gets 300GB/s, that’s: 300GB/s / 5.98 GB = 50.5t/s. Realistically it’s probably a bit slower, but that is your theoretical maximum.
- Dwedit 2mo agoIt needs more VRAM than just the model weights. With 6GB of VRAM, I got 44/65 layers loaded into VRAM. Has anyone tested 8GB yet?
- kennywinker 2mo ago“Without context” was me gesturing at that. Interesting you can’t fit the whole model tho - why is beyond my current understanding :)
- Dwedit 2mo agoEven with a small context size (4096 at 64KB per token), that's like 256MB for the context. It's more than just the context that's eating up VRAM.
- g023 2mo agoThey need to make a Big Bonsai, something at the enterprise levels that can compete with DSV4 Flash etc.
- blactuary 2mo agoWhat is never totally clear with a lot of these releases is the scope of what it's good at. Models that can run with good speed on affordable consumer hardware for coding only is the dream. I am never going to use this for writing, images, or "general knowledge". Coding only
- nullbio 2mo agoAny time I'm doing frontend work it involves images. I think images are important.
- redox99 2mo agoI tried their WebGPU version and it immediately started looping. Yeah "near lossless" my ass. Plus the reasoning that it looped on was clearly wrong and unlike the non quantized 27B
- respectattentio 2mo agoNever heard of Bonsai before, but that looks great and promising for local on-device inference. Yet, seems like there is still another year for improvements. I like local models (but not mainly using them) for offline needs.
- jocelyner 2mo ago[dead]
- sb057 2mo ago[flagged]
- avaer 2mo agoLLM quants seem to eerily converge to modern/not so modern graphics techniques. You wouldn't think it would apply but it's obvious in hindsight. In fact mining graphics ideas is probably a good inspiration for efficient LLM architecture. For example, the Hadamard activation transform used here feels a lot like multiplying Fourier basis ala DFT; strong parallels to how image codecs work to make the residuals more compressible (especially discrete block codecs like are used in GPU compressed textures). I thought I was being clever suggesting that you could even abuse texture decode units to efficiently sample compressed LLMs with hardware; turns out Apple foundation models are already doing this [1]. [1] https://arxiv.org/abs/2507.13575 https://arxiv.org/abs/2507.13575
- Dwedit 2mo agoI tried it on my 6GB GPU and got 0.67 tokens per second. Need more than 6GB to run it well.
- huseyinkeles 2mo agoTesting on a MBP m4 pro 24gb ~100t/s prefill, ~15t/s, dropping to ~10t/s later with 64k context. The issue is I have yet to find a useful agentic local llm that I can run on this machine. Just given a relatively simple task on a swift app, took 25 minutes, brainstorming like crazy but can not decide on what to do. Eventually I killed it. GPT 5.6 sol-medium took 3 minutes to complete the same task for reference.
- aetherspawn 2mo agoGemma 30B with 256K context runs at 20 tok/sec on my M3 Max with 128GB RAM so I think there’s something wrong with your setup. This should run at ~30-40 toks. Maybe your inference engine is not optimised for Mac.
- huseyinkeles 2mo agoI just used their `Bonsai-demo` repo like this; `cd ~/Code/Bonsai-demo && BONSAI_CTX=65536 ./scripts/start_llama_server.sh` then used it in a very minimalistic pi with a very small system prompt. Didn't spend much time to try to optimize it tbh, but my issue was not the speed. it just could not make a decision on how to implement the task, kept going on an on.
- yearolinuxdsktp 2mo agoMaybe you have to set reasoning effort to low. 3.8 27B on x-high (default) reasons forever on anything complex. I asked it to write down the answer plan so far leaving open questions as open and it wrote the plan twice in reasoning (and more times partially) while it dilly-dallied about open questions before realizing “ok the user just asked to leave questions open.” That was a Q6o quant with unquantized KV cache.
- aetherspawn 2mo agoYes, I use LM Studio with MLX support, which is specifically faster on M series Macs. I am not sure if llama is the same, but I guess what I’m saying is if you want the performance to be good on M series you have to use models packaged in the right format.
- zhiyan 2mo agoAwesome results. Opens up doors for a lot of people.
- deleted 2mo ago[deleted]
- hilti 2mo ago[flagged]
- nilsherzig 2mo agoFyi, if you're trying to run this under AMD/HIP: PTQ1_0 has no optimized MMQ-Path in their llama-cpp fork, try running PTQ2_0 (needs a bit more vram, but is about 2x faster on my 6700 XT) https://gist.github.com/nilsherzig/b8266d001c5c01bdb3d81d20915572fc https://gist.github.com/nilsherzig/b8266d001c5c01bdb3d81d209...
- jakswa 2mo agowhat kinda speeds do you see on 6700 XT? i'm always conflicted on investing time chasing speed-vs-quality tradeoffs. I've got a 7900 XT (about double the IO throughput). I'll probably end up giving it a go when I find time.
- verytrivial 2mo agoThere's a chap called Bijian Bowen who does very quick agentic coding challenges for new models (very soon after release!) mainly for toy games or websites. He just did one for this model and included a comparison with the base model Qwen 3.8 which shows the "near-lossless" claim should be taken with a grain of salt. It is an interesting model if you are GPU starved and want local, but you might have trouble finding things it is good at.
- Aurornis 2mo ago> which shows the "near-lossless" claim should be taken with a grain of salt I agree. I don’t know how they get such good results on these benchmarks because using them gives a very different experience. They’re kind of cool for doing short free form outputs in memory constrained systems, but I don’t think they’re useful as coding agents.
- UrineSqueegee 2mo agothese are very saturated benchmarks
- qingcharles 2mo agoThe only two I trust are Bijan and simonw.
- mpweiher 2mo agoIs it just me or are local models getting better (catching up) a lot faster than the frontier models are getting better (creating distance)? If true, that would be a very welcome development.
- deleted 2mo ago[deleted]
- hvhvubufyvycjcx 2mo agoHello! May I ask, is this model compatible with my RX 9070 on Linux?
- hvhvubufyvycjcx 2mo agoThe dev didn’t list AMD Radeon support, and I couldn’t get it to work
- Chance-Device 2mo agoLet’s see, so if you get the same 1/9th the size compression ratio with GLM-5.3-Flash, then you’d end up with a ~72GB model that’s about as good as GPT-5.6 Sol (high), according to artificialanalysis.ai Which is within reach of some higher end consumer hardware, especially with layer offloading. You have to wonder what kind of trouble the “labs” are in when this is becoming possible. Lots of money, where’s the moat?
- indy 2mo agoThey're trying to build a moat with legislation, using fear over 'safety' as an excuse to ban these open models
- DoctorOetker 2mo agoA gentleman's agreement between the western bloc, China, India, Russia, ... ? It would be a lip service agreement, and all would continue the machine learning race... The mere suggestion of slowing down AI progress signals to adversaries that there is some novel power just identified: do you think adversaries would agree and slow down, or calculate a little harder and longer in order to also figure out what it unlocks?
- indy 2mo agoDo you ever wonder how Anthropic or OpenAI can release a new state of the art model and then a few months later there's a Chinese open model that's nearly as good?
- kllrnohj 2mo agoThe labs still have performance as a differentiator for coding usages and similar, and for other things there's still all the same reasons people switched cloud hosted stuff in the first place. AWS & friends didn't get popular because the hardware was out of reach, after all.
- Chance-Device 2mo ago
- claud_ia 2mo ago[flagged]
- nullbio 2mo agoWell this is quite impressive!
- cregy 2mo agoOn openrouter I had to filter out glm 5.3 flash instances running fp4 - making sure it only ran fp8 - as the quantized models kept going crazy / off track. Assume bonsai has the same ticks
- thway15269037 2mo agoI wonder if they could apply the same to Qwen3.8-Flash-Next. If they did 60gb -> 6gb to Qwen-27B, could they possibly do the same to the MoE model. 36gb lossless 126B model seems like an impossible task.
- v3ss0n 2mo agoOn actual agentic task , it just fail.
- antonly 2mo agoWhat did they do with their benchmarks? I've never seen Qwen3.6 27B this close to Qwen3.8 27B in any aggregated summary... Makes one questions the entire accuracy section.
- petrenk0n 2mo agoFor anyone who wants to try Bonsai 2 without setting up runtimes, downloading the right quant etc, try here - https://triangllabs.ai/otis https://triangllabs.ai/otis
- itsmeduncan 2mo ago[flagged]
- euroderf 2mo agoStupid question: Does "total model footprint of 5.9GB" mean it will run in 8GB of RAM ? Or is that the size on disk ?
- Aurornis 2mo agoThat’s the size of the model. Running it requires additional space for the KV cache depending on how much context you use. You can probably get it running in 8GB of RAM for short outputs but to get okay context length you’d want more.
- jakswa 2mo agoBonsai 2 27B · Radeon RX 7900 XTX - 89 tokens/sec generation with speculative decoding - 81 tokens/sec at 20k context - 474 tokens/sec ingestion at 20k — about 42 seconds - 10.1 GiB peak VRAM with a 24k context window ROCm 7.2.3 · PQ2_0 · Qwen Q4 MTP, draft length 2 ---- versus ---- Qwen3.8-27B IQ3_S · Radeon RX 7900 XTX - 79 tokens/sec generation on a short coding prompt - 61 tokens/sec at 60k context - 53 tokens/sec at 95k context - 558 tokens/sec ingestion at 60k — about 108 seconds - 19.9 GiB peak VRAM during coding tests with a 100k context window Vulkan · GSQ-RCO IQ3_S · MTP, draft length 2 · vision projector loaded
- imagetic 2mo agoHas the hype fizzled out yet? Can anyone post a link to something they've done with success?