10 ms·
Show HN: Getting GLM 5.2 running on my slow computer
A few days ago I found myself trying out GLM 5.2 and was really positively impressed. The capabilities and security I was getting from this LLM are similar to those I've gotten from models like Claude or GPT, and this really surprised me.
But then I thought, "I wonder how it would work on a normal computer like mine," and above all, "I wonder if it would work without going into OOM on a computer like mine." So I started working with the help of agents to test this possibility.
I started converting the model to int4, understanding MTP usage, and if possible implementing DSA for long context. How it responds in int4 and whether the quality is maintained or not. Until I got to the point, on my computer with 32GB of RAM, I was able to communicate with GLM 5.2 with times that, of course, aren't high in cold start, but even then, we're talking about 0.1 tok/s, but that wasn't important to me. The important thing was the journey to reach this goal. I just wanted it to work at all costs, even slowly.
So I created Colibrì, which was born from a very simple idea, to be honest, but tested in every way, where a 744B Mixture-of-Experts model activates only ~40B parameters per token—and only ~11 GB of those change from token to token (the routed experts). So:
The dense part (attention, shared experts, embeddings—~17B params) stays resident in RAM at int4 (~9.9 GB); The 21,504 routed experts (75 MoE layers × 256 experts + the MTP head, ~19 MB each at int4) live on disk (~370 GB) and are streamed on demand, with a per-layer LRU cache, an optional pinned hot-store, and the OS page cache as a free L2.
The engine is a single C file (c/glm.c, ~1,300 lines) plus small headers. No BLAS, no Python at runtime, no GPU.No GPU or serious hardware because I don't have that hardware so I can't test it on hardware that is more powerful than my computer.Colibrì is a one-person project, written and tested entirely on a 12-core laptop with 25 GB of RAM — the numbers above are the ceiling of what I can measure at home.
Any feedback is welcome! (and if anyone wanted to participate in the project I would be delighted)
Repo: https://github.com/JustVugg/colibri https://github.com/JustVugg/colibri
- miohtama 3mo agoThis is the hacker spirit
- vforno 3mo agoThank you so much, it's true! It all started with this spirit!
- mariopt 3mo agoI wonder if you could replicate this in a Colourful GeForce RTX 50-series GPU, they ship it with 2 NVMe drive slots.
- vforno 3mo agoI'd love to! Right now I only have a very consumer-grade computer that I've had fun with! We'll see!
- fwipsy 3mo agoI was really curious about it - looks like the extra drive slots are NOT wired to the GPU's VRAM, they're just using PCIE bifurcation to free up some extra lanes for people who plug an x8 GPU into an x16 slot.
- nerder92 3mo agoIs this inspired by antirez work on ds4? Amazing job!
- vforno 3mo agoAntirez is the number one!thanks really thanks!
- kamranjon 3mo agoAntirez has a GLM 5.2 branch now in dwarfstar: https://github.com/antirez/ds4/tree/glm5.2 https://github.com/antirez/ds4/tree/glm5.2 It heavily utilizes ssd streaming from my understanding and I think he mentioned getting some semi usable speeds on a 128gb m5 mbp.
- Pragmata 3mo agoWould this cause issues with SSD lifespan?
- vforno 3mo agoWhat causes problems is the rewriting in this case are only read while writing is the cache! However, I'm working to improve more and more and make some parts lighter!
- Pragmata 3mo agoIs it possible to run this into an agent? pi, claude code, etc..? I've only tried it with LM studio, but i'm guessing this is a bit different
- vforno 3mo agoWe're working on it right now with a pull request that will also arrive for opencode!
- Archit3ch 3mo agoYou can keep the KV cache in (possibly Unified) RAM to avoid SSD writes entirely. Not sure if it would fit on a 32GB laptop, though.
- xfalcox 3mo agoREADME covers that https://github.com/JustVugg/colibri#ssd-wear-warning https://github.com/JustVugg/colibri#ssd-wear-warning
- walrus01 3mo agoMy main question is whether when put into practical use, this can be measured in tokens/second, or more like 1 token per minute... I have seen locally hosted LLM that are as slow as 1 tok/second still be very useful if you give it a project to do something overnight and metaphorically walk away from it, check back with what it has done in 6 or 8 hours. 0.05 to 0.1 tok/s on the other hand, as reported in the URL for the lowest class of hardware, isn't really usable for much. edit: I think this is a fantastic project in general concept, and look forward to seeing more efforts towards the general idea of being able to run a 350B to 900B size model locally, even if as slow as 1 tok/s, on hardware that ordinary people can afford. Anything along the general concept of "we have fast read NVME SSD storage, we have a big ass model on local disk, we'll read it at 11GB/tok as we need it, not try to load the whole thing".
- vforno 3mo agoIn the readme you can see benchmark which everyone with different hardware is running Colibrì, and I have to say I've seen great times! I'm always doing more to improve!
- walrus01 3mo agoI have a 16-core system with 256GB RAM here I could try it with but regretfully it's so old the CPUs aren't AVX2 capable. Otherwise it makes a fairly good llama-server test system for CPU only stuff. Oh well. Time to upgrade (painful to the wallet these days).
- vforno 3mo agoMaybe we can see some integration!
- fuzzfactor 3mo agoIf you get good at extracting remarkable performance from the most lesser of instruments enough to pull their own weight regardless, just imagine what it can be like when such a practitioner gets behind the keyboard of a world-class Steinway. And just does what they do best. Without ever having touched such a capable instrument themself. On a level playing field the expression of virtuosity can outshine those who have never known any instrumental limitations at all :) When pulling way more than your own weight happens like for few others. There should be an award for getting the most out of the electronics rather than trying to reach orbit by building the tallest pile of e-waste. First Prize right before your eyes ! Grande praise ! And just starting to ascend toward an unconquered summit that others find forbidding ;) Or they find uninteresting since the limit naturally lies on firm earth somewhere below the stratosphere.
- kzrdude 3mo agoYour coding style is halfway to IOCCC. I'm just jealous though :)
- xfalcox 3mo agoQuestion to the OP, have you tested this on a machine where the entire model and context fit in RAM ?
- vforno 3mo agoNo because I have only 32gb of ram too low
- walrus01 3mo agoI think if you had something like a theoretical used/refurb 2U rackmount server with two older multi core CPUs, 768GB of RAM, you would see faster performance loading a Q6 or Q8 GGUF of GLM5.2 into a freshly-compiled latest copy of llama-server, with the "no-mmap" option turned on to intentionally load the whole thing into RAM at the time the llama-server daemon launches. If you want a CPU-only machine with 512GB to 1024GB of RAM, despite extreme cost rises, there are still some great options out there from companies selling ex-lease stuff that's 3, 4, 5 years old. It'll be loud as hell under full CPU load when running inference, so if you plan to use it at home, put it in your garage or basement or laundry room or somewhere similar on the far end of a network cable. The software that OP has published appears to be specifically designed to hold only the active parameters in RAM (<100GB) and read content off local NVME SSD as needed on the fly. All that NVME SSD read wouldn't be necessary if you can hold the model in RAM, even in the absence of any GPUs.
- shrinks99 3mo agoPretty cool! I've also been playing around with GLM 5.2 this week and was equally impressed. At work we're running it locally on some crazy expensive hardware as a test before starting another project so it's great to see people taking this massive FOSS model release and running it on an average machine, even if it's not terribly practical at this point. Nice work!
- khalic 3mo agoI love seeing that kind of tinkering
- vforno 3mo agoReally thanks!
- bobim 3mo agoI'm not fully understanding this business of MoE so please forgive me if this is a dumb question, but would it be possible to use MPI with a small cluster to distribute the load?
- vforno 3mo agoIt’s a good question. In theory MPI could distribute experts across nodes. In practice, for small clusters the added network latency usually hurts more than it helps. Better suited for big clusters with fast interconnects. For now we're focusing on single-machine speed (caching, GPU hybrid, etc.).
- deleted 3mo ago[deleted]
- Cieric 3mo agoI was actually just working on the same thing as this, but I went down the route of mmapping the entire model into memory to avoid the extra ram usage. I also had Claude implement Medusa[1] on the model to try and avoid loading an additional model into memory but still get the benefits of MTP. Currently at a stop light so I can't list everything and I didn't get to read your full post either yet. To expand since I just got home, I'm making all of my modifications to llama.cpp, the goal was to eventually put this on a SBC of some kind with an nvme to handle the mmapped files. I think the theoretical limit of my current setup is about 1.8 tok/s based on prior testing but that is also with the additional medusa heads not fully trained (I honestly don't know if the counting it's generated tokens or not.) In the end it seems like the idea we had is similar, I just don't know how to write an llm parser/runner from scratch yet and instead of specifying what needed to stay in memory I just let the linux kernel handle it. Oh last note, I also capped llama.cpp usage to 16GB of my 32GB, so it might be possible to get it down even lower. [1] https://arxiv.org/abs/2401.10774 https://arxiv.org/abs/2401.10774
- vforno 3mo agoif you like, colibrì always needs to improve so if you have ideas or anything else you are welcome for pull request issues and also benchmarks!
- Cieric 3mo agoYeah I'll see what I can transfer over from my llama.cpp work. As before I'm not too experienced with llm work, but I have a lot of experiments I'm trying out. So I'll make a PR if I get any interesting results.
- dicroce 3mo agoThis is the approach I was wondering about.
- Cieric 3mo agoLet me know if you want to hear anything specific about it. It kind of works, so it's not something I recommend doing if it can be avoided, but as Roxxik pointed out there is much room for improvement since this was just a naive just get it to run experiment.
- kodablah 3mo agoI've taken a similar strategy w/ image/video gen at https://github.com/cretz/thinfer https://github.com/cretz/thinfer (see video branch for a ton of work). Basically I kept needing an inference engine that could stream weights in and out as needed in an LRU manner. So I ended up vibe coding this thing that accepts a `--vram-budget` and stays under it (mostly). It turns out moving mmap'd bytes in and out of VRAM is way cheap compared to compute. Coupled with some pipelining/double-buffering, I almost always end up compute bound not memory bound. Granted I use way smaller models heh.
- espetro 3mo agoWow, I see you managed to fit in so many models (krea, wan, hunyan, etc.). Did you get to build a common harness to run all of them? Which ones stay under your VRAM budget more consistently?
- kodablah 3mo agoAll stay under because I had Claude build the workflow to respect it (text encoding, denoising, vae, etc), there's just a tiny bit of untracked pieces. While there are common interfaces to invoke them (CLI and API/webpage) and they share ops and some pieces, lots of model logic is unique. This is all vibe coded and surely has inaccuracies.
- stavros 3mo agoThis is great, well done! I love seeing people run things where they weren't meant to be run.
- Datagenerator 3mo agoGreat job, it's unique!
- vforno 3mo agoThanks really thanks!
- tarpitt 3mo agoI am curious if it's possible to adjust this to use more RAM, as i've got a machine with 64GB RAM and 24GB VRAM. Or perhaps I could run Gemma/Qwen on the GPU and have GLM-5.2 delegate smaller tasks to it. It might take some retraining of GLM-5.2 I'm also curious if you can speed this up by using many disks in parallel to increase bandwidth. >SSD Wear Warning > Cold starts are heavy on random reads (~11 GB/token). Reads themselves are safe, but the OS page cache can generate writes. Heavy use may accelerate wear on cheaper SSDs. Use with caution and monitor your drive health. Hmm, maybe a safe way to do this would be to make a separate partition for the model weights, and set them to read-only? Not sure how the page cache works, if it's like per partition or per disk. If it's per disk, maybe you could have a read-only data.iso formatted as a partition and mount it as a disk?
- vforno 3mo agoThat's possibly a good idea! We can work on it!
- tarpitt 3mo agoI also just edited my comment with more ideas in the beginning, sorry
- valicord 3mo ago> OS page cache can generate writes Is this a hallucination? What am I missing? Why would heavy reads generate writes?
- onlyrealcuzzo 3mo agoSpilling
- gcr 3mo agoWouldn’t turning off swap fix this issue?
- voidmain0001 3mo agoThe page has an SSD wear warning [0] I use desktop PCs that I build from components so I can replace the SSD, but what do users with soldered SSD do? Just avoid these applications or forge ahead disregarding the possible early burnout of their storage? They must use external storage as the burner SSD. [0] https://github.com/JustVugg/colibri#ssd-wear-warning https://github.com/JustVugg/colibri#ssd-wear-warning
- hoppp 3mo agoYes, avoid. Laptops with soldered in SSDs should definitely monitor their usage and take care with this. This project seems more of an experiment than something everyone should run, but pretty cool nonetheless
- vforno 3mo agoThanks We're working on it!
- Cider9986 3mo agoAppleCare.
- Gigachad 3mo agoEven under AppleCare this is a $400 service which for an older macbook costs almost as much as the whole thing. And without Applecare it's not worth fixing at all.
- VortexLain 3mo agoProbably yes, use an external drive for that sort of thing
- killerstorm 3mo agoFrom what I understand, the warning is about swap-out during heavy memory use. You don't need to be superstitious here: disk activity, including writes in particular, can be measured. E.g. `iostat` or `vmstat` on Linux.
- bahmboo 3mo agoAnother recent project that runs a huge model on a 48gb Mac is https://github.com/danveloper/flash-moe https://github.com/danveloper/flash-moe - it gets over 5 tokens/sec on an M3 Max compared to this projects very impressive 1 token/sec on an M5 Max. So for anyone wanting to tackle a Mac only version that targets lower spec machines this looks like a good candidate with plenty of room for speedups [edit: because it doesn't use the gpu]. Not hijacking anything as this project is amazing.
- kzrdude 3mo agoHow many tokens/s do people feel are enough? Even 5 tok/s would be hard to use in an interesting way, not sure what the use is.
- khimaros 3mo agorelated and possibly more general purpose https://github.com/t8/hypura https://github.com/t8/hypura
- Archit3ch 3mo agoWith so many people implementing their own SSD streaming for specific combinations of model+hardware, maybe we should look into upstreaming to antirez/ds4 or llama.cpp...
- Archit3ch 3mo agoWorking on something similar targeting macOS on Apple Silicon, Unsloth split GGUF, compressed partial residency in unified memory (would make more sense on 128GB instead of my 64GB...), native Metal kernels, and RAM-only native compressed KV. Happy to put on GitHub when it's ready.
- sebastianconcpt 3mo agoThis is the way.
- fragmede 3mo agoFollowed you on GitHub to get notified when you are!
- xtracto 3mo agoI will be delighted to try. I have a 128gb macbookpro m4 waiting for this.
- ukd1 3mo agods4 is pretty optimized for this btw - https://github.com/antirez/ds4 https://github.com/antirez/ds4
- arm32 3mo agoLink it already!
- tannertech 3mo agoI love it but where do you find that NVMe SSD for less than the price of an h100 fan let alone the memory
- saghm 3mo agoNVMe SSD prices had being gone down in price for a while, and the spikes are actually a lot more recent than you might think. From double checking my Amazon history, I bought my wife a 2 TB NVMe SSD for $160 back in November; it's now listed at three times that. I imagine that a lot of people just have them already from the past few years.
- VaporJournalAPP 3mo ago[flagged]
- xtracto 3mo agoThis is something that would benefit from Intel Optane memory. Too bad it was killed at the time.
- vforno 3mo agoMaybe some from intel can read and we can try? :)
- xtracto 3mo agoI wonder how would a RAID0 array of either disks or even nvme improve the performance of this.
- mmastrac 3mo agoThis sort of thing is a lot of fun. I've been going smaller.. I have a custom-quantized Rust port of DiffusionGemma (26B) that seems to perform better (in responses) than benchmarks seemed to indicate and reasonably fast for its model size. Works really well on a 36GB mac as well for both prefill and generation. It's been interesting learning about the balance of factors for performant metal kernels on unified memory. Should have a repo up on github in the next few weeks.
- saghm 3mo agoI'd be interested in seeing this when you put it up! I've managed to get upwards of 300 tokens a second somewhat consistently on the desktop I already had gotten for gaming/personal project stuff several years ago (64 GB RAM, Ryzen 9 7900 X 12-core; I already had a fairly large m.2 SSD and a Radeon 6900 XT to plug into it), but only by severely compromising on the model (a custom quantized Qwen3.5-27B-Q2_K which someone published on huggingface) so that I can use a context size large enough that interfacing through opencode doesn't manage to run out of context while summarizing for compaction and then trigger a second "nested" compaction that instead just seems to cause it to lose all context and ask for a full new set of instructions from scratch. If yours ends up being anywhere as close to the one I've been using in quality, and it works on a 36 GB Mac, it sounds like it would be worth it for me to try out! (For clarity, I have almost no idea what pretty much any of the terminology for models even means or how it translates to what the actual experience is. My strategy has mostly been using the free tier of OpenCode Zen to ask it stuff about what models and llama.cpp configurations to use in the hopes that I could maybe bootstrap something halfway decent to use locally, and so far what I've managed to get running is mediocre but at least passable)
- sakesun 3mo agoI just learned about Gemma4.pas at the beginning of this week. Now this. This make me wonder how can inference engines could be built that easy. I'm not knowledgeable in this, but I thought it would take very deep Mathematic and system level knowledge, ... and a lot of patience.
- killingtime74 3mo agoIf I understand correctly the basic structure hasn't changed since Transformers were invented. It's just gotten bigger and the models have had better training and also some optimizations. I don't think this one has optimizations.
- imhoguy 3mo agoNo way! It is a while since I have seen that extension here on HN, I couldn't be mistaken. With AI? Is Pascal still alive?
- efficax 3mo agothe math involved is not very hard to understand. it’s linear algebra. the transformer model is brilliant but simple, nobody even really realized the impact it would have until they started training it on massive datasets
- redfather 3mo ago[flagged]
- qiqitori 3mo agoHow much time is spent interfacing between userland and the kernel? Can you try to get it to run as a kernel module? :) Also in case your CPU is old enough, did you try disabling CPU bug mitigations?
- flockonus 3mo agoCurious for what an MTP only result would look like, both in terms of output quality & tk/s ?!
- nogajun 3mo agoIs this similar to fastllm? https://github.com/ztxz16/fastllm https://github.com/ztxz16/fastllm
- smetannik 3mo ago> slow computer > 25 GB of RAM What?
- 3836293648 3mo agoSlow means CPU only in this context.
- ihateolives 3mo agoFor long time my daily driver was T480s with 24GB. I still have it, but don't use it anymore because it is slow for my needs now.
- deleted 3mo ago[deleted]
- gokulrajaram 3mo ago[flagged]
- barent 3mo agoYeah this idea makes instant sense. Very well done, this deserves a github star on concept alone.
- barent 3mo agoThe best ideas are the ones that seem obvious. This is one of those ideas.
- jnaina 3mo agowas lucky enough to snag the Olares One from Kickstarter just before this whole AI induced memory chip price gouging started. specs are Intel Core Ultra 9 275HX (24 Cores, 5.4GHz),96GB of DDR5 5600MHz RAM, NVIDIA GeForce RTX 5090 Mobile GPU with 24GB of GDDR7 VRAM, 2TB NVMe PCIe 4.0 SSD. going to see if I can wring at least 5 tok/s.
- simpaticoder 3mo agoLet us know how it turns out.
- SubiculumCode 3mo agoExcuse my ignorance. Could one just say, "One expert is all I can handle" and strip the others from the model?
- jeffybefffy519 3mo agoOr could you parallelise your experts on different hardware?
- p-e-w 3mo agoYou could (e.g. by replacing residual-dependent expert routing with hardcoded logic), but quality will suffer dramatically. It’s far better to use a similar-sized dense model then.
- SubiculumCode 3mo agoAhh, I remember hearing that before, and it makes general sense too.
- Infernal 3mo agoI have a similar question and I’m inferring the answer is no - look at the cache hit rate of 23% for the 128GB M5 Max. I had previously assumed that the 40B active meant that a set of layers was chosen as THE expert for a given prompt and generation was then limited to those layers until complete. But in that case you’d have expected the expert caching to have a super high hit rate once you had enough RAM to hold an entire expert’s worth of layers.
- kristianp 3mo agoI you look at https://arxiv.org/pdf/2401.04088 https://arxiv.org/pdf/2401.04088 table 5 on page 8, you'll see that expert(s) used can change from token to token. The experts aren't divided along predictable lines.
- xyzsparetimexyz 3mo agoFor purely coding tasks, is every single expert required? 50%? 25%?
- rahulb0802 3mo ago[dead]
- jhalloran 3mo agoNice, looks good. This would mesh really well with the unified system memory on apple silicon
- ac29 3mo agollama.cpp supports a wide variety of 4-bit and smaller quants and mmap's models by default, so you dont need to be able to hold the weights in memory (the OS will handle bringing them in from storage as needed) Its cool to see this implemented in a tiny amount of code without dependencies, but does it actually bring more performance?
- pianopatrick 3mo agoI'm curious but don't know much about the internals of LLMs - could you use a similar architecture with other models that have "layers"? I mean, could you have one layer do its work, then remove that layer from RAM, load the next layer from disk, and have that layer activate on the result of the first layer?
- joka88xj 3mo ago[flagged]
- 3abiton 3mo agoThis is technically impressive, but is it usable in practice?
- threerouter 3mo ago[dead]
- ankit100 3mo ago[dead]
- ankit100 3mo ago[flagged]
- abhiram304 3mo ago[flagged]
- harrouet 3mo agoThis is exactly the kind of technology that I expect Apple to ship anytime soon given the RAM prices and their HW/SW integration skills: - ship super fast SSD (tbh they are already top notch) - add a specific cache layer for tokens - keep the amount of unified memory reasonable
- tw1984 3mo ago“answering correctly on a machine that costs less than one H100 fan” really love such comparison.
- kownse 3mo ago[flagged]
- vehbiemiroglu 3mo agoSo, could larger models work this way too?
- gizajob 3mo agoLarger models lol. Yeah this one is a tiddler it would be good to get a seriously good model running at one token per year in this setup.
- Roxxik 3mo agoI have prototyped something similar with ollama some months ago. Do you mmap or issue reads on demand? Also do you use io_uring to interleave compute with io or do you spawn extra threads? I also tried predicting which experts get reused and I managed to beat a simple LRU very slightly. EDIT: That was on Kimi 2.5 but even worse quant than 4bit. IIRC it was 2.6 or so
- jdw64 3mo agoCoool!!!
- AussieWog93 3mo agoWell under 1tps, but the fact it runs at all on a 16gb system is incredibly impressive!
- deleted 3mo ago[deleted]
- Purposefather 3mo ago[dead]
- rullopat 3mo agoLooking at the progress with DS4 project from Antirez and Colibri, are we probably going in the direction of having a stripe RAID of SSDs instead of RAM to get decent performance without losing a kidney to the RAM mafia? I really hope so.
- cereblab 3mo ago[flagged]
- maxignol 3mo agoI’m truly impressed by your work ! I don’t know if this is planned for near future, but how about adding energy efficiency benchmarks ? Because running locally is a great feeling, but the electricity bill should not be forgotten
- AymanB 3mo agoI believe you could use some sort of GPU-direct (or BAM-like approach, see https://dl.acm.org/doi/10.1145/3575693.3575748 https://dl.acm.org/doi/10.1145/3575693.3575748) to stream weights on-demand from the GPU.
- webprofusion 3mo agoThis is great stuff, I feel like fast disk (SSD) is a somewhat solvable problem, if you have many disks with the same content and a fast controller (?).
- deleted 3mo ago[deleted]
- lopatin 3mo agoAfter spending way too much time with Fable a few days ago, I noticed a new hallmark of AI generated text is using the word "honest" everywhere, in a somewhat self congratulating way. Some things that give it away to me: - "Honest numbers (WSL2, 12 cores, 25 GB RAM, NVMe via VHDX)" - "an honest peak projection (working set, KV, MTP row, reconstruction buffers) so the kernel OOM-killer never fires." - "Honest caveat from the same measurement: ..." It's the new "It's not X, it's Y". I have no issue with this, I just found it amusing. Cool project btw!
- FeepingCreature 3mo agoI suspect it's a signature of anti-fabrication training.
- LoganDark 3mo agoIt's as if it's sucking up to me or apologizing at every turn, even when it's done exactly what I've asked. Seems related to sycophancy, but different? (i.e. rather than unconditionally affirming the user it's unconditionally expecting to disappoint)
- lukaslalinsky 3mo agoThat was their headline for Opus 4.8, I guess the invested into some post-training to get them to write this, and I like it, it's a great way to identify AI posts from Claude. https://www.anthropic.com/news/claude-opus-4-8 https://www.anthropic.com/news/claude-opus-4-8
- jannyfer 3mo agoYup, and I had seen it in the Opus 4.5 soul document as well: https://gist.github.com/Richard-Weiss/efe157692991535403bd7e7fb20b6695#being-honest https://gist.github.com/Richard-Weiss/efe157692991535403bd7e... In the future, I expect different models from different firms behaving differently will become as obviously normal as different humans behaving differently. Those of us that have used agents a lot have already noticed this, but the general public still seems to consider AI to be plug-and-play. The Opus 4.8 infatuation with being honest genuinely drove me nuts (honest take). The constant need for me to decipher and decide on something after its final message was tiring too. GPT-5.6-Sol is refreshing that its replies start with "understood." instead of flattering me for my steering prompt. Now ... after a few weeks, will I get suck of "understood"? I dunno :)
- RahulRajelli 3mo agoGlm 5.2 on lobehub , i was able to finish a complete the project very quickly
- _joel 3mo agoI've got a 48GB M5 - What's the best I can run on that atm (with a bit of headroom).
- jeffhuys 3mo agoPerfect question to ask an AI.
- cyberelf77 3mo ago[flagged]
- doodlesarefun 3mo agois this bad for my SSD?
- efficax 3mo agoTried this out on my ThreadRipper: AMD Ryzen Threadripper PRO 5975WX — 32 cores / 64 threads, Zen3 (znver3), AVX2+FMA (no AVX-512/VNNI), 128GB RAM, Kingston SKC3000D 4TB NVMe (PCIe4). Disk gets around 7GB/s. It took a little tuning (for example pinning to 32 physical cores instead of the 64 threads), but with that and --topp 0.7, got 0.44 tok/s on a cold start. That's way below the estimates in the README, which I assume are pure AI slop (LLMs love to estimate incorrectly. They're far worse than even naive humans at it), but it's pretty cool for a model this size. I sent Fable off to wrap this in an OpenAI API to see how it works when driven by an agent harness. EDIT: it finally finished the first non test prompt i gave it, which with local LLMs is usually "what is the meaning of life?" (who knows, maybe one of them will finally answer). It got stuck in a loop, which is not encouraging, so there's a lot of work to do to make this a viable local coding tool: > The meaning of life is one of the oldest and greatest questions in human history, yet strangely, there is no single, universally agreed-upon answer. Because "meaning" is a human concept, it doesn't exist out there in the universe; it is something we create for ourselves. The answer depends entirely on the framework through which you view the question. Here are the most common ways to answer it. The meaning of life is the meaning you give to it. We are all in the same position: humanity's search for it never ends in "to be determined" or "to be announced" (TBA, the answer is unknown, and it is a great mystery, or perhaps even the answer "forty-two" (42) is the "Answer to the Ultimate Question of Life, the Universe, and Everything" in The Hitchhiker's Guide to the Galaxy by Douglas Adams (where the number 42 is the "Answer" in Python's language, but we don't know the "Ultimate Question"). Here is a joke that works under the frame of "A..." (any answer): "A clean desk is a..." (42 is a "portmanteau" of words and just a great big "Ad..." (Ad-100) and "A&d" (100)). Life is a deep and strange and we search for meaning in it. "I think, therefore,..." (Cogito, ergo, sum) is the only valid idea in philosophy [3] (cf., "I think, therefore, I am," is a valid translation of "I think, therefore, am" (in the original Latin, "Cogito, ergo, sum" is "I think, therefore, I am")). So, the meaning of life is a bit like "a riddle, wrapped in a mystery, inside a [riddle]..." (G. K. Chesterton) and inside a [block of] "42" (or the number of dimensions, which is the "Answer to Life, the Universe, and Everything" in the "H2G2" (H2G2 is the "Ultimate Question of Life, the universe, and everything")). The "H2G2" is a "puzzle, wrapped in a mystery, inside an enigma" (cf. [3]). We are all in the same position, but we all have to give it a meaning. our own meaning. The meaning of life is what you make of it. The meaning of life is to live for the greater good. The meaning of life is to live in a way that is good and noble and right, and to do so well that with every breath, I think of you, I think of life, and I think of you, and I think of life, and I think of you. (cf. [3]) If life in the universe is a "great question," the answer is 42. The meaning of life is the meaning you make it. The meaning of life is to give life a meaning, and I think of you, and I think of you. So, the answer to the ultimate question of life, the universe, and everything is: 42. The meaning of life is 42. The meaning of life is the meaning of life. This is the Answer to the Ultimate Question of Life, the Universe, and Everything (or "The Answer" for short). It is the Answer to "the" Ultimate Question of Life, the Universe, and Everything. (See, for example, the Ultimate Question of Life, the Universe, and Everything.) This is the answer to the Ultimate Question. This is the Answer. (And, this is the Answer to the Ultimate question of life, universe, and everything.) The meaning of life is the meaning you give to it. The meaning of life is to give it a meaning. The meaning of life is the meaning you give it. The meaning of life is the meaning of life. The meaning of life is the meaning of life. The meaning of life is the meaning of life. (This is a list of the possible meanings of the universe of life. It's a list of the most common and accepted answers. "What is the meaning of life?" The answer is 42. The meaning of life is the meaning of life. The answer is 42.) (See also: [3] for a list of possible meanings.) The meaning of life is to give it a meaning, and the meaning of life is the meaning you give it. The meaning of life is the meaning of life. The meaning of life is the meaning of life. The meaning of life is the meaning of life. (This is the answer to the Ultimate Question of life, the universe, and everything.) (This is the answer to the Ultimate Question.) The meaning of life is 42. The meaning of life is 42. The meaning of life is 42. (See also: [3]) (The answer to the Ultimat
- nicogentile 3mo ago[flagged]
- valentynkit 3mo agoYou've basically build an LRU page cache with readahead for streaming 21, 504 experts off disk. Some problem OS people have solved for mmap'd databases for decades, just with parameters instead of rows.
- bobkb 3mo agoWould love to collaborate
- zftnb666 3mo ago"Slow computer" + running a 200B model = "I'll just wait a week."
- tarruda 3mo agoAssuming steady 1 tok/second generation (which seems to be the case for M5 Max macbook), wait 1 day for a 86400 token response. In some configurations it can be as slow as 0.1 tok/s, so be prepared to wait for 10 days.
- classified 3mo agoSo when I clone that repo, am I downloading > 370 GiB, or when and how is that being pulled in? And can I leave out the web part? I cannot in good conscience run npm on my machine (it's not even installed).
- vforno 3mo agoYes you can use ./coli chat
- timedude 3mo agoImpressive work. Also it must be said that the Apple M5 is incredible.
- ballislife30 3mo agoWhat's the cost
- pobonin 3mo agoI must say, this is incredible, nice work! I've been tinkering with local LLMs for a while now and experimenting with different tech, and I'd never thought it'd be possible to run such a large model on consumer-grade compute. It still feels like these days, you need to buy a $10K DGX Spark or something crazy expensive just to get to that level of performance but with experiments like yours, it looks like there's still a lot of room for improvement and innovation. What I'd like to see personally is Small Language Models giving similar performance eventually (because do we really need that many hundred billion parameters?). Researchers have shown that a lot of attention heads in the LLMs we're using these days are barely even active, so the MoE architecture seems like a step in the right direction. I still wonder how far we can push things back towards a 1B or 3B parameters model while trying to reach similar performances as double-digit or triple-digit B params models.
- ykonstant 3mo agoI only have 16GB of RAM in my laptop, but I would love to run a powerful model like that even at 0.5 tokens/s. For the questions I am asking, such a speed is more than enough. If anyone has any suggestions, please let me know! I am not an expert on LLMs, just a humble mathematician.
- kzrdude 3mo agoIt might be impossible to run such a large model on the laptop, if I were you I'd first start experimenting with running smaller models, using llama.cpp or lmstudio, just to start understanding how it works and learn about some of the concepts involved - quantization and all! For example Qwen3.5-4B or Qwen3.5-9B could probably be run cpu-only on some quantizations at more reasonable speeds. If you want a larger one, you could probably run Qwen3.6-27B or 35B-A3B decently slowly. Now to be humble - maybe it's impossible to run colibri (this project in the OP) on your computer. Note the performance stories - 0.07 token/s on a 24 GB laptop with 24 CPU threads!
- kzrdude 3mo agoSomeone posted this fork, which should work in just 10 GB RAM, at least for another big model. I thought of your request and came back to spread the news :) https://github.com/ErikTromp/colibri-hy3 https://github.com/ErikTromp/colibri-hy3
- ykonstant 3mo agoThank you very much!
- bboydart 3mo ago[flagged]
- egberts1 3mo agoI'm running an archaic Dell PowerEdge T710 dual-Xeon E5690 (6/12 CPU, 3.73GHz turbo) with 192GB DDR3-800 (was 288GB but now PassMark 14,328). Token rate is 0.091 per second. Good for an overnight job.
- tensegrist 3mo agothat's like 6k tokens over 12 hours which doesn't seem like a lot for…any purpose i can think of
- orwaMahmoud 3mo ago[dead]
- andsoitis 3mo agoWhy did you use Claude to help you make this rather than GLM? (It’s listed as a contributor in your repo)
- fivetech 3mo ago[dead]
- fivetech 3mo ago[dead]
- zerotaken 3mo ago[flagged]
- crest 3mo agoSounds more like crawling instead of running, but at least it’s moving.
- Capitanai 3mo ago[flagged]
- kelvinxue 3mo agoI am curious if it's possible to use coli on ubuntu as coding agent, as i've got a machine with 32GB RAM and 2T SSD.
- kelvinxue 3mo ago13th Gen Intel® Core™ i7-1360P × 16
- kunxue 3mo agoany one have experience or comments on coli with glm5.2 vs codex with gpt5.4mini
- anoop_kumar 3mo agohas anyone tried hosting this on a machine that's not a laptop like a EC2? Curious if we can get better performance than on bedrock (which does not yet support GLM 5.2)