4 ms·
My main question is whether when put into practical use, this can be measured in tokens/second, or more like 1 token per minute... I have seen locally hosted
by walrus01 4mo ago
My main question is whether when put into practical use, this can be measured in tokens/second, or more like 1 token per minute... I have seen locally hosted LLM that are as slow as 1 tok/second still be very useful if you give it a project to do something overnight and metaphorically walk away from it, check back with what it has done in 6 or 8 hours.
0.05 to 0.1 tok/s on the other hand, as reported in the URL for the lowest class of hardware, isn't really usable for much.
edit: I think this is a fantastic project in general concept, and look forward to seeing more efforts towards the general idea of being able to run a 350B to 900B size model locally, even if as slow as 1 tok/s, on hardware that ordinary people can afford. Anything along the general concept of "we have fast read NVME SSD storage, we have a big ass model on local disk, we'll read it at 11GB/tok as we need it, not try to load the whole thing".
- vforno 4mo agoIn the readme you can see benchmark which everyone with different hardware is running Colibrì, and I have to say I've seen great times! I'm always doing more to improve!
- walrus01 4mo agoI have a 16-core system with 256GB RAM here I could try it with but regretfully it's so old the CPUs aren't AVX2 capable. Otherwise it makes a fairly good llama-server test system for CPU only stuff. Oh well. Time to upgrade (painful to the wallet these days).
- vforno 4mo agoMaybe we can see some integration!
- fuzzfactor 4mo agoIf you get good at extracting remarkable performance from the most lesser of instruments enough to pull their own weight regardless, just imagine what it can be like when such a practitioner gets behind the keyboard of a world-class Steinway. And just does what they do best. Without ever having touched such a capable instrument themself. On a level playing field the expression of virtuosity can outshine those who have never known any instrumental limitations at all :) When pulling way more than your own weight happens like for few others. There should be an award for getting the most out of the electronics rather than trying to reach orbit by building the tallest pile of e-waste. First Prize right before your eyes ! Grande praise ! And just starting to ascend toward an unconquered summit that others find forbidding ;) Or they find uninteresting since the limit naturally lies on firm earth somewhere below the stratosphere.
- vforno 4mo agoThanks for kind words!
- ProofHouse 4mo agoAgreed!
- joshsantiago01 4mo ago[flagged]
- charcircuit 4mo agoFor most projects the more practical solution is to use clouds offering GLM 5.2 for free. 1 token per minute is minuscule compared to their rate limits for free usage.
- bigiain 4mo ago> on hardware that ordinary people can afford These days, can "ordinary people" afford 24GB of ram and half a TB of NVME ssd? sigh
- walrus01 4mo agoThe very boring pair of two 16GB ddr5 6000 I had in my newegg shopping cart went from $399 to $475, so increasingly the answer will be "no".
- fuzzfactor 4mo agoMaybe that's a measure of the self-fulfilling dollar incentive toward "renting" someone else's RAM in the future rather than trying to actually own such an outlandishly luxury item :\
- bigiain 4mo agoI bought whole Intel N100 mini pc with 16GB of DDR5 in it in 2023 for $AUD289 (so about $US200). I got a 16GB (DDR4) SODIMM in 2022 for $AUD88 ($US60).
- fwipsy 4mo agoDoes it have to be DDR5? Is the limit RAM speed, or SSD speed?
- walrus01 4mo agoI was just using that as an example of constant on going price rises, it was the most mundane and not particularly fast ddr5 6000 stuff. The 6400 is even more ridiculous.
- fuzzfactor 4mo agoIdeally this engineer's approach will yield better performance on lesser equipment in the future, if they keep up the good work after they get more-capable gear to experiment with as time goes by :)
- 4mo ago
- codazoda 4mo agoI’ve been wondering if chat is the wrong interface for slower local models (and some projects) and maybe something like a ticket system is a better fit. I just decided how I would test this idea on my available hardware before I go drop money on a Mac Studio or GPUs. I’ll probably have a POC this week. There is nothing novel here, just need to spend the time to get it working for me.
- alexhans 4mo agoHaving a thin python/ts orchestrator and workers that pick up tasks from the directories like events and decide whether to make deterministic calls and wait is pretty standard albeit custom way of doing things in this space where you're bottlenecked by the concurrent call your workers/agents can make. The hard thing is always keeping complexity low and being ZeroOps.
- antondd 4mo agoAre there any frameworks/scaffolding/harnesses or general resources on this you can share? I’d love to learn more
- walrus01 4mo agoMost any ticketing system can integrate with ordinary IMAP and smtp email flow, so you can really use any agent that can "do" inbound and outbound email to talk to a self hosted ticket queue.
- stavros 4mo agoI no longer use a harness directly, instead I use Github issues/Linear to work on multiple tickets in parallel while the agents are doing work: https://github.com/skorokithakis/symphony https://github.com/skorokithakis/symphony
- digitaltrees 4mo agoThis is actually really smart. It would be like working with a team of humans. I have a 3 Mac Studio set up and built an IDE / harness (propelcode.app) and would be interested in contributing if you’re open to collaboration
- JumpCrisscross 4mo agoThe funny thing is Claude Cowork has taught me to be patient with response timelines. I’m now figuring I’ll be running locally no later than 2028. (I want to spend no more than $10k. And I want to run a model comparable to today’s SOTA.)
- paytonjjones 4mo agoToday's SOTA also sounds totally sufficient to me, but I wonder how much our standards will inflate by 2028. Maybe a lot, maybe not at all...very hard to say.
- anon373839 4mo agoThis seems to vary by person. I get immense value in coding assistance from Qwen 3.6 35B-A3B which is like a frontier model from a year ago. But a lot of people say it’s stupid, useless, a toy, etc. I do work by the “short leash” method and mainly just use the model for brainstorming/planning/design assistance and zipping through the drudgery of boilerplate and executing refactors. I don’t think this tier of model is good for “hey LLM, build me a Github clone” ... but I also don’t see the value in that use anyway.
- 3836293648 4mo agoCould you expand more on what you do with qwen3.6? Because I couldn't get the denser 27B version to do trivial "take this pattern, repeat it over a single file with minimal thought, just slightly beyond what I can do with sed" reliably.
- kgeist 4mo agoHow was qwen3.6 launched? The thing is, everyone has their own variant of "qwen3.6 27b" depending on the launch parameters, ranging from "SOTA in its class" to "completely broken"
- 3836293648 4mo ago
- zozbot234 4mo ago0.05 to 0.1 per sec could still be quite useful if it was the speed for inferring a whole batch of tokens concurrently. Of course this actually requires fairly good SSD read performance (since you need to read a sizeable fraction of the complete model at every token batch in order to get good reuse) and is ultimately limited by CPU/GPU thermals which are a tight constraint on typical inference platforms. It's also only really feasible with tiny KV caches, which requires either a very small context or sticking to KV-cache efficient models such as the DeepSeek V4 series. Still, this might be one way of making use of existing lower-end hardware for practical inference of non-tiny models.
- adrian_b 4mo agoNow many mini-PCs and desktops are able to read simultaneously from 1 PCIe 5.0 SSD and 1 PCIe 4.0 SSD. This can ensure a reading throughput around 20 GB/s, i.e. 20 times faster than on author's system. With only 1 PCIe 5.0 SSD, the reading throughput is still significantly more than 10 times faster than on author's system. So it is likely that inference speeds around 1 token/s are achievable on something like a NUC mini-PC.
- walrus01 4mo agoI'm curious, where are you seeing M.2 2230 to M.2 2280 size NVME SSE that exceed 4.5 to 5GB/s sequential reads for large files such as a GGUF (likefrom an ordinary ext4fs file system with default options)? The PCI-E 4.0 or 5.0 bus they're attached to might be capable of greater speeds, but the bottleneck is the flash and the flash controller. The benchmarks I'm seeing for many of them don't really make me think that a pair of consumer grade NVME SSD you could fit in a mini-PC or mini-itx size desktop would, added together, be capable of 20GB/s reads.
- adrian_b 4mo agoI do not think that I have ever seen any benchmark for a PCIe 5.0 SSD that did not have sequential read speeds well over 10 GB/s. If there were such a slow SSD, it would not make sense to buy it instead of a cheaper PCIe 4.0 SSD. For PCIe 4.0 SSDs, I have seen a very large number of benchmarks where the SSDs achieved read speeds close to the theoretical limit, i.e. around 7 GB/s. Searching now randomly for recent SSD reviews, I find many reviews for "SanDisk WD_BLACK SN8100", which achieve between 13 GB/s and 15 GB/s reading speed, which is better than most other consumer PCIe 5.0 SSDs. Of course, if you write a very simple program that invokes something like "fread" or "read" in a loop, you will not reach such speeds. Achieving a SSD throughput close to the limit requires a more complex program that can ensure that the SSD controller is permanently busy with pipelined read commands.