5 ms·
AirLLM 70B inference with single 4GB GPU
- cpfohl 2mo agoI’m still slightly confused on what this adds. Let’s say I wanted to run a full size open weight model. I have a 128GB m3 max laptop. Does this basically load layers in and out on demand? So I still have to download the full model to disk, but the RAM requirements go way down? The readme calls out that one still needs to connect HuggingFace, which leads me to believe that maybe you don’t even need to download the full model?
- dofm 2mo agoIf you point it at a huggingface model identifier, it will download it, I assume. No way around that. It reads like it is keeping only the core and the active layer loaded at any one point, and streams layers from disk; there are several other solutions like this and if my understanding is right, this is probably better than an mmap implementation or just streaming experts in.
- pvtmert 2mo agoNot an expert in this field, but the "expert" is consisting of multiple layers. To keep it small in terms of memory print, this project streams each layer (dividing even further). It also requires extra space because of decomposition of the layers. Normally the file format optimized for compute intense workloads. But here the bottleneck is the memory capacity. Also guessing that you need to be able to hold at least 3-layers at once in the memory, given M x N = R operation, M is the previous layer, N is next, and R is the result. on the next "layer", the R (result) becomes M, gets computed against the next layer, N, yielding the further result R'. And so on, until all layers are processed. I assume it's horribly slow, but can be put in a non-intrusive background task...
- dofm 2mo agoThanks for the extra detail here.
- cpfohl 2mo agoObviously needs downloading eventually :). It seems like this tool saves on both disk space and RAM, then. Classic trade off: speed vs space.
- jedbrooke 2mo agoI mean, if you had fast enough internet connection, you could just stream it over https
- hna8hjbqzy 2mo ago[dead]
- imenani 2mo agoFor anyone wondering “how slow is this?” IIUC, Kimi K3 on RTX 6000 Ada (48GB) takes 292 s/token https://github.com/lyogavin/airllm/releases/tag/v3.1.0 https://github.com/lyogavin/airllm/releases/tag/v3.1.0
- jackb4040 2mo agoAhaha thank you, I naively assumed the unlabeled graph in the readme was tps, not spt!
- throwawayffffas 2mo agoRunning 292 tps on K3 would make your gpu a money printer.
- meneton 2mo agospt not tps
- bensyverson 2mo agoI do think these “run a bigger model than will fit in VRAM” projects are necessary steps, but are they functionally useful or helpful to anyone currently? For example, is anyone out there running a big Qwen for coding on a 16-32GB machine with these techniques?
- pizza234 2mo ago> but are they functionally useful or helpful to anyone currently? Yes and no, depend on your expectations. Some/many like to run local LLMs just for the sake of it, so anything will do. MoE are useful on PC systems, at the condition of having high enough memory bandwidth (and large amounts of RAM) - that is, Threadripper/Pro. The advantage of MoE is that only a subset of the model's experts is used for each token, so not all weights need to be present in VRAM at once. The remaining weights can reside in system RAM, although moving and accessing them still carries a substantial performance cost (and that's why high memory bandwidth is needed).
- roger_ 2mo agoSeeing a lot of these “run 1TB models with 1GB RAM” projects recently. Most seem vibe coded and probably won’t be maintained. Hoping a winner emerges with some real momentum behind it.
- hiramwen 2mo agoYou don't really need a maintainer when codex or claude code can set it up for you; thats how I got Trellis2 working on windows and tiny VRAM despite Microsoft recommending you have 24GB VRAM and Linux. Models are pretty disposable now.
- titularcomment 2mo agoThis is complete dependency on LLM tools all the way from development to usage, and is risky as well as prone to failure
- hiramwen 2mo agoLocal inference without access to internet and tools is pretty safe, worst you get is a slop output, best case it solves your prompt.
- tecleandor 2mo agoSo, instead of having one or two maintained projects that work super well, now we have hundreds or thousands of people half-assing it locally each time they need it? Doesn't sound efficient.
- esafak 2mo agoI have the same objection to forking open source projects; changes should be upstreamed when possible.
- IAmGraydon 2mo agoIt might make sense to do this if there literally isn't an existing solution out there, but I think most would find that it's better to use an existing solution than to recreate the wheel every time there's a need. It's just less effort/time/money.
- ilaksh 2mo agoI guess the use case is something like: you have a slightly obsolete Mac or PC or a whole bunch of them, and just need to compose one or more convincing spam emails, but it's fine if it takes a full week to do it?
- speedgoose 2mo agoAnd you don’t pay for the electricity.
- book_mike 2mo agoWe will see if this project has legs. This is the kind of efficiency we desperately need. Now if we can address efficiency with llm training.
- myshapeprotocol 2mo ago[flagged]
- mrwaip 2mo agoWhat device do I need and how much will it cost to install one at home so that it works as quickly as the Claude Code answer (and it answers quite slowly)?
- SwellJoe 2mo agoIf you want it to respond in under a minute, you need more hardware than this application is intended for. This thing's response is measured in seconds per token, not tokens per second. To get Claude Code responsiveness from even a pretty small (but still usable) model, you need, maybe two 32GB GPUs? I run Gemma 4 31B and Qwen 3.6 27B on my dual 32GB GPU setup (cheap old Radeon Pro V620 GPUs) at about 20 t/s, which is not fast enough for comfortable interactive agentic use. A couple of new GPUs, like Radeon AI Pro 9700 at $1400 each, probably gets you fast enough for comfortable interactive use with small models like those. Those small models are not competitive with Claude models, however (maybe they beat Haiku sometimes). They can write a little Python or make a web page, they can't architect a real application. To run a near-frontier model like Kimi K3 or GLM 5.2 at comfortable speeds, you need serious hardware with 768GB VRAM, minimum. I think Asus is releasing something like that for about $150k soon. You can run DeepSeek V4 Flash at almost comfortable speeds and in a decently capable quantization on two DGX Sparks or Asus GX10s (about $10,000). Or, you could use DeepSeek V4 Flash from DeepSeek.com, at blistering speeds and with huge contexts, for something like a decade or two for that same $10,000.
- f6v 2mo ago> two DGX Sparks or Asus GX10s (about $10,000) > for something like a decade or two for that same $10,000 I've been using DS V4 Flash through OpenCode and it's mind-blowing how I get near-SOTA AI model for the price of two coffees. But let's put the broken economy of AI APIs aside for a second. Running such a model with under 6 figures of fixed costs is equally mind-blowing. And we know that hardware costs are real, Nvidia is selling those at a profit. I'm not talking about consumers, but businesses. I remember paying much more for something more trivial, like Datadog.
- SwellJoe 2mo ago
- xg15 2mo agoYou can run any frontier model on your PC if you just wait long enough...
- seu 2mo agoI love it how the rampocalypse is pushing people to squash all the performance they can. I hope that this also leads to rethinking model architecture so that they can do the same with less.
- bhawika_kaushik 2mo ago[flagged]
- hatthew 2mo agoI've been seeing a lot of projects that let one use large models on machines with small amounts of memory. They seem to all be doing significant quantization and/or expert streaming. What's the benefit of these projects over something like taking an unsloth quant and running llama.cpp with appropriate flags (-cmoe/-mmap) to manage VRAM vs RAM vs SSD?
- niemenghui 2mo ago[flagged]
- junsu22 2mo ago"The layer-by-layer streaming is clever. Curious how the throughput compares to running a quantized model on the same GPU — seems like quantization might still win on speed?"
- Alisaqqt 2mo ago“Can run” and “is useful interactively” are different benchmarks. At this latency, I can still imagine batch or overnight jobs being interesting; for chat, time to first useful answer matters much more than whether the weights technically fit. A workload/latency/energy table would make projects like this easier to evaluate.