3 ms·

Openrouter is not ideal, because you don't know who they send your traffic to and there are rumours of vendors cheating by providing quantized models. I'm very

by Roark66 12d ago

Openrouter is not ideal, because you don't know who they send your traffic to and there are rumours of vendors cheating by providing quantized models. I'm very happy with Deepinfra. Less model coverage, but good prices and quite fast. However I have to caution you about one thing. No one will give you as many input tokens for so little money as Claude Max x5 (maybe x20 too, I use x5). I tend to use 1.1B to 1.4B a week about 0.8-1B cached. Even with cache were talking thousands of $ in API prices a week. Hundreds if we're talking cheap cloud like Deepinfra. However, local AI well setup is actually a good alternative for this if Claude Max was unavailable. For example my system a ryzen 7950x 192GB ram, 5x rtx3090 plus an rtx5060 ti 16gb. (3 rtx3090 cards via usb4 egpu dock). Let's me run Qwen3.8-Flash-Next with 3slots (no rtx5060 used) at 55tok/s decode dropping to 50 at the end of a 260k context, 1200tok/s refill dropping to 950 at the end of context. With RAM and ssd cashing and 80% cache were talking on the order of 4B a week could be ingested by this setup (roughly) if it was running 24/7. I found 6 interactive cloud code sessions are fairly pleasant with this 3 user setup. If I include the rtx5060 in the mix I can bump to 5 users, but it slows down by about 15% (note the speeds are give are for one active user, multiple users at once see maybe 70% of tgat per user so aggregate is much higher in multi user setup). So in theory I should be able to run 10 cloud code sessions. Although I'm testing CC alternative now (pi with own plugins) because this model, while multimodal has only 260k context 30k of which CC eats on the getgo. Many people say local AI makes no sense financially. But in the event you process huge inputs that are often cached it does make sense.

bix6 12d ago
I haven’t heard of Deepinfra so I’ll check them out! I think your local setup would be a bit much for me capability / price wise. But maybe I can find a scaled down version. I don’t need insane tok/s. Oftentimes I just let things run and come back later.
- Roark66 9d ago
  If I was building it from scratch today on a budget I'd replace all my rtx3090s with modded rtx2080 ti 22gb. If they support nvlink od connect each pair. They are near a third of the cost of an rtx3090 while the performance is much better than a third. They cost $550 day before yesterday on Aliexpress here in EU. The problem is there aren't any cheaper gpu-less setups that could run this at let's say around half of my speed including prefill. While some people reported 20t/s on a strix I never saw a prefill number. I think it would be pretty bad (like 150-200tok/s). And these 20t/s are probably single user only at small context. So not worth the money for me. A Ram based alternative is a threadripper system with 8 ram channels, because it can have memory bandwidth comparable to cheaper gpus. But you need to use registered RAM and that is bonkers prices now. So personally I think sticking to a desktop pc MB, 2 or 3 gpus inside, plus 3 via usb4 is probably optimal. Using usb4 leaves your nvme slots for nvme. I'd consider 96GB RAM minimum comfortable (to keep kv cache of 10-12 claude code tabs you may work in). Don't forget the cost of the eGPU docks and psus. It's not trivial when you have 3-4 of them. I paid around $250 each. I only have this system because I was lucky to buy 80% of it when prices were better. If I was buying today I'd be hard pressed to justify even the 192gb of ddr5 (normal, not registered). If you or anyone else does this mind you'll spend a couple days getting resizable BAR working reliably. There is one more option. Tesla v100 cards. 16gb and 32gb. I would disregard 16gb cards immediately. Why? Pipeline paralellism allows you to split a model between cards for almost "free" (latency), but the layers are usually few GB big and you can never allocate it to consume all vram. You always have 0.5-2gb unused per card. 2gb is a lot for a 16gb card. So 32gb v100 sounds good right? Maybe... But if I was going towards v100 I'd not buy pcie version but the datacenter grade (I forgot the interconnect name). There are big adapter pcbs on Aliexpress that take 4 of those v100s and they allow you to connect all to single pcie, but the 3 v100s are all nvlinked. The pcb costs in the region of $500-600. But it doesn't make any nvlink exit the board. If they did... I'd be buying two such systems. Linking 8 32gb v100s together and with fast interconnect you can run tensor paralellism which uses compute of all those cards at once. It would prebeat my system 4x at least. Consider Qwen3.8-27B 4bit 8bit kv, q4 (if I remember correctly) run at 30-40 tok/s on a single rtx3090. Two cards in tensor paralellism and nvlink run it at over double at 90t/s and prefill, was amazing too.
Ciantic 12d ago
> Openrouter is not ideal, because you don't know who they send your traffic to and there are rumours of vendors cheating by providing quantized models. OpenRouter allows to make an API key locked to certain provider, I do that myself.

by Roark66 12d ago

bix6 12d ago

I haven’t heard of Deepinfra so I’ll check them out! I think your local setup would be a bit much for me capability / price wise. But maybe I can find a scaled down version. I don’t need insane tok/s. Oftentimes I just let things run and come back later.

Roark66 9d ago
If I was building it from scratch today on a budget I'd replace all my rtx3090s with modded rtx2080 ti 22gb. If they support nvlink od connect each pair. They are near a third of the cost of an rtx3090 while the performance is much better than a third. They cost $550 day before yesterday on Aliexpress here in EU. The problem is there aren't any cheaper gpu-less setups that could run this at let's say around half of my speed including prefill. While some people reported 20t/s on a strix I never saw a prefill number. I think it would be pretty bad (like 150-200tok/s). And these 20t/s are probably single user only at small context. So not worth the money for me. A Ram based alternative is a threadripper system with 8 ram channels, because it can have memory bandwidth comparable to cheaper gpus. But you need to use registered RAM and that is bonkers prices now. So personally I think sticking to a desktop pc MB, 2 or 3 gpus inside, plus 3 via usb4 is probably optimal. Using usb4 leaves your nvme slots for nvme. I'd consider 96GB RAM minimum comfortable (to keep kv cache of 10-12 claude code tabs you may work in). Don't forget the cost of the eGPU docks and psus. It's not trivial when you have 3-4 of them. I paid around $250 each. I only have this system because I was lucky to buy 80% of it when prices were better. If I was buying today I'd be hard pressed to justify even the 192gb of ddr5 (normal, not registered). If you or anyone else does this mind you'll spend a couple days getting resizable BAR working reliably. There is one more option. Tesla v100 cards. 16gb and 32gb. I would disregard 16gb cards immediately. Why? Pipeline paralellism allows you to split a model between cards for almost "free" (latency), but the layers are usually few GB big and you can never allocate it to consume all vram. You always have 0.5-2gb unused per card. 2gb is a lot for a 16gb card. So 32gb v100 sounds good right? Maybe... But if I was going towards v100 I'd not buy pcie version but the datacenter grade (I forgot the interconnect name). There are big adapter pcbs on Aliexpress that take 4 of those v100s and they allow you to connect all to single pcie, but the 3 v100s are all nvlinked. The pcb costs in the region of $500-600. But it doesn't make any nvlink exit the board. If they did... I'd be buying two such systems. Linking 8 32gb v100s together and with fast interconnect you can run tensor paralellism which uses compute of all those cards at once. It would prebeat my system 4x at least. Consider Qwen3.8-27B 4bit 8bit kv, q4 (if I remember correctly) run at 30-40 tok/s on a single rtx3090. Two cards in tensor paralellism and nvlink run it at over double at 90t/s and prefill, was amazing too.

Ciantic 12d ago

> Openrouter is not ideal, because you don't know who they send your traffic to and there are rumours of vendors cheating by providing quantized models. OpenRouter allows to make an API key locked to certain provider, I do that myself.