3 ms·

Cloud > Local I have a stack of ten or so 3090s sitting in boxes, but it's not worth the hassle to use them. You can easily run models as cheap as water in the

by echelon 2mo ago

Cloud > Local I have a stack of ten or so 3090s sitting in boxes, but it's not worth the hassle to use them. You can easily run models as cheap as water in the cloud. Sitting around 15 minutes for local Minimax is stupid when you're trying to be productive. You can spin up parallel job instances and multitask in the cloud. If you want freedom, build open source cloud infra. You rent your ISP line. Why isn't renting GPU compute seen the same way? You still have compete ownership over your stack, you're just letting someone else deal with the capital outlay and headache.

knicholes 2mo ago
Why not rent those out on vast.ai and make a little money each month?
TacticalCoder 2mo ago
> You rent your ISP line. Why isn't renting GPU compute seen the same way? Is it mostly seen the same way. What is unacceptable is removing the freedom (which you mentioned) of people who prefer to run models locally. Just as people have the right to tinker at home with DIY and electronics, fully knowing they won't compete with the latest ASML machine, people are free to use open-weight models at home. BTW Minimax H3 running local can generate amazing short vids quite fast: about 50 seconds to generate a 7 seconds vids on a 4090 (depending on the settings). I've got a friend who spams my Telegram daily with such (no censorship and NSFW btw) vids. I don't run local but I'll defend the rights of people who want the freedom to do so and I won't look down at them from my high-horse talking about "electricity" and "productivity".
trouve_search 2mo ago
The value prop really depends on what you're doing. If you're just vibe coding with giant frontier models, yes, the value will be worse. Especially now, where GPU prices have spiked another 20% last month. For some tasks where owning the setup and full kv cache matters, the payoff calculation is ridiculously in favor of running your own deployment. For instance for some batch classifications jobs where the prefix cache hit rate will be >95%. The calculus also changes if you just use AI as a light tool while coding and don't need the giant models; qwen3 27B runs at 80TPS on a 5090 properly deployed.
- drewnick 2mo ago
  The KV cache is a great example. When you own hardware, vllm etc can keep your KV cache warm indefinitely for bursty large token loads vs clouds will purge quite often. One of my tasks has a very large but very static system prompt and instructions, think something like 128k tokens. On our own hardware, we keep that cached, sprinkle in the 8k of goodness needed for output, and DSV4 Flash 0731 can generate our output at ~50 tps on $10k worth of hardware. I've been benchmarking vs publics clouds (which are tbf insanely cheap!) and it's basically a break even in 24-36 months if nothing changes, which it could for better or worse. Combined with the security and stability of internal hardware, to me it's in "no brainer" territory for this workload, even if models don't improve. Our only "risk" is better cheaper hardware or cloud costs, which given the trajectory is murky at best short of a bust.
gessha 2mo ago
Why are you hoarding the 3090s T.T They've jumped from $700 to almost double on eBay.
fancyfredbot 2mo ago
Why do you have ten or so 3090s sitting in boxes? I mean obviously it's worth it just so you can flex on HN. But curious whether there was any other reason? Retired scalper?
- echelon 2mo ago
  I ran a popular gen AI audio website on them as an on-prem GPU cluster. 6.5M MAU with $650k ARR at peak. Half the inference happening locally. We got destroyed by bigger players (ElevenLabs, Suno, and more recently Fish) that fundraised and had better in-house models. It was a headache to run, I wanted to pursue a different market (video/creative), and I pivoted. My pivot failed and was dead for over a year as I went too far into controllability. One of my changes then got a semblance of PMF and it's doing $4M annualized run rate (but it's not fundable in its current shape). I ultimately want to build open source cloud infra and open source workflow products - more friendly than Comfy, more free than RunPod. In any case, I can't run my current work on the GPUs. But I'm close enough to pretraining and fine tuning smaller models that I want to use them for future experiments. Not that I wouldn't do that in the cloud.
  - fancyfredbot 2mo ago
    Thanks very interesting. Can imagine it being a pain.Ten is an unusual number, hard to divide evenly across hosts. To be honest I doubt you'll ever use them again, especially if you have dismantled the cluster and put them back in their boxes again. The cluster would be cheaper than cloud but you don't seem that cost constrained.
hluska 2mo ago
Do you really keep ten cards in boxes just to brag on Hacker News? Geez, our industry has gotten pathetic.
- echelon 2mo ago
  They ran production on-prem inference for millions of users on one of my older projects, but I've moved on from that and they're impractical for my current work.

by echelon 2mo ago

knicholes 2mo ago

Why not rent those out on vast.ai and make a little money each month?

TacticalCoder 2mo ago

> You rent your ISP line. Why isn't renting GPU compute seen the same way? Is it mostly seen the same way. What is unacceptable is removing the freedom (which you mentioned) of people who prefer to run models locally. Just as people have the right to tinker at home with DIY and electronics, fully knowing they won't compete with the latest ASML machine, people are free to use open-weight models at home. BTW Minimax H3 running local can generate amazing short vids quite fast: about 50 seconds to generate a 7 seconds vids on a 4090 (depending on the settings). I've got a friend who spams my Telegram daily with such (no censorship and NSFW btw) vids. I don't run local but I'll defend the rights of people who want the freedom to do so and I won't look down at them from my high-horse talking about "electricity" and "productivity".

trouve_search 2mo ago

The value prop really depends on what you're doing. If you're just vibe coding with giant frontier models, yes, the value will be worse. Especially now, where GPU prices have spiked another 20% last month. For some tasks where owning the setup and full kv cache matters, the payoff calculation is ridiculously in favor of running your own deployment. For instance for some batch classifications jobs where the prefix cache hit rate will be >95%. The calculus also changes if you just use AI as a light tool while coding and don't need the giant models; qwen3 27B runs at 80TPS on a 5090 properly deployed.

drewnick 2mo ago
The KV cache is a great example. When you own hardware, vllm etc can keep your KV cache warm indefinitely for bursty large token loads vs clouds will purge quite often. One of my tasks has a very large but very static system prompt and instructions, think something like 128k tokens. On our own hardware, we keep that cached, sprinkle in the 8k of goodness needed for output, and DSV4 Flash 0731 can generate our output at ~50 tps on $10k worth of hardware. I've been benchmarking vs publics clouds (which are tbf insanely cheap!) and it's basically a break even in 24-36 months if nothing changes, which it could for better or worse. Combined with the security and stability of internal hardware, to me it's in "no brainer" territory for this workload, even if models don't improve. Our only "risk" is better cheaper hardware or cloud costs, which given the trajectory is murky at best short of a bust.

gessha 2mo ago

Why are you hoarding the 3090s T.T They've jumped from $700 to almost double on eBay.

fancyfredbot 2mo ago

Why do you have ten or so 3090s sitting in boxes? I mean obviously it's worth it just so you can flex on HN. But curious whether there was any other reason? Retired scalper?

echelon 2mo ago
I ran a popular gen AI audio website on them as an on-prem GPU cluster. 6.5M MAU with $650k ARR at peak. Half the inference happening locally. We got destroyed by bigger players (ElevenLabs, Suno, and more recently Fish) that fundraised and had better in-house models. It was a headache to run, I wanted to pursue a different market (video/creative), and I pivoted. My pivot failed and was dead for over a year as I went too far into controllability. One of my changes then got a semblance of PMF and it's doing $4M annualized run rate (but it's not fundable in its current shape). I ultimately want to build open source cloud infra and open source workflow products - more friendly than Comfy, more free than RunPod. In any case, I can't run my current work on the GPUs. But I'm close enough to pretraining and fine tuning smaller models that I want to use them for future experiments. Not that I wouldn't do that in the cloud.
- fancyfredbot 2mo ago
  Thanks very interesting. Can imagine it being a pain.Ten is an unusual number, hard to divide evenly across hosts. To be honest I doubt you'll ever use them again, especially if you have dismantled the cluster and put them back in their boxes again. The cluster would be cheaper than cloud but you don't seem that cost constrained.

hluska 2mo ago

Do you really keep ten cards in boxes just to brag on Hacker News? Geez, our industry has gotten pathetic.

echelon 2mo ago
They ran production on-prem inference for millions of users on one of my older projects, but I've moved on from that and they're impractical for my current work.