5 ms·
Hetzner introduces GPU server for AI training
- justmarc 2y agoAlthough R2 and B3 are excellent alternatives to S3, go get them Hetzner! Hetzner is a great, reliable company with fantastic offerings and excellent support.
- lvl155 2y agoI always looks at these prices and think it’s a complete rip off for anyone doing less than 4 GPUs.
- loughnane 2y agoWhat’s the most cost effective option for hosting an llm these days? I don’t need to train, I just want to use one of the llama models for inference to reduce my reliance on 3rd parties.
- l5870uoo9y 2y agoHave you tried how far you will get with a Hetzner VPS with dedicated CPU(s)?
- PaulRobinson 2y agoDepends on the specific model and your perf requirements, but lots of them will run on a single box with a middle of the road GPU. If your invocation rate is low, hosted solutions like AWS Bedrock or using hosted APIs might be cheaper.
- rglullis 2y agolocally? I purchased a XTX 7900 GPU for ~900€ and I'm using ollama to run it and I've been trying different models.
- exe34 2y agoConsider also an online llama as a service like deepinfra. I have a local 3090 for playing around with the smaller models, but it's nice having the option of calling the 405b.
- loughnane 2y agoOoh, I like that. Can see using them as a stepping stone where I'm using an open source model but without the hassle of setting up my own machine (but that I could later).
- weweersdfsd 2y agoIf you don't need a big model and are fine with hosting locally, an RTX3060 with 12GB VRAM is going to do just fine. Can be bought for about 200-300 USD. I've been pleasantly surprised by what such a mediocre GPU and Llama3 8B can do for certain (simple) use cases. Ollama makes it all pretty easy.
- andersa 2y agoHmm. Seems like a bad deal. This is a monthly reservation for a single 6000 Ada for $940. You can get the same on RunPod for $670. And to actually train stuff you'd likely want nodes with more of them, like 8, or just different GPUs all together (like A100/H100/etc).
- indigodaddy 2y agoHow long I wonder before these sorts of things will be affordable to the more everyday person say sub-$100?
- andersa 2y agoNot before the bubble pops.
- jsheard 2y agoEven when it does, GPUs with more VRAM than a flagship gaming model have always and will always come at a massive price premium. That ceiling is currently 24GB, if you need more than that it's going to cost you.
- rafaelmn 2y agoI don't see why. AMD could buy a lot of free community work just by putting out a 32 GB version of their next gen flagship at around 1k. Not chasing the margin for a generation would buy them a decent amount of community support - they need it if they want to compete with Nvidia at any point. Ahh I just saw OP was saying sub 100$ - yeah that's never going to happen.
- InvaderFizz 2y agoVRAM is such a small portion of the BOM that they could double it, keep the same margins, and the MSRP would only increase by like $300. The problem isn't margin on the consumer cards, the problem is competing with their even higher margin workstation cards.
- milabogdanova 2y ago[dead]
- yk 2y agoSo 1kEUR/month for a 6kEUR GPU. Pretty sure there are a lot of drug dealers who wish they had gone into cloud training instead.
- bluecalm 2y agoDrugs don't require electricity and a CPU and RAM though. People running those will use a lot power.
- deleted 2y ago[deleted]
- dist-epoch 2y agoIt's the same for CPU servers, but maybe 9 months instead of 6 to fully cover the cost.
- dist-epoch 2y agoPricing is surprising, typically Hetzner has extremely low prices, yet here there are 50%-70% more expensive then the competition, and you also pay a one time setup cost.
- mromanuk 2y agoI use runpod or vast for training my (small) models (a few million parameters) mostly using RTX4090 up to 4 GPUs. Training is a sporadic task. Is not worth it for me to book it monthly (at these prices)
- kirab 2y agoHetzner customer here. It’s a little hard to understand in the UX, but the price shown is the monthly max price. It is actually paid by the hour. The price per hour for this server is € 1.5980 more info: https://docs.hetzner.com/general/others/new-billing-model/ https://docs.hetzner.com/general/others/new-billing-model/
- naiv 2y agoBut you have to pay the setup fee of 94 Euros every time you would choose this server, or?
- elaus 2y agoThat's the second time I read this comment and I still don't believe it: it's listed as a "dedicated root server" (usually billed by the month) with no mention of typical cloud offers. Could you please clarify?
- gosub100 2y agoDo any of these offer training data as a service? Seems like they could charge a premium for a continuous multicast of a large dataset on say a 10g or higher connection. A one-to-many reply and charge the customer to sit under the firehose.
- deleted 2y ago[deleted]
- krick 2y agoWhat's currently the cheapest/easiest way to deal with relatively lightweight GPU tasks, that are not lightweight enough for my PC? Consider this use case: I want to upload 50 GB of audio somewhere and run whisper (biggest model) on it. I imagine the processing should be doable in minutes for a powerful GPU and must be very cheap, the script will be like 20 LOC, but I'll spend some time setting stuff up, uploading this and so on (which for example, makes colab a no-go for this). Any recommendations? Also, when they say it's "per hour" do they mean an hour of GPU-time, or an hour of me "renting the equipment", so to say?
- demarq 2y agoCerebrium https://news.ycombinator.com/item?id=41579777 https://news.ycombinator.com/item?id=41579777
- rkwasny 2y agoAccording to the benchmarks: https://github.com/mag-/gpu_benchmark https://github.com/mag-/gpu_benchmark RTX 6000 Ada is ~A100
- Blaec 2y agoCoCalc offers On-Demand GPU servers with H100s starting at $2.01 per hour (metered per second) through its integration with Hyperstack... It also has more budget-friendly options, like RTX A4000s at $0.18 per hour. https://cocalc.com/features/compute-server https://cocalc.com/features/compute-server In case you are not familiar, CoCalc is a real-time collaborative environment for education and research that you can access via your web browser at https://cocalc.com/ https://cocalc.com/