3 ms·
I'm a bit surprised by the amount of comments comparing the cost to (often cheap) cloud solutions. Nvidia's value proposition is completely different in my opin
by derbaum 2y ago
I'm a bit surprised by the amount of comments comparing the cost to (often cheap) cloud solutions. Nvidia's value proposition is completely different in my opinion. Say I have a startup in the EU that handles personal data or some company secrets and wants to use an LLM to analyse it (like using RAG). Having that data never leave your basement sure can be worth more than $3000 if performance is not a bottleneck.
- sensesp 2y ago100% I see many SMEs not willing to send their data to some cloud black box.
- jckahn 2y agoExactly this. I would happily give $3k to NVIDIA to avoid giving 1 cent to OpenAI/Anthropic.
- originalvichy 2y agoEven for established companies this is great. A tech company can have a few of these locally hosted and users can poll the company LLM with sensitive data.
- lolinder 2y agoHeck, I'm willing to pay $3000 for one of these to get a good model that runs my requests locally. It's probably just my stupid ape brain trying to do finance, but I'm infinitely more likely to run dumb experiments with LLMs on hardware I own than I am while paying per token (to the point where I currently spend way more time with small local llamas than with Claude), and even though I don't do anything sensitive I'm still leery of shipping all my data to one of these companies. This isn't competing with cloud, it's competing with Mac Minis and beefy GPUs. And $3000 is a very attractive price point in that market.
- ynniv 2y agoI'm pretty frugal, but my first thought is to get two to run 405B models. Building out 128GB of VRAM isn't easy, and will likely cost twice this.
- rsanek 2y agoYou can get a M4 Max MBP with 128GB for $1k less than two of these single-use devices.
- lolinder 2y agoDon't these devices provide 128GB each? So you'd need to price in two Macs to be a fair comparison to two Digits.
- ynniv 2y agoThese are 128GB each. Also, Nvidias inference speed is much higher than Apple's. I do appreciate that my MBP can run models though!
- layer8 2y agoBut then you have to use macOS.
- ganoushoreilly 2y agoI read the Nvidia units are 250 Tflops vs the M4 Pro 27 Tflops. If they perform as advertised i'm in for two.
- logankeenan 2y agoHave you been to the localLlama subreddit? It’s a great resource for running models locally. It’s what got me started. https://www.reddit.com/r/LocalLLaMA/ https://www.reddit.com/r/LocalLLaMA/
- lolinder 2y agoYep! I don't spend much time there because I got pretty comfortable with llama before that subreddit really got started, but it's definitely turned up some helpful answers about parameter tuning from time to time!
- diggan 2y agoThe price seems relatively competitive even compared to other local alternatives like "build your own PC". I'd definitely buy one of this (or even two if it works really well) for developing/training/using models that currently run on cobbled together hardware I got left after upgrading my desktop.
- btbuildem 2y agoYeah that's cheaper than many prosumer GPUs on the market right now
- 627467 2y ago> Having that data never leave your basement sure can be worth more than $3000 if performance is not a bottleneck I get what you're saying, but there are also regulations (and your own business interest) that expects data redundancy/protection which keeping everything on-site doesnt seem to cover