3 ms·
AI Server Cost Analysis – Memory Is the Biggest Loser
- gymbeaux 3y agoI guess the justification for the title comes beyond what we are allowed to read for free because my takeaways were that A) as is always the case, stock market bubbles include companies that shouldn’t be included and B) Nvidia is enjoying an immense profit boost due to being “effectively a monopoly.” I’m still skeptical that the demand for AI-training hardware will persist. How many companies need to train their own AI models vs “just use ChatGPT”? ESPECIALLY if ChatGPT introduces a means of keeping user data out of its training data. At the snap of a finger, millions of dollars of server equipment would become redundant. No skin off Nvidia’s back, but the party would be over at that point and its stock price would come back to merely “excessive” rather than “ridiculous”. Ironically, I’m seeing a need for lots of RAM for loading models locally. We had that one recently that required around 67GB of RAM to load. It was already trained, so no GPU required, but the desktop RAM requirement was what kept most of you from running it.
- gymbeaux 3y agoAs an analogy, it’s like if, when Azure and AWS really started to gain popularity, everyone went out and bought their own racks to create their own “clouds”. I realize at that point they wouldn’t be “clouds” but that’s not the point… the point is that most people seem to assume that every company is going to want their own in-house $10,000 Nvidia GPU and that’s almost certainly not going to happen. Nvidia’s best years are behind it if you ask me. They aren’t going bankrupt, but certainly from a share price perspective, it’s downhill from here (there may be a little more bubbling, no one knows, but the current share price is unreal).