12 ms·
$2 H100s: How the GPU Rental Bubble Burst
- bsder 2y agoSo, where can a plebian like me buy a (or 10) used H100?
- wmf 2y agoI don't expect them to hit the used market before 2026-2027. Data centers will start replacing H100 with R100 at that time.
- h_tbob 2y agoI'm really hoping Jim Keller and Tenstorrent crew drop it even further with GDDR backed AI compute.
- askl 2y ago$2/h rental, not $2 sales price. Pretty misleading.
- pico_creator 2y agoIf we $2 H100 this year or next. Either AI is super dead, or a new alien GPU rained from the sky
- ranger_danger 2y agoBlackwell B100/B200 did kinda rain down, also the AMD MI300X and increased availability of H200. There's also cheaper NVIDIA L40/L40S if you don't need FP64.
- askl 2y agoI'm hoping for the first one
- marcyb5st 2y agoThere's option 3: current capacity is enough for our AI needs and so GPUs now the market is flooded. I think AI is not gonna die even in its current stocastic parrot incarnation. It is a useful tool for some tasks and, albeit not transformative like some CEOs, I believe it's gonna stay. At most I believe we will enter another AI winter until there's the next algorithmic breakthrough.
- friendzis 2y agoCurrent stochastic parrots do not have to be transformative, they have to appear smart enough for a critical mass of dumb enough people. And judging anecdotally from scanning social media - they already do. Even here, on HN, you find numerous comments of the shape: "${my favorite gpt} says this: <insert some gibberish>"
- dangsux 2y ago[dead]
- renewiltord 2y agoBruh
- stego-tech 2y agoAgreed, and I doubt we’ll see one retail at that price even on the secondhand market anytime soon. That said, could I see them being offloaded in bulk for pennies on the dollar if the (presumed) AI bubble pops? Quite possibly, if it collapses into a black hole of misery and bad investments. In that case, it’s entirely plausible that some enterprising homelabs could snatch one up for a few grand and experiment with model training on top-shelf (if a generation old) kit. The SXMs are going for ~$26-$40k already, which is cheaper than the (worse performing) H100 Add-In Card when brand new; that’s not the pricing you’d expect from a “red hot” marketplace unless some folk are already cutting their losses and exiting positions. Regardless, interesting times ahead. We either get AI replacing workers en masse, or a bust of the tech industry not seen since the dot-com bubble. Either way, it feels like we all lose.
- osigurdson 2y ago2 bucks for a GPU? Maybe a PIC microcontroller.
- askl 2y agoThey don't even have HDMI ports so they are pretty useless, but I'd buy one at $2 as a desk ornament.
- qingcharles 2y agoGPU display stand: https://www.reddit.com/r/nvidia/comments/1fw68rl/retiring_a_gtx980/ https://www.reddit.com/r/nvidia/comments/1fw68rl/retiring_a_...
- dchftcs 2y agoYou don't need an HDMI port, you just need a driver to support running the right graphics calculations and producing image to funnel to another output port. The GPU may lack some features, may have an architecture that is bad for rendering, and may be suboptimal in delivering the performance per watt. Exactly like how a CPU doesn't have a display port.
- qingcharles 2y agohttps://news.ycombinator.com/item?id=41806396 https://news.ycombinator.com/item?id=41806396
- qingcharles 2y agoSome of the Tesla GPUs are almost at this price per unit on eBay now. I've seen them go for under $15 online. Here's one for ~$18 inc shipping with 6GB DDR5: https://www.ebay.com/itm/Nvidia-Tesla-K20X-6GB-90Y2351-C7S15A-Dell-GJD61-GPU-1yr-Warranty-Fast-Ship/156086508249 https://www.ebay.com/itm/Nvidia-Tesla-K20X-6GB-90Y2351-C7S15...
- barrenko 2y agoAre these a viable buy?
- chessgecko 2y agoYou’d get better perf training on a current gen phone than that gpu, but it probably functions
- pico_creator 2y agoOnly if ur a collector (so no if ur plugging it in)
- pico_creator 2y agoThat hurts - i used those GPUs before at their peak Now any random GPU in the computer store murders it
- chessgecko 2y agoNot just gpus, the k20 was at 3.9 Tflops (fp32) and the new iPhone is at 4.3 (fp16). If you don’t need the precision it got passed by the phones
- dplgk 2y agoIt appears this GPU cost $7700 when it launched in 2012? GPUs have gotten that much better that this thing isn't even with $100?
- squigz 2y agoMisleading? Anyone who read this title and thought it was referring to the full purchase price might deserve to be misled.
- hackernewds 2y agoThat is what the title says explicitly. That's how click bait works
- squigz 2y agoIt also explicitly says "rental", so I'm not sure how one can possibly arrive at the conclusion that they meant "$2 to own an H100"
- gnabgib 2y agoIt didn't say that at the time, the article still has the submitted title: $2 H100s: How the GPU Bubble Burst
- squigz 2y agoEven so, I genuinely don't see how anyone who might be clicking this article could possibly interpret it the way GP is saying.
- cuu508 2y agoWell, case in point, I did. When I read the title I thought – "IIRC these were going for thousands, could they have really dropped so hard? Well, sometimes companies, cars, real estate properties cost $1, but there's always of course a catch. Let's see what the catch is here... <click> ah, it's a 4x reduction of rental price, boring"
- nottorp 2y agoAnyone who isn't an "AI" fanatic can and will interpret the title as the sale price :)
- two_handfuls 2y agoAgreed, "$2/h" would be the correct unit, "$2" reads to me like a typo.
- hislaziness 2y agoTLDR: Don’t buy H100s. The market has flipped from shortage ($8/hr) to oversupplied ($2/hr), because of reserved compute resales, open model finetuning, and decline in new foundation model co’s. Rent instead. Is the AI infra bubble already bursting?
- TechDebtDevin 2y agoNo, but the prices will likely converge with MSRP pricing. A lot of datacenter were filled with h100s that cost a premium to get ahold of.
- pico_creator 2y agoCovered in the article. They are below MSRP essentially
- hislaziness 2y agoIt is not just MSRP, management and operations cost too. The article goes into the details of this.
- pico_creator 2y agoQ_Q yes - ur right on that - and i wrote the article (about a month ago)
- pico_creator 2y agoI’m hopping more for an open weights AI boom With cheap compute for everyone to finetune :)
- justahuman74 2y agoYes, please only rent instead - sincerely, all of the cloud providers
- pico_creator 2y ago~Cough~ not all cloud provider (there are many still willing to charge you an arm and a leg) Only the ones who can give you below MSRP essentially
- TechDebtDevin 2y agoI've been saying this would happen for months. There (was) a giant arbitrage for data centers that already have the infra. If you could get a hold H100s and had an operational data center you essentially had the keys to an infinate money printer on anything above $3.50/hr. Of course, because we live in a world of effecient markets that was never going to last forever. But they are still profitible at $2.00 assuming they have cheap electricity/infra/labor.
- pico_creator 2y agoProblem is - u can find some at $1
- startupsfail 2y agoThe screenshot there is 1xH100 PCIE, for $1.604. Which is likely promotional pricing to get customers onboarded. With promotional pricing it can be $0 for qualified customers. Note also, how the author shows screenshots for invites for private alpha access. It can be mutually beneficial for the data center to provide discounted alpha testing access. The developer gets discounted access, the data center gets free/realistic alpha testing workflows.
- electronbeam 2y agoThe PCIE has much lower perf than even a 1x slice of an SXM
- pico_creator 2y agoI really suggest shopping around. <$2 SXM is a real thing, if your patient enough on the schedule.
- pico_creator 2y agoWhen I did the screenshot a month ago, it wasn't public info yet. Now its public: SFCompute list it on their main page - https://sfcompute.com/ https://sfcompute.com/ And they are *not* the only one
- bugbuddy 2y agoAt $2 per hour, factoring in the overall hardware cost, labor, electricity, and other sunk costs like floor space and bandwidth, how many total hours does it take to break even? What is the expected hardware operation lifespan in hours of this system? How much would the hardware cost have to drop for the economic of $2/hour to work?
- deleted 2y ago[deleted]
- hislaziness 2y agoThe details are in the article. They have done the math.
- bugbuddy 2y agoThere was no answer to my last question which I think is the most important thing when considering if we are going to have another GFC this year or next year.
- rsynnott 2y agoDoes "GFC" stand for "global financial crisis" here? It seems implausible that the collapse of the LLM bubble will cause one; it might have a pretty dramatic impact on the markets, but it's unclear how it would cause the sort of systemic failure that we saw in the noughties.
- pico_creator 2y agoYou could technically break even at $2, assuming 100% allocation, and cheap electricity. But reality is not 100%, so I would argue at-least 25% or even 50% drop in the H100 price (approx 50k each, after factoring other overheads)
- latchkey 2y ago> What is the expected hardware operation lifespan in hours of this system? Better question: what support contract does the provider have with their manufacturers? For example, we buy Dell pro support 3 year next business day contracts on all of our gear.
- ctrlGsysop 2y agoA good in depth mkt analysis. While it’s not crypto, many of the key points are rinse and repeat of mining - things like insatiable demand and projected ROI. Markets and tech solve high costs all the time. Great point made about the $4/hr number that was most likely a top bullet in a 1000 pitch decks citing NVIDIA. Bagholders could just be all the nations buying all the billionaire’s stories.
- aurareturn 2y agoThe only difference is that LLMs have a real world value.
- bugbuddy 2y agoThere is one big exception in the list of all nations. I don’t know what to make of it. Irony?
- wmf 2y agoYeah, I did this same kind of math all the time back during the early ASIC mining days except it was accelerated; you had to break even in 9 months or never due to the exponentially growing difficulty.
- pico_creator 2y agoYea, the older GPU providers, were pushing 3-5 year commits for a reason. They seen this before
- ranger_danger 2y agoLast year we reached out to a major GPU vendor for a need to get access to a seven figure dollar amount worth of compute time. They contacted (and we spoke with) several of the largest partners they had, including education/research institutions and some private firms, and could not find ANYONE that could accommodate our needs. AWS also did not have the capacity, at least for spot instances since that was the only way we could have afforded it. We ended up rolling our own solution with (more but lower-end) GPUs we sourced ourselves that actually came out cheaper than renting a dozen "big iron" boxes for six months. It sounds like currently that capacity might actually be available now, but at the time we could not afford to wait another year to start the job.
- InkCanon 2y ago[dead]
- chronogram 2y agoIf you were able to make do with cheaper GPUs, then you didn't need FP64 so you didn't need H100s in the first place right? Then you made the right choice in buying a drill for your screw work instead of renting a jackhammer even if the jackhammer would've seemed cooler to you at the time.
- KeplerBoy 2y agoDoes anyone doing AI need FP64, and yet they sell well.
- ranger_danger 2y ago> didn't need H100s I think we're splitting hairs here, it was more about choosing a good combination of least effort, time and money involved. When you're spending that amount of money, things are not so black and white... rented H100s get the job done faster and easier than whatever we can piece together ourselves. L40 (cheaper but no FP64) was also brand new at the time. Also our code was custom OpenCL and could have taken advantage of FP64 to go faster if we had the devices for it.
- grues-dinner 2y ago> For all the desperate founders rushing to train their models to convince their investors for their next $100 million round. Has anyone actually trained a model actually worth all this money? Even OpenAI is s struggling to staunch the outflow of cash. Even if you can get a profitable model (for what?) how many billion dollar models does the world support? And everyone is throwing money into the pit and just hoping that there's no technical advance that obsoletes everything from under them, or commiditisation leading to a "good enough" competitor that does it cheaper. I mean, I get that everyone and/or they investors has got the FOMO for not being the guys holding the AGI demigod at the end of the day. But from a distance it mostly looks like a huge speculative cash bonfire.
- jordwest 2y ago> I get that everyone and/or they investors has got the FOMO for not being the guys holding the AGI demigod at the end of the day Don't underestimate the power of the ego... Look at their bonfire, we need one like that but bigger and hotter
- justahuman74 2y ago> For all the desperate founders rushing to train their models to convince their investors for their next $100 million round. I would say Meta has (though not a startup) justified the expenditure. By freely releasing llama they undercut every a huge swath of competition who can get funded during the hype. Then when the hype dies they can pick up what the real size of the market is, with much better margins than if there were a competitive market. Watch as one day they stop releasing free versions and start rent seeking on N+1
- evbogue 2y agoI was surprised recently when I fired up ollama on my refurbished Thinkpad -- a laptop that doesn't even have a GPU. All of the hype had me convinced that I couldn't run any of this LLM stuff at home! It's a little bit slower, but while I wait for the text to generate I have another cup of coffee. Sometimes I even prompt myself to generate some text while I'm waiting.
- heiploy 2y agotraining is the phase that needs all that compute
- evbogue 2y agoThis is good to know. I had read somewhere (that was probably on the Internet) that every time I submitted a prompt at the Meta AI web site that I was vaporizing an entire bottle of water, so imagine how thrilled I was to be saving so much water by prompting AI at home! But alas, the water was already vaporized. The climate? Already changed.
- gloflo 2y agoNope, climate is changing to even worse. It's not a "oops, OK now we live with this new reality" but "oh fuck, the rollercoaster is getting steeper AND is accelerating more and more, the breaks are lose and we already lost half of the wagons".
- evbogue 2y agoMaybe with enough H100s we can next word predict a solution to this global issue.
- m3kw9 2y agoCurrent 1b model will do you no good, just rotate through all the free stuff and it would cover most of you usecases
- electronbeam 2y agoThe real money is in renting infiniband clusters, not individual gpus/machines If you look at lambda one click clusters they state $4.49/H100/hr
- latchkey 2y agoI'm in the business of mi300x. This comment nails it. In general, the $2 GPUs are either PE venture losing money, long contracts, huge quantities, pcie, slow (<400G) networking, or some other limitation, like unreliable uptime on some bitcoin miner that decided to pivot into the GPU space and has zero experience on how to run these more complicated systems. Basically, all the things that if you decide to build and risk your business on these sorts of providers, you "get what you pay for".
- jsheard 2y ago> slow (<400G) networking We're not getting Folding@Home style distributed training any time soon, are we.
- krasin 2y agoDistributed training data creating & curation is more useful and feasible. Training gets cheaper 1.5x every year, but data is just as expensive, if not more, given that the era of "free web crawls of human knowledge" is over.
- pico_creator 2y agoExactly, it covered in the article that there is a segmentation happening via GPU cluster size. Is it big enough for foundation model training from scratch = ~$3+ Otherwise it drops hard Problem is "big enough" is a moving goal post now, what was big, becomes small
- swyx 2y agoso why not buy up all the little h100s and enough together for a cluster? seems like a decent rollup strategy? ofcourse it woudl still cost a lot to do... but if the difference is $2/hr vs $4.49/hr then there's some size where it makes sense
- anshulbhide 2y agoThis reminds me of the boom and bust oil cycle as outlined in The Prize: The Epic Quest for Oil, Money & Power by Daniel Yergin.
- swyx 2y agocare to summarize key points for the class?
- dplgk 2y agoIt seems appropriate, in this thread, to have ChatGPT provide the summary: In The Prize: The Epic Quest for Oil, Money & Power, Daniel Yergin explains the boom-and-bust cycle in the oil industry as a recurring pattern driven by shifts in supply and demand. Key elements include: 1. Boom Phase: High oil prices and increased demand encourage significant investment in exploration and production. This leads to a surge in oil output, as companies seek to capitalize on the favorable market. 2. Oversupply: As more oil floods the market, supply eventually exceeds demand, causing prices to fall. This oversupply is exacerbated by the long lead times required for oil development, meaning that new oil from earlier investments continues to come online even as demand weakens. 3. Bust Phase: Falling prices result in lower revenues for oil producers, leading to cuts in exploration, production, and jobs. Smaller or higher-cost producers may go bankrupt, and oil-dependent economies suffer from reduced income. Investment in new production declines during this phase. 4. Correction and Recovery: Eventually, the cutbacks in production lead to reduced supply, which helps stabilize or raise prices as demand catches up. This sets the stage for a new boom phase, and the cycle repeats. Yergin highlights how this cycle has shaped the global oil industry over time, driven by technological advances, geopolitical events, and market forces, while creating periods of both rapid growth and sharp decline.
- DebtDeflation 2y agoThis isn't just the story of GPUs or Oil, this is the entire story of capitalism going back to the early Industrial Revolution in the 1700s. The economist Hyman Minsky added asset prices and debt financing to it to round out a compelling theory of the business cycle including the extreme bubbles and depressions sometimes seen.
- kristopolous 2y agoThis sounds like bad news for the gpu renter farms. Am reading this right?
- swyx 2y agothe marketplaces like sfcompute do great, bc so much cheap supply and theres lots of demand. its the foundation model startups who locked into peak hype contracts for access that are eating a lot of losses right now... (which perhaps explains why the bigcos are acquiring only the founders and not assuming the liabilities of the oldco...)
- sgu999 2y ago> which perhaps explains why the bigcos are acquiring only the founders and not assuming the liabilities of the oldco... Who did?
- lamontcg 2y agoso, time to short NVDA?
- Ekaros 2y agoOld adage still stands. But I would certainly unload some if I had any.
- pico_creator 2y agoHard to say, i mean A100's had the same freefall - and nvidia just grew with H100's
- swyx 2y agocan you do a quick rerun of the ROI math with BH200 numbers now that we know them? minus the fp4 shenanigans ofc
- pico_creator 2y agoDo we have actual fp8 numbers? (or i could proxy it by /2 the fp4)
- aurareturn 2y agoThe fact that cheaper GPU prices have drawn so much interest here should tell you that prices will bounce back. The lower the price, the more people will experiment with fine tuning and inferencing.
- physicsguy 2y agoOpen models like Llama make it pointless for the majority of companies to train from scratch. It was obvious this would happen.
- Der_Einzige 2y agoI just want to observe that there are a lot of people paying huge amounts of money for consulting about this exact topic and that this article is jam packed with more recent and relevant information than almost any of these consultants have.
- swyx 2y agoauthor @pico_creator is in here actively replying in case u have any followups.. i just did the editing
- deleted 2y ago[deleted]
- pico_creator 2y agoFeel free to forward to the clients of "paid consultant". Also how do i collect my cut.
- pico_creator 2y agoAlso: how many of those consultants, have actually rented GPU's - used them for inference - or used them to finetune / train
- aurareturn 2y agoI’m guessing most of them are advising Wallstreet on AI demand.
- wg0 2y ago> Collectively there are less than <50 teams worldwide who would be in the market for 16 nodes of H100s (or much more), at any point in time, to do foundation model training At best 100 and this number will go down as many would fail to make money. Even traditional 100 software development companies would have a very low success rate and here we're talking about products that themselves work probabilistically all the way down.
- pico_creator 2y agoIm quite sure there is more than a 100 clusters even. Though that would be harder to prove. So yea, it would be rough
- latchkey 2y agoI am building a bare metal mi300x service provider business. Anyone offering $2 GPUs is either losing money on DC space/power, or their service is so sketchy under the covers, which they do their best to hide. It is one thing to play around with $2 gpus and another to run a business. If you're trying to do the latter, you're not considering how you are risking your business on unreliable compute. AWS really twerked people's perception of what it takes to run high end enterprise GPU infrastructure like this. People got used to the reliability hyperscaler offers. They don't consider what 999999% uptime + 45kW+ rack infrastructure truly costs. There is absolutely no way anyone is going to be making any money offering $2 H100s unless they stole them and they get free space/power...
- dijit 2y ago> 999999% uptime Assuming you mean 99.9999%; your hyperscaler isn't giving you that. MTBF is comparable. It's hardware at the end of the day, the VM hypervisor isn't giving you anything on GPU instances because those GPU instances aren't possible to live-migrate. (even normal VMs are really tricky). In a country with a decent power grid and a UPS (or if you use a colo-provider) you're going to get the same availability guarantee of a machine, maybe even slightly higher because less moving parts. I think this "cloud is god" mentality betrays the fact that server hardware is actually hugely reliable once it's working; and the cloud model literally depends on this fact. The reliability of cloud is simply the reliability of hardware; they only provided an abstraction on management not on reliability.
- llm_trw 2y agoI think people just don't realize how big computers have gotten since 2006. A t2.micro was an ok desktop computer back then. Today you can have something 1000 times as big for a few tens of thousands. You can easily run a company that serves the whole of the US out of a closet.
- dartos 2y agoWell you’d probably also at least want a cdn in each region, so like 3 closets.
- frhack 2y agoArtificial Intelligence will replace many jobs and business. So the race is on to become the main AI providers of the future. For the big players this is an opportunity and a necessity. The question is: - how long will this race last? - how long will NVIDIA be the main GPU provider and beneficiary of this race? Predicting the future is very difficult, especially in an unprecedented revolution like this. As Nobel Prize winner Parisi said: "No matter how hard you try to predict the future, the future will surprise you"
- Havoc 2y agoThere is also the small matter of a new gen coming out… Not convinced anything has burst yet. Or will for that matter. The hype may be bubble like but clearly we will need a lot of compute.
- amelius 2y agoDoes that include electricity?
- sva_ 2y agoI've been wondering if any state actors might seem it favorable to offer gpus and sniff on the training data/model architectures
- Der_Einzige 2y agoI'm sure this is happening. Hell, weights and biases was doing this years ago to early free users (maybe they never stopped). I assume that anyone doing good work in the AI space is being "sniffed" on, and if not, than the relevant "sniffers" are failing to do their jobs!
- authorfly 2y agoHaha. Cries in sadness that my university lab was unable to buy compute from 2020+ when all the interesting research in AI was jumping up and now AI is going into winter finally compute will be cheap again.
- 7734128 2y agoI don't feel any winter yet.
- thelastparadise 2y agoAt least not until LLM gains hit a wall. So far every open weight model has far surpassed the previous releases at the same model size.
- danpalmer 2y agoBut closed models are clearly slowing. It seems reasonable to expect that as open weight models reach the closed weight model sizes they’ll see the same slowdown.
- alecco 2y agoIf you remove LLMs, there is absolutely an AI winter.
- kkzz99 2y agoAudio generation (music, tts, voice cloning), Video and Image generation, multi-modal models, protein simulation... where is the winter?
- authorfly 2y agoWell, it's in academia, in traditional universities, any way. I think corporates are still thriving. I can say from an academic point of view, I knew 4 PhDs who started in 2018/2019, all 4 got depressed and left the field. Their research was obsolete before they were halfway through. Usually some PhD students get depressed, but these 4 had awful timing. Their professors were stuck on 3-10 year grants doing things like BERT finetuning or convolution or basic level AI work - stuff that as soon as GPT-3 came out, was clearly obsolete, but nobody could admit that and lose the grants.. In other cases, their work had value, but drew less attention than it should have became all attention went to GPT-3 or people assumed it was just some wrapper technology. The nature of academia and the incentive system caused this; academia is a cruise ship which is hard to turn. If the lighthouse light of attention moves off your ship on to another fancy ship, your only best is lifeboats(industry) or hoping the light and your ship intersect again. The professors have largely decided to steer either right into Generative AI and using the larger models (which they could never feasibly train themselves) for research, or gone even deeper into basic AI. The problem? The research grants are all about LLMs, not basic AI. So basically a slew of researchers willing and able to take on basic AI research are leaving the field now. As many are entering as usual ofcourse, but largely on the LLM bandwagon. That may be fine. The history of AI winters suggests putting all the chips on the same game like this is folly. I recall journals in the 90s and 2000s (my time in universities was after they were released, but I read them), the distribution of AI was broad. Some GOFAI, some neural nets, many papers about filters or clear visual scene detection etc. Today it's largely LLM or LM papers. There is not much of a "counterweight underdog" like neural networks served the role off in the 90s/00s. At the same time, for people working in the fields you mention, double check the proportion of research money going into companies vs institutions. While it is true things like TortoiseTTS[1] were an individual effort, that kind of thing is now a massive exception. In stead companies like OpenAI/Google literally have 1000+ researchers each developing the cutting edge in about 5 fields. Universities have barely any chance. This is how the DARPA AI winter went to my understanding(and I listened to one of the few people who "survived via hibernation" during my undergraduate); over promising - central focus on one technology - then company development of projects - government involvement - disappointment - cancellation. [1] https://github.com/neonbjb/tortoise-tts https://github.com/neonbjb/tortoise-tts
- bjornsing 2y agoThanks for the heads-up. I just increased my short position in NVDA a tiny bit. The peak should be near. (This is not financial advice.)
- alecco 2y ago"Markets can stay irrational for longer than you can stay solvent"
- bjornsing 2y agoI know. :) That’s why I keep it small. And I’m long semiconductors as a whole.
- deleted 2y ago[deleted]
- aurareturn 2y agoI would not bet against Nvidia right now. Yes, H100s are getting cheaper, but I can see the cheap price drawing in a wave of fine tuning interest, which will result in more GPU demand for both training and inferencing. Then there’s the ever need for bigger data centers for foundational model training, which the article described as completely separate from public auction prices of H100s. I don’t think the world has more GPU compute than it knows what to do with. I think it’s still the opposite. We don’t have enough compute. And when we do, it will simply drive a cycle of more GPU compute demand.
- bjornsing 2y agoI don’t think I’m betting against Nvidia. I’m betting against Nvidia being worth 3.3 trillion.
- Der_Einzige 2y agoStill a bad bet. Their moat is deeper now than it was in 2022. The engineers you need to poach are all paid well over 1+ million USD per year now. The number of people capable of writing quality CUDA code to optimize transformer language models world-wide is likely less than 10000, and I'm being very generous. Nvidia holds a significant portion of that group, and some of the others you'll never find in the market at all since they hide behind discord profile pictures and mental illness.
- hamilyon2 2y agoIs this the most computational bang for buck one ever seen? Another question: what is the maximum size of model I can fine-tune on 1 H100?
- yalogin 2y agoWhat does it mean for OpenAI? As open source models improve, OpenAI needs to keep on improving their models to stay ahead of them. Over time though, if it hasn’t already happeened, the advantages of OpenAI will not matter to most. Will OpenAI be forced to bleed money training? What does it mean for them over the next few years?
- murtio 2y agoEnjoyed the article and I was ready to try the promoted featherless.ai. I signed up and spent 15 minutes trying load or chat with Llama 3 models. All attempts failed. Naturally I would ask, if it's so cheap to run GPU's, why I would need to sign up to try a model?