6 ms·
LLM's are incredibly expensive to run. I imagine the huge demand that ChatGPT is seeing would make any cloud vendor sweat if you were to suddenly lump it on to
by TOMDM 4y ago
LLM's are incredibly expensive to run.
I imagine the huge demand that ChatGPT is seeing would make any cloud vendor sweat if you were to suddenly lump it on top of the usual demand.
To me it's entirely unsurprising that OpenAI would have trouble keeping up.
Good luck to them.
- la64710 4y agoThis is precisely why they should really open source their model so that anyone can download and run it on their own infrastructure. Just like google or others have done and one is free to run it on their own laptop (some even without a GPU) , on premise or on any cloud provider infrastructure. They can continue to provide a hosted service for their model but they should allow it to be downloaded just like BERT.
- scratchyone 4y agoWhy did you comment this 3 separate times?
- deleted 4y ago[deleted]
- TOMDM 4y agoDon't get me wrong, I'd love to be able to run GPT3 and the subsequent finetuned versions thereof myself, but OpenAI has essentially no financial incentive to do so. A few years ago, we could maybe lean on their open aspirations to get that done, but with the "limited profitibility model" they've since instead adopted, I think that dream is mostly gone. At least we still get the occasional treat like Whisper out of them.
- ronsor 4y agoIt's not as if anyone could afford to self-host the giant GPT-3 model anyway.
- TOMDM 4y agoI'm rather sure there are a mid sized number of "anyones" who would buy the hardware required to run it given the opportunity. But I certainly agree that not everyone can, and very few individuals. The thing is a monster of a model.
- deleted 4y ago[deleted]
- drdaeman 4y agoWhy not? Please correct me if I'm wrong but from what I've heard, It's not millions or hundreds of thousands, it's "only" tens of thousands dollars' of hardware we're talking about. There are enthusiasts that spend comparably on their hobbies. And a few dozen like minded people banding up together, shelling out a couple grands each? I'd say that's a totally realistic scenario.
- deleted 4y ago[deleted]
- t3rmi 4y agoOpen AI's CEO Sam Altman's take is that they will only ever allow API access to their models to avoid misuse. I don't get HN's take with wanting everything open sourced. Some things are expensive to create and dangerous in the wrong hands. Not everything can and should be open sourced.
- smeagull 4y agoYeah, just like the existence of Windows prevented Linux from ever existing. The wrong hands have the money to seek alternatives. All this policy does is keep it out of the hands of the public, and ensure that whatever open alternatives start up won't be OpenAI's.
- scifibestfi 4y agoIsn't an open source ChatGPT inevitable? There's already open source AI art.
- TOMDM 4y agoCheck out the BLOOM models if you want to see a first stab at that. If you can find an economical way of running it though, let me know.
- generalizations 4y agoA cluster of 6 year old 24GB NVIDIA Teslas should do the trick...they run for about $100 apiece. Put 12 or so of them together and you have the VRAM for a GPT3 clone.
- TOMDM 4y agoHuh. Tempting. Amazon has them listed at $200, but still, that's only $2,400 for 12 of them. Still, adds up once you get the hardware you'd need to NVlink 12 of them, and then on top of that, the price of power/perf you get probably isn't great compared to modern compute. Wonder what your volume would have to be before getting a box with 8 A100's from Lambdalabs would be the better tradeoff.
- kube-system 4y agoTheir end goal is to be stinkin’ rich, and it looks like that might happen. They’re not going to let the fear of an hour of downtime undermine that.
- motoboi 4y agoSomething like 1,5Tb memory to run this model in inference mode.
- antonvs 4y agoIs that really all? We regularly run multi TB memory clusters for big data processing and ML. I imagined it would be much bigger than that. To put that in perspective, 24x 64 GB nodes is 1.5 TB.
- albntomat0 4y agoMy understanding is that all the memory has to be GPU memory, with proper interconnects. Still not that crazy, all things considered
- lossolo 4y ago> 24x 64 GB nodes is 1.5 TB Looking at your calculations indicates that you mean RAM but it's 1.5 TB GPU VRAM (but this is assuming they use 64 bit precision, which is likely wrong so it's ~750 GB), not RAM.
- lossolo 4y agoYou meant 700 GB? 32 bit precision in 175B model is around ~700GB + overhead or around 350 GB if they use half-precision.
- motoboi 4y agoI was wrong. It seems they use fall precision. So 350gb. What is amazing, human language (languages?) and knowledge encoded in so little space.
- alfor 4y agoDon’t worry, we will see plenty of other GPT very soon. The genie is out of the bottle.
- whazor 4y agoI see OpenAI as a 'company' that (eventually) wants to earn money with their models. The 'open' part is trying to commercialize part it in such a way that everyone can use it. Also be open with the research and risks. But not open as in open-source.
- deleted 4y ago[deleted]
- osigurdson 4y agoThis was my assumption but it would be great to see it quantified. Does anyone know the average cost per query?
- earthboundkid 4y agoI think from Apple’s POV, this is great news. Moore’s Law has been dead for years, and there has been no good reason to upgrade your devices until now. AI means it’s 1990 again, and you need to buy a new device every 18 months because the performance leap is so meaningful to the UX.
- TOMDM 4y agoI'd agree if Apple were in the business of making datacentre infrastructure. Nobody is running these large scale models on their personal devices. Sure, some of the image generation tech is seeing personal use, so you'd have a point there, but these immense language models are something else entirely.
- earthboundkid 4y agoSomeone has to pay for that compute power. There are returns to scale and savings from sharing a machine when it would be idling, but those aren’t a factor of 10x. The AI will get in your pocket eventually and they’ll be ready to charge a premium for it.
- somenameforme 4y agoGLM-130B[1] (a 130 billion parameter model vs GPT-3's 175 billion parameter model) is able to run optimally on consumer level high-end hardware, 4xRTX 3090 in particular. That's < $4k at current prices, and as hardware prices go one can only imagine what it'll be in a year or two. It also enables running with degraded performance on lesser systems. It's a whole lot cheaper to run neural net style systems than to train them. "Somebody on Twitter"[2] got it setup, and broke down the costs, demonstrated some prompts, and what not. Cliff notes being a fraction of a penny per query, with each taking about 16s to generate. The output's pretty terrible, but it's unclear to me whether that's inherent or a result of priority. I expect OpenAI spent a lot of manpower on supervised training, whereas this system probably had minimal, especially in English (it's from a Chinese university). [1] - https://github.com/THUDM/GLM-130B https://github.com/THUDM/GLM-130B [2] - https://twitter.com/alexjc/status/1617152800571416577 https://twitter.com/alexjc/status/1617152800571416577
- spaceman_2020 4y agoIs this something that can come down with better tech (both hardware and software) or is the high cost just baked into LLMs?
- jsemrau 4y agoThat's the motivation for Microsoft in a nutshell.
- seydor 4y agoThey should sell chatGPT hardware boxes