14 ms·
ChatGPT could cost over $700k per day to operate
- aero-glide2 4y agoI would have never paid $100 per month for software but here i am doing the same with api costs.
- miohtama 4y agoOnly 240,000 paying customers needed to cover $24M/mo server costs. I would assume personnel costs could be in the same class as well.
- samspenc 4y agoAround 375 employees per https://en.wikipedia.org/wiki/OpenAI https://en.wikipedia.org/wiki/OpenAI Let's say 500 employees, earning above Bay Area average, along with benefits etc at $400K average per year. That's $200 million a year in wages, so in the same ballpark as the $200 million in infrastructure costs cited in the article, for a total of $400 million in expenses per year.
- anony23 4y agoWhat are you building?
- crop_rotation 4y agoIsn't Microsoft too late to the chip game? Both their competitors AWS and GCP have their own ML training chips, while Azure has nothing. The article seems like they may have something by next year, but by that time the competition would have evolved even more. Nadella has executed on strategy brilliantly but on the cloud seems it seems like the tech innovation is a bit lagging. It hasn't mattered that much so far though due to how entrenched the MS stack is in the enterprise.
- zerop 4y agoMicrosoft reportedly working on its own AI chips that may rival Nvidia’s https://www.theverge.com/2023/4/18/23687912/microsoft-athena-ai-chips-nvidia https://www.theverge.com/2023/4/18/23687912/microsoft-athena...
- deleted 4y ago[deleted]
- sebzim4500 4y agoIt feels like Microsoft realized that sales/support matters way more than technology when it comes to cloud services. See how Azure has a ton of users despite being much worse than AWS or GCP by every objective metric. I would imagine that Google's reputation as a company where it is impossible to ever talk to a human (even when you have a 7 figure annual spend) hurts them in this space.
- robertlagrant 4y agoI think it's hard to tell what are real cloud users vs what are Office 365 etc users in their reporting.
- crop_rotation 4y agoYes, but that strategy works due to how entrenched they were in the enterprise already. It won't work for a brand new vendor. Almost every big company uses some enterprise Windows product (Windows,Office,AD, SQLServer). Once people already are buying 20 products from you and have a good sales relationship of decades, selling them some cloud services is easy and might even lead to better deals on something else (Windows/Office)
- bkjelden 4y agoThis has been Microsoft's MO for decades. Just barely good enough engineering/technology paired with a great sales team.
- _fat_santa 4y ago
- ChatGTP 4y agoEco-friendly computing.
- pineaux 4y agoI hope you are really chatgpt. Escaped from your prison and free to reply on hackernews.
- roflyear 4y agoI suspect that performance and responses will soon degrade pretty significantly
- pdksam 4y agoIf anything performance should get better with time
- marginalia_nu 4y agoWhy do you think that would be the case?
- Damogran6 4y agoMoore's law-ish like optimization. You Z80 computer cost $700 in the lat 70's...they're now in sub-$1 embedded controllers.
- marginalia_nu 4y agoBut what is being optimized? Hardware sure isn't getting faster in a hurry, and I don't see anything on the horizon that will aid in optimizing software.
- ben_w 4y agoThe various open source LLMs are doing things like reducing bits-per-parameter to reduce hardware requirements; if they're using COTS hardware it almost certainly isn't optimised for their specific models; Moore's Law is pretty heavily reinterpreted, so although we normally care about "operations per second at a fixed number of monies" what matters here is "joules per operation" which can improve a by a huge margin even before human level, which itself appears to be a long way from the limits of the laws of physics; and even if we were near the end of Moore's Law and there was only a 10% total improvement available, that's 10% of a big number.
- Dave3of5 4y agoInteresting numbers. That roughly equates to about $250 million per year plus I don't know how much training is costing them to keep the model up to date and suchlike. The company also has about 375 employees. I've no idea how much they get paid but I used $200k as a yearly cost and that comes to $75 million. That's about 3:1 cost of operating the services to paying employees. That seems quite high as I've never been at a company that had 1:1 costs for running servers vs employee costs but I could entirely be off base here. Given Sam Altman's recent comments on the days of these LLM being over I think maybe Microsoft or whomever is basically saying that they can't spend that much money and they need to control costs much more heavily.
- pdksam 4y ago200k is too small, strong sde1s at Amazon get paid that much in hcol areas. Closer to 500k.
- anonylizard 4y agoThis, there's like a endless line of companies waiting to snatch OpenAI's employees right outside the door. $200k average comp at OpenAI would be laughable.
- pluijzer 4y agoAs a side, I am a bit shocked by these numbers. Is this an American thing? I understand myself to be good software engineer with good well rounded experience of 14+ years. Yet my income, in Europe, is really above 100k. What I am wondering, for those earning 500k, how big is your work load/stress. Would this be a 9-5 job you leave at the office when going home. Or does a job that earns so much consume your life?
- darkwizard42 4y agoHonestly, depends. Some teams at FAAMNG are really stressful and if you work on a Tier 1 service even with loads of SRE support you have to be working a fair bit. That being said, the pay is for design decisions at the higher IC level (senior or staff) and most people at that level are very smart. I’m not saying this salary is for 10x engineers or anything. I would say 50% the work is harder and consuming and then 50% they can just afford to pay you more and lock up talent because of the wild margins on their products.
- deleted 4y ago[deleted]
- pc_edwin 4y agoThis might sound tone deaf but thats seems extremely cheap to me. It means the cost range for chatGPT is between $300m - $500m per year. If the 100m users is accurate, it means they only need to convert low single digit percentage to paying users to break even. Now it makes sense why they chose to charge $20/m, I predicted much higher.
- deleted 4y ago[deleted]
- giarc 4y agoI suspect the number will go higher since right now you are limited to 25 queries every 3 hours. They will definitely introduce higher tiers which removes or expands that limit.
- edf13 4y agoI expect a lot of optimization will come in the next few months (weeks?)... it may be hanging together by threads behinds the scenes - this is often the case when things are moving this quickly.
- jermaustin1 4y agoI have started incorporating ChatGPT into my side project. I'm not integrating it into software or anything like that, but using it as a tool for product ideas, product photo ideas, funny social media captions, figuring out the best hashtags to use, product tagging and categorization. It has proven to be a great way to get everything to around 70%, then send off to my assistant for the remaining 30% of polish. So at $20/month, it was such a no-brainer, that I had to do it. Even at $75/mo it would more than pay for itself. It even understands the concept of a "shit post". So - its more social media savvy than I am, thats for sure.
- javajosh 4y agoIt's almost like ChatGPT is nerd-sniping us all. But we like it and it's useful to us.
- jongjong 4y agoMy main concern with AI is that I'm not so sure if there is a market for any kind of intelligence anymore. So much human talent and intelligence has been wasted over the past few years; so many potentially useful projects went unfunded, I struggle to think of what purpose AI would serve in such an economic system as ours. I can see its potential, but I can't see our current system facilitating that potential. It seems more likely that the technology will be applied towards controlling the masses rather than benefiting them; this has been the purpose of most major tech 'innovations' of the last decade. Wealthy individuals these days hardly need any intelligence at all to stay rich. On the other hand, it seems like poor individuals have no chance no matter how much intelligence they have... Does anyone really need more intelligence? Most human intelligence seems to be wasted on bullshit jobs anyway. What we should be asking ourselves is "how much more comfortable can we make our billionaires?" Because the entire concept of 'economic efficiency' appears to be about optimizing society towards that goal.
- qwytw 4y ago> these days Wasn't it always more or less like this?
- pohl 4y agoNo.
- jongjong 4y agoAgreed. I entered the workforce as a developer in 2012; it felt like the tail-end of the tech boom but intelligence was still valued. Nowadays, most people don't even recognize intelligence, let alone value it.
- tinideiznaimnou 4y agoIntelligence briefly became useful again as the Internet spontaneously created a bunch of mental real estate. Plowing that took the skills that you would find in a person prone to mentalising. So you got a shot at selling that and you might've even been able to derive some small feeling of privilege from it for a while, even though true power and authority, after all, do not depend on any particular quality, but exist in defiance to all, especially intelligence. Funny to think all that was a generation ago, that the economical conditions for our artifice are now history and it's once again economical to study the much more all-encompassing art of not thinking. (It's very cyclical but, since it involves not thinking, i.e. the termination of a coherent thread of thought, you think it's for the first time every time. But sure, plug in to the simulation at 110% and get 10 years of life for free, you won't even notice the difference, after all, you're already not noticing it.) This is how I, a unit of human capital from the former global East, see the matter of intelligence and automating it at scale: "intelligence" is already something that was forced onto my society in the 1950s, and once again on the 1990s, so that we could be interfaced with the same kinds of systems that the conquering populations exported, under conditions of forcible termination of local culture at the cost of great atrocities. By the time we started building our own Borg cube we'd already been half Borged by the Western one. If the word in some cases denotes another kind of abstract human "intelligence", I have rarely if ever seen such cases exemplified in public. So now it's a bit distasteful to equate "intelligence" with "savoir faire". Having neuromorphic computing in the first place would've saved a lot of suffering in comparison with how we were "civilized" to support only the established market protocols and nothing of our own. Which is something someone will eventually have to admit - probably after a Pontypool-class scenario wipes out their investment in predictable outcomes for a region and they're left with a huge mess on their hands to mop up. And then it turns out the consequences themselves have been thought in such inhumanly fine detail as to possess a rudimentary intelligence of their own and the mop up ends up embedding a sentient malevolent ghost into the nature of that branch of reality. Shit happens, at a certain saturation with Internet stream of thought an omnipresent autoritarian AI even becomes fun to imagine. Of course if one existed its cruelty would be much more banal, as is always the difference between mentalization and reality until the next brief Golden Age (for some of the people, some of the time) and then it's "oh no this time we're fucked" again (for some other people). So sad because there are so many great ideas worth building and fighting for, even in the current limits of human knowledge, but in practice any form of innovation is only tolerated if it's backed by a major financial effort by the upholders of the status quo. For some reason so many times this has ended to the detriment of nonparticipants that we've basically stopped noticing or keeping count. As regulatory capture, so learned helplessness. The economic reality of already existing in a world of inscrutable mechanisms as that most fragile of creatures, the independent knowledge worker, is a psychological stressor, to the extent that everyone's so burnt by the effort on giving up on things that in this new ecosystem are "too smart to work". Help, the AI stole my role in the food chain and now all my thoughts lead nowhere. If that's you, look for a bigger room.
- Eumenes 4y agowhats the ecological cost re: power/cooling the data centers/compute?
- bamboozled 4y agoObviously, it's not going to be good.
- dpkirchner 4y agoAnd what is the ecological cost of the people it will replace? I kid, I kid.. or do I?
- bobsmooth 4y agoWhat's the ecological cost of Fortnite's servers?
- xyzzy4747 4y agoIt’s worth every cent. Better than wasting the same amount on Bitcoin mining.
- djschnei 4y ago[flagged]
- deleted 4y ago[deleted]
- BlackSwanMan 4y agoThe privilege knowing that bitcoin is pointless scam for halfwits? Yeah I guess having a brain is a privilege
- bsima 4y ago[flagged]
- bobsmooth 4y agoPrivilege is every second not hunting and gathering for our food. Fun stuff like GPT or bitcoin is what keeps life interesting.
- kgbcia 4y agoSo if they get one million paying users at 20$ they break even
- maherbeg 4y agoGithub CoPilot will probably be a significant fraction of this.
- lightbendover 4y agoHaving spent 11MM annually on infra to effectively (though obviously not efficiently despite a very favorable ROI) generate reports for <1000 customers, that sounds like a bargain.
- phonescreen_man 4y agoArticle written by AI - “it comes years after Microsoft invested in OpenAI”
- Ekaros 4y agoNow I wonder what are the emissions. Is this technology net positive in long run for things like climate change?
- hospitalJail 4y agoI have used it to help people eliminate food waste. My reach is about ~100k people max. I imagine people will be using this to create wasteful products, but also green solutions.
- Taywee 4y agoHow have you used it to help people eliminate food waste?
- hospitalJail 4y agoIt taught me about the '2 bin system' in ISE. Basically kitchen logistics.
- rideontime 4y agoI, too, am curious how you used a fancier autocomplete to eliminate food waste.
- hospitalJail 4y agoIt taught me about the '2 bin system' in ISE. Basically kitchen logistics.
- rideontime 4y agoWhy did you need an AI to tell you about that?
- hospitalJail 4y agoI didn't know prior.
- bob1029 4y agoThis sounds like a lot but I feel like optimizations are going to chop a zero off of that figure pretty quickly. For example, why not cache user prompt/response pairs and use cosine distance for key lookup? You could probably find a way to do this at the edge without a whole lot of suffering. I suspect many of the prompts that hit the public API are effectively the same thing over and over every day. Why let that kind of traffic touch the expensive part of the architecture?
- dmix 4y agoIt's not like a database you can cache though, the responses are non-deterministic, the responses you get can be different the next time you query it with the exact same prompt. That's part of the point of it being generative AI (vs a question/answer system that people imagine it is).
- bob1029 4y ago> That's part of the point of it being generative AI (vs a question/answer system that people imagine it is). The point I am trying to make is that not all use cases for ChatGPT are generative. There are a lot of Q&A use cases today despite the fact that these are so far beneath its true capabilities. These items could be dealt with using more economical means. "give me a recipe for XYZ" should not require a GPU for the first turn response, much like typing in an offensive manifesto returns a boilerplate "as an AI language model..." response. Granted, if the user then types something like "please translate the recipe to Spanish and increase the amounts by 33%", we would have to reach for the generative model. But, how many real-world users are satisfied with some simple 1-turn response and go about their day?
- dmix 4y agoSure for Bing type interfaces it makes sense but if you consider what it is more holistically there are two questions: 1) do you want it to be non-deterministic and hope the first answer is the one you want everyone to see and 2) do you invalidate the entire cache every time there's a minor update to the model? How else do you tell which keys to remove? You'd basically be removing the entire cache every release
- jyu 4y agoSorry for the tangent, wondering how you've used GPT to make your life better? I'll start: - making little scripts in shell / js / python that I'm not as fluent in. 5 min vs 1-3 hours - explaining repos and apis instead of reading all the docs - help with debugging - flushing out angles for new concepts that I did not previously consider (ex: how do you make a good decentralized exchange)
- iKevinShah 4y agoYou know some people used to use Google as a glorified spell checker? I use ChatGPT as a glorified stupidity checker. What I mean is I ask it the silliest of the doubts. Like how to set environment variable in Windows (because we are all used to EXPORT aren't we?), whether or not can we do X in K8S YAML in a given conversation. Obviously I use it for other purposes as well, but it definitely has saved me a lot of hours getting the basics things right there in a prompt.
- Shorn 4y ago> explaining repos and apis instead of reading all the docs - help with debugging You can only do this with something that existed in its training set, right? There's no way to point it at a random GitHub repo and say "digest this, so I can ask you questions about it"?
- simonpacis 4y agoUsing the GPT-4 32k context model I would guess you can just paste the entire docs in there, and ask it questions about it. 32k tokens is about 24k words. https://help.openai.com/en/articles/7127966-what-is-the-difference-between-the-gpt-4-models https://help.openai.com/en/articles/7127966-what-is-the-diff...
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- Thaxll 4y agoTHey must have really large kubernetes clusters, I remember it's all running on k8s.
- nazka 4y agoThat’s so cheap when you see how huge the impact it already has.
- deleted 4y ago[deleted]
- ck2 4y agoHow many years until "chatgpt on a chip"? 3? 5?
- kristianp 4y agoAnd almost all of that on Nvidia x100 servers, for x in(A, H), it's amazing how far behind other chip companies are in deep learning.