23 ms·

OpenAI is too cheap to beat

386 points by cgwu 3y ago

eurekin 3y ago
Didn't see batching taken into equation, might skew a bit
- sidnb13 3y ago
  Yep, batching is a feature I really wish the OpenAI API had. That and the ability to intelligently cache frequently used prompts. Much easier to achieve this with a hosted OS model, so I guess it's a speed + customizability/cost tradeoff for the time being.
  - advaith08 3y ago
    imo they dont have batching because they pack sequences before passing through the model. so a single sequence in a batch on OpenAI might have requests from multiple customers in it
    - deleted 3y ago
      [deleted]
    - sidnb13 3y ago
      Ah that would make sense. Similar to vLLM which does dynamic packing.
jonplackett 3y ago
Is this a reflection of OpenAI’s massive scale making it so cheap for them? Or is it the deal with Microsoft for cloud services making it cheap? Or are they just operating at a massive loss to kill off other competition? Or something else?
- ShadowBanThis01 3y ago
  They're mining the gullible for phone numbers, among other things.
- 4death4 3y ago
  Probably all three: 1) They hiring too talent to make their models as efficient as possible. 2) They have a sweetheart deal with MS. 3) They’re better funded than everyone else and bringing in substantial revenue.
  - smachiz 3y ago
    deleted
    - ryduh 3y ago
      Is this a guess or is it informed by facts?
    - sebzim4500 3y ago
      Are just suggesting this as an option or do you have evidence that it is true?
  - ugjka 3y ago
    They are also trying to lobby the government for AI "regulation" in order limit any competitors ability achieve OpenAI's level
  - wkat4242 3y ago
    They basically are MS by now. Everyone at Microsoft I work with literally calls it an 'aquisition'. Even though they only own a share. It's pretty clear what their plans are.
    - imchillyb 3y ago
      > Microsoft will reportedly get a 75% share of OpenAI's profits until it makes back the money on its investment, after which the company would assume a 49% stake in OpenAI. 49% isn't _just_ a share, it's a significant portion of the company.
ilaksh 3y ago
I think the weird thing about this is that it's completely true right now but in X months it may be totally outdated advice. For example, efforts like OpenMOE https://github.com/XueFuzhao/OpenMoE https://github.com/XueFuzhao/OpenMoE or similar will probably eventually lead to very competitive performance and cost-effectiveness for open source models. At least in terms of competing with GPT-3.5 for many applications. Also see https://laion.ai/ https://laion.ai/ I also believe that within say 1-3 years there will be a different type of training approach that does not require such large datasets or manual human feedback.
- sidnb13 3y ago
  > I also believe that within say 1-3 years there will be a different type of training approach that does not require such large datasets or manual human feedback. I guess if we ignore pretraining, don't sample-efficient fine-tuning on carefully curated instruction datasets sort of achieve this? LIMA and OpenOrca show some really promising results to date.
  - sharemywin 3y ago
    distilbert was trained from Bert. there might be an angle using another model to train the model especially if your trying to get something to run locally.
- nico 3y ago
  > I also believe that within say 1-3 years there will be a different type of training approach that does not require such large datasets or manual human feedback This makes a lot of sense. A small model that “knows” enough English and a couple of programming languages should be enough for it to replace something like copilot, or use plug-ins or do RAG on a substantially larger dataset The issue right now is that to get a model that can do those things, the current algorithms still need massive amounts of data, way more than what the final user needs
- Dwedit 3y ago
  Abbreviate Mix of Experts as "MoE" and the Anime fans immediately start rushing in...
daft_pink 3y ago
I’m confused don’t a100s cost 10,000 to buy? Why would you pay 166k per year to rent?
- sidnb13 3y ago
  I would assume the datacenter and infra needed would also contribute a sizeable chunk to the costs when you consider upkeep to run it 24/7
  - deleted 3y ago
    [deleted]
- latchkey 3y ago
  For the same reason people use AWS. Spending the capex/opex to run a cluster of compute isn't easy or cheap. It isn't just the cost of the GPU, but the cost of everything else around it that isn't just monetary.
  - etothepii 3y ago
    This could be an interesting comparison. My experience with AWS is that it was super easy and cheap to start on. By the time we could use whole servers we were using so much AWS orchestration that it's going to be put off until we are at least $1M ARR, and probably til we are at $5M. Make adoption easy, give a free base tier but charge more could be a very effective model to get start ups stuck on you. It even probably makes adoption by small teams in big companies possible that can then grow ...
- dekhn 3y ago
  How much does an A100 consume in power a year (in dollar costs)? How much does it cost to hire and retain datacenter techs? How long does it take to expand your fleet after a user says "we're gonna need more A100s?" How many discounts can you get as a premier customer? Answer these questions, and the equation shifts a bunch!
  - 3y ago
SkyMarshal 3y ago
I think OpenAI may eventually have to go upmarket, as basic "good enough" AI becomes increasingly viable and cheap/free on consumer level devices, supplied by FOSS models and apps. Apple may be leading the way here, with Apple Silicon prioritizing AI processing and built into all their devices. These capabilities are free (or at least don't require an extra sub), and just used to sell more hardware. OpenAI is clearly going to compete in that market with its upcoming smart phone or device [1]. But what revenue model can OpenAI use to compete with Apple's and not get undercut by it? I suppose hardware + free GPT3.5, and optional subscription to GPT4 (or whatever their highest end version is). Maybe that will be competitive. I also wonder what mobile OS OpenAI will choose. Probably not Android, otherwise they would have partnered with Google. A revamped and updated Microsoft mobile OS maybe, given their MS partnership? Or something new and bespoke? I could imagine Johnny Ive demanding something new, purpose-built, and designed from scratch for a new AI-oriented UI/UX paradigm. A market for increasingly sophisticated AI that can only be done in huge GPU datacenters will exist, and that's probably where the margins will be for a long time. I think that's what OpenAI, Microsoft, Google, and the others will be increasingly competing for. [1]:https://www.reuters.com/technology/openai-jony-ive-talks-raise-1-bln-softbank-ai-device-venture-ft-2023-09-28/ https://www.reuters.com/technology/openai-jony-ive-talks-rai...
- vsreekanti 3y ago
  Yep, we agree that the obvious direction of innovation for OSS models is smaller and cheaper, likely at roughly the same quality: https://generatingconversation.substack.com/p/open-source-llms-shouldnt-try-to https://generatingconversation.substack.com/p/open-source-ll...
  - smcleod 3y ago
    Also more privacy respecting, and more customisable / flexible.
  - wrsh07 3y ago
    I actually expect open source models will be small _but larger than they are today_ because phones and laptops will get dedicated chips and software for running eg the best open source (weights?) model So eventually you could be running decent sized models locally (iOS could even provide an API with fine tuning etc)
latchkey 3y ago
I just paid the $20 for a month to try it out. In my super limited experience, GPT-4 is actually impressive and worth the money.
- smileysteve 3y ago
  I've spent the last few weeks comparing Google Duet with Chat GPT 3.5, and Chat GPT seems years ahead.
- a_wild_dandan 3y ago
  The value I get for that $20/month is astonishing. It's by far the best discretionary subscription I've ever had. That scares me. I hate moats and actively want out. Running the uncensored 70B parameter Llama 2 model on my MacBook is great, but it's just not a competitive enough general intelligence to entirely substitute for GPT-4 yet. I think our community will get there, but the surrounding water is deepening, and I'm nervous...
  - sharemywin 3y ago
    tentatively called “Claude-Next” — that is 10 times more capable than today’s most powerful AI, according to a 2023 investor deck TechCrunch obtained earlier this year. this is the thing that scare me. when do these models stop getting smarter? or at least slow down?
- matteoraso 3y ago
  Unless you're an extremely heavy user, it's cheaper to just use the API. I've been tempted to do that, but OpenAI doesn't have a free trial for me to see the quality of GPT-4 first.
  - latchkey 3y ago
    $20/mo... ~$1.3 day. I'm good with "vastly cheaper than a latte" pricing model and $20 to try something out for a month isn't bad at all.
minimaxir 3y ago
When the ChatGPT API was released 7 months ago, I posted a controversial blog post that the API was so cheap, it made other text-generating AI obsolete: https://news.ycombinator.com/item?id=35110998 https://news.ycombinator.com/item?id=35110998 7 months later, nothing's changed surprisingly. Even open-source models are trickier to get to be more cost-effective despite the many inference optimizations since. Anthropic Claude is closer to price and quality effectiveness now, but there's no reason to switch.
- cainxinth 3y ago
  These are still early days. All the major players are willing to lose billions to be top of mind with consumers in an emerging market. Either there will be some major technological breakthrough that lowers their costs, or they will all eventually start raising prices.
Eumenes 3y ago
"too cheap to beat" sounds anti-competitive and monopolistic. Large LLM providers are not dissimilar to industrial operations at scale - it requires alot of infrastructure and the more you buy/rent, the cheaper it gets. Early bird gets the worm I guess.
- stevenae 3y ago
  Not sure I understand your comment, but generally you have to prove anti-competitiveness /beyond/ too cheap to beat (unless it is a proven loss-leader which, viz all big tech companies, seems very hard to prove)
talent_deprived 3y ago
[flagged]
- roofio 3y ago
  "iamverysmart" comment. Failing to embrace it will leave you behind. Consultants that use OpenAI’s GPT-4 language model are “significantly more productive and produced significantly higher quality results” than those who do not, according to a new study from the Harvard Business School: https://aibusiness.com/nlp/harvard-study-gpt-4-boosts-work-quality-by-over-40- https://aibusiness.com/nlp/harvard-study-gpt-4-boosts-work-q...
  - talent_deprived 3y ago
    You misconstrue, I'm not failing to embrace it, I am a conscientious objector.
- vore 3y ago
  [flagged]
deleted 3y ago
[deleted]
Havoc 3y ago
Yep. Building a project that needs some LLMs. I'm very much of the self-hosting mindset so will try DIY, but it's very obviously the wrong choice by any reasonable metric. OpenAI will murder my solution by quality, by availability, by reliability and by scalability...all for the price of a coffee. It's a personal project though & partly intended for learning purposes so there is scope for accepting trainwreck level tradeoffs. No idea how commercial projects are justifying this though.
- nine_k 3y ago
  One small caveat: OpenAI gets to see all your prompts, and all the responses. Sometimes this can be unacceptable. Law,, medicine, finance, all of them would prefer a self-hosted, private GPT.
  - kevlened 3y ago
    Their data retention policy on their APIs is 30 days, and it's not used for training [0]. In addition, qualifying use cases (likely the ones you mentioned) qualify for zero data retention for most endpoints. [0] - https://platform.openai.com/docs/models/how-we-use-your-data https://platform.openai.com/docs/models/how-we-use-your-data
    - nine_k 3y ago
      In sensitive cases you do not think about the normal policy, you think about the worst case. You just can't afford a leak. Your local installation may be much better protected than a public service, by technology and by policy.
      - BoorishBears 3y ago
        For years people have essentially made a living off FUD like "ignore the literal legal agreement and imagine all the worst case scenarios!!!" to justify absolutely farcical on-premise deployments of a lot of software, but AI is starting to ruin the grift. There are some cases where you really can't afford to send Microsoft data for their OpenAI offering... but there are a lot more where some figurehead solidified their power by insisting the company build less secure versions of public offerings instead of letting their "gold" go to a 3rd party provider. As AI starts to appear as a competitive advantage, and the SOTA of self-hosted lagging so ridiculously far behind, you're seeing that work less and less. Take Harvey.ai for example: it's a frankly non-functional product and still manages to spook top law firms with tech policies that have been entrenched for decades into paying money despite being OpenAI based on the simple chance they might get outcompeted otherwise.
fulafel 3y ago
This focuses on compute capacity but wouldn't the algorithmic improvements be much more important in bang for the buck at this stage as there's so much low hanging fruit as evidenced by constant stream of news about getting better results with less hardware.
debacle 3y ago
Open source always wins, in the end. This is a fluff piece.
- downWidOutaFite 3y ago
  Where's the open source web search that is beating Google?
- paul_funyun 3y ago
  On the other hand, Libreoffice and GIMP.
serjester 3y ago
I think this is under appreciated. I run a "talk-to-your-files" website with 5ish K MRR and a pretty generous free tier. My OpenAI costs have not exceeded $200 / mo. People talk about using smaller, cheaper models but unless you have strong data security requirements you're burdening yourself with serious maintenance work and using objectively worse models to save pennies. This doesn't even consider OpenAI continuously lowering their prices. I've talked to a good amount of businesses and 90% of custom use cases would also have negligible AI costs. In my opinion, unless you're in a super regulated industry or doing genuinely cutting edge stuff, you should probably just be using the best that's available (OpenAI).
- vsreekanti 3y ago
  I completely agree — open-source models and custom deployments just can't compete with the cost and efficiency here. The only exception here is if open-source models can get way smaller and faster than they are now while maintaining existing quality. That will make private deployments and custom fine-tuning way more likely.
  - SkyMarshal 3y ago
    Or FOSS models remain the same size and speed, but hardware for running them, especially locally, steadily improves till the AI is "good enough" for a large enough segment of the market.
- hobs 3y ago
  How do you deal with the fact that Azure et al are not appearing to sell anyone additional capacity?
- jejeyyy77 3y ago
  how do ur customers feel about you uploading potentially confidential documents to a 3rd party?
  - CDSlice 3y ago
    If they are confidential they probably shouldn’t be uploaded to any website no matter if it calls out to OpenAI or does all the processing on their own servers.
zzbn00 3y ago
p4d.24xlarge spot price is $8.2 / hour in US East 1 at the moment...
- deleted 3y ago
  [deleted]
- charlesischuck 3y ago
  Good luck getting that lol
- saminaustin 3y ago
  Can't run a business on spot availability.
tester756 3y ago
>iPhone of artificial intelligence It feels like the biggest investor bait of this year Will it beat ARM IPO?
lossolo 3y ago
It's also worth noting that if you build your business on using OpenAI's LLM or Anthropic etc, then, in the majority of cases I've seen so far (no fine tuning etc), your competitor is just one prompt away from replicating your business.
beauHD 3y ago
I signed up for OpenAI's ChatGPT tool, and entered a query, like 'What does the notation 1e100 mean?' (just to try it out). And then when displaying the output it would start outputting the reply in a slow way, like, it was dripfeeded to me, and I was like: 'what? surely this could be faster?' Maybe I'm missing something crucial here, but why does it dripfeed answers like this? Does it have to think really hard about the meaning of 1e100? Why can't it just spit it out instantly without such a delay/drip, like with the near-instant Wolfram Alpha?
- baby 3y ago
  You can but it’ll take longer. So one way to get faster answers is to stream the response as it is generated. And in GPT-based apps the response is generated token by token (~4chars), hence what you’re seeing.
- maccam912 3y ago
  Its a result of how these transformer models work. It's pretty quick for the amount of work it does, but it's not looking up anything, it's generating it a token a time.
- notRobot 3y ago
  Under the hood, GPT works by predicting the next token when provided with an input sequence of words. At each step a single word is generated taking into consideration all the previous words. https://ai.stackexchange.com/questions/38923/why-does-chatgpt-not-give-the-answer-text-all-at-once https://ai.stackexchange.com/questions/38923/why-does-chatgp...
- swatcoder 3y ago
  The non-technical way to think about it is that ChatGPT “thinks out loud” and can only “think out loud”. Future products would be able to hide some of that, but for now, that’s what the ChatGPT / Bing Assistant product does.
- codedokode 3y ago
  Because it needs to do billions of arithmetic operations to generate a reply. Replying to questions is not an easy task.
iambateman 3y ago
This is _the_ playbook for big, fast scaling companies...Uber subsidized every ride for _a decade_ before finally charging market price, just to make sure that Uber was the only option which made sense. While it's nice to consume the cheap stuff, it is not good for healthy markets.
matteoraso 3y ago
It's not even just the cost of finetuning. The API pricing is so low, you literally can't save money by buying a GPU and running your own LLM, no matter how many tokens you generate. It's an incredible moat for OpenAI, but something they can't provide is an LLM that doesn't talk like an annoying HR manager, which is the real use case for self-hosting.
rosywoozlechan 3y ago
The service quality sucks. You're getting what you pay for. We switched to Azure Open AI APIs because of all the service quality issues.
layer8 3y ago
Isn’t OpenAI too cheap to be sustainable, and currently living off Microsoft’s $10B investment?
xnx 3y ago
Nothing in that article convinces me the situation couldn't change entirely in any given month. Google Gemini could be more capable. Any number of new players (AWS, Microsoft, Apple) could enter the market in a serious way. The head-start OpenAI has in usage data is small and probably eclipsed by the clickstream and data stores that Google and Microsoft have access to. I see no durable advantage for OpenAI.
- freedomben 3y ago
  Gemini very well might be the biggest threat to OpenAI. ChatGPT has first-mover advantage so has a decent moat, but the amount of people willing to pay $20 per month for something worse[1] than they get for free with google.com is going to dwindle. I'd be very worried if I were them. [1]: That knowledge cutoff and terrible UX of browse the web is brutal compared to the experience of Bard
appplication 3y ago
The premise of this is flawed. OpenAI is cheap because of has to be right now. They need to establish market dominance quickly, before competitors slide in. The winner of this horse race is not going to be the company with the best performing AI, it’s going to be the one who does the best job at creating an outstanding UX, ubiquitously presence, entrenching users, and building competitive moats that are not feature differentiated because at best even cutting edge features are only 6-12 months ahead of competition cloning or beating. This is Uber/AirBnB/Wework/literally every VC subsidized hungry-hungry-hippos market grab all over again. If you’re falling in love because the prices are so low, that is ephemeral at best and is not a moat. Someone try calling an Uber in SF today and tell me how much that costs you and how much worse the experience is vs 2017. OpenAI is the undisputed future of AI… for timescales 6 months and less. They are still extremely vulnerable to complete disruption and as likely to be the next MySpace as they are Facebook.
- shaburn 3y ago
  Your Uber/AirBnB/Wework all have physical base units with ascending costs due to inflation and theoretical economies of scale. AI models have some GPU constraints but could easily reach a state where the cost to opperate falls and becomes relatively trivial with almost no lowerbound, for most use cases. You are correct there is a race for marketshare. The crux in this case will be keeping it. Easy come, easy go. Models often make the worst business model.
  - monocasa 3y ago
    Probably why Altman has been talking so much about how dangerous it is and how regulations are needed. No natural moat, so building a regulatory one.
    - shaburn 3y ago
      Only hundreds of billions at stake if successful.
- blackoil 3y ago
  This point is discussed in the article. Title is not for Google/Meta, they'll invest all the billions that they have to. It is for the consumers of these models, is there even a point to train your own or experiment with OSS!
kcorbitt 3y ago
Eh, OpenAI is too cheap to beat at their own game. But there are a ton of use-cases where a 1 to 7B parameter fine-tuned model will be faster, cheaper and easier to deploy than a prompted or fine-tuned GPT-3.5-sized model. In fact, it might be a strong statement but I'd argue that most current use-cases for (non-fine-tuned) GPT-3.5 fit in that bucket. (Disclaimer: currently building https://openpipe.ai https://openpipe.ai; making it trivial for product engineers to replace OpenAI prompts with their own fine-tuned models.)
kristjansson 3y ago
This article might have a point about the data flywheel, but it's lost in the confused economics in the second half. Why would we expect to hire one engineer per p4.24x instance? Why do we think OpenAI needs a whole p4.24x to run fine tuning? Why do we ignore the higher costs on the inference side for fine-tuned models? Why do we think OpenAI spends _any_ money on racking-and-stacking GPUs rather than just take them at (hyperscaler) cost from Azure?
oceanplexian 3y ago
Has anyone actually used GPT4? It's not "cheap". It was roughly $150 for me to build a small dataset with a few thousand quarter-page chunks of text for a data project using GPT4. GPT3 is substantially cheaper but it would hallucinate 30% of the time; honestly a nice fine-tune of LlaMA is on-par with GPT3 and after the sunk cost all it costs is a few $0.01 in electricity to generate the same sized dataset.
- slowhadoken 3y ago
  It's insanely expensive to run and operate "AI". Meredith Whittaker's talk on AI is very insightful https://www.youtube.com/watch?v=amNriUZNP8w https://www.youtube.com/watch?v=amNriUZNP8w
- redox99 3y ago
  Even GPT3.5 can be much more expensive. In some specific tasks, a finetuned 7B llama can work as well as GPT3.5. You can rent a 3090 at $0.20/h on vast.ai, or $0.40/h on runpod. Using VLLM at 400t/s that's 1440000 generated tokens. Generating that amount of tokens with GPT3.5 would be $2.88.
  - agnokapathetic 3y ago
    > "In some specific tasks, a finetuned 7B llama can work as well as GPT3.5." "some" is doing a lot of heavy lifting here. Also: don't discount the labor cost of curating a fine tuning dataset, running a FT training run, even if the hardware is cheap.
  - moffkalast 3y ago
    Unless your use case isn't in English in which case LLama is as useful as a one-legged man in an ass-kicking contest. LLama models only really shine for things that GPTs would refuse to even consider because of corporate RLHF, and if you need to keep your data local I suppose. For the rest they're second rate at best.
- deleted 3y ago
  [deleted]
slowhadoken 3y ago
Thanks to traumatized $2 an hour Kenyan labor, yeah https://time.com/6247678/openai-chatgpt-kenya-workers/ https://time.com/6247678/openai-chatgpt-kenya-workers/
- hansoolo 3y ago
  That is disturbing. I didn't know about that dark side to it. Still thx for posting.
  - slowhadoken 3y ago
    Meredith Whittaker talks about it in this interview https://www.youtube.com/watch?v=amNriUZNP8w https://www.youtube.com/watch?v=amNriUZNP8w
pimpampum 3y ago
Classic anti-competition strategy, sell below cost and burn money until competition is out, then sell higher than you could have ever sold with competition.
BrunoJo 3y ago
We just started a service different open source models and with an OpenAI compatible API [1]. The pricing isn't final and we haven't officially launched yet but you should be able to save at least 75% compared to GPT 3.5. [1] https://lemonfox.ai/ https://lemonfox.ai/
- Meegul 3y ago
  Are you doing this profitably? If so, does that entail owning your own hardware or renting from cheaper services such as Lambda?
- Cholical 3y ago
  Hey BrunoJo, saw your posts on a couple of threads. Love what you're doing at lemonfox! Do you have any troubles with finding cheap GPUs to host models on? If so, I'm working on service that provides a single API and UI for launching cloud GPUs across 10 different cloud providers so you can always find available gpus. Let me know if this might be useful for you!
slowhadoken 3y ago
none of it is cheap, "AI" insanely expensive. Meredith Whittaker talks about it in this interview https://www.youtube.com/watch?v=amNriUZNP8w https://www.youtube.com/watch?v=amNriUZNP8w She's the president of the Signal Foundation.
AJRF 3y ago
I read this and think "That won't last long". The pricing is too good to be true with you think about it rationally. If they raise prices they seem much, much less attractive than using AWS or Azure. Amazon seem to have a much better business built around their Bedrock offering. And all their other tools are available there like SageMaker, ec2, integration with MLFlow, etc, etc. I guess the same goes for Azure, if you are already using it it's much easier to just stick with whatever they are offering for LLM Ops. OpenAI offering just models doesn't seem like it can last forever, and to compete with AWS or Azure at enterprise level they need to build all the things Amazon/MS have built. The other side of that coin seems much more realistic.
- BoorishBears 3y ago
  > The pricing is too good to be true with you think about it rationally In what way shape or form? > If they raise prices they seem much, much less attractive than using AWS or Azure. They're already significantly more expensive than Azure. OpenAI charges something like $30k a month for dedicated capacity on a "call our sales team" basis: GPT 3.5/ GPT-4 on Azure comes with that for free. And GPT-4 is already slow and expensive enough that no one just chooses it arbitrarily... they're using it for things no other model can do. They could charge double for GPT-4 and GPT-4 would still be the only model that can do those tasks: you wouldn't get to just switch off to some other GPT-4 equivalent provider. > Amazon seem to have a much better business built around their Bedrock offering Amazon is literally doing the same thing with Bedrock! They're offering Anthropic at competitive prices to OpenAI for a model that's no cheaper to run based on their own dedicated capacity numbers. > OpenAI offering just models doesn't seem like it can last forever, and to compete with AWS or Azure at enterprise level they need to build all the things Amazon/MS have built. OpenAI is not trying to become Azure: They actively go out of their way to hide the fact they even offer half the things they offer to enterprises, instead relying on Azure absorbing demand as much as possible. OpenAI wants ChatGPT Plus to be the new Prime, as in no one should be able to afford to not pay OpenAI for their immensely valuable offering. Except unlike Prime, the offering is software, not commerce: If Amazon could get AWS-like margins from their e-commerce business, AWS would be a footnote.
DominikPeters 3y ago
> While per-token inference costs for fine-tuned GPT-3.5 is 10x more expensive than GPT-3.5 it is still 10x cheaper than GPT-4! Not quite accurate; finetuned 3.5 is only 4x cheaper than GPT-4. Cost per million output tokens from https://openai.com/pricing https://openai.com/pricing $ 2 - GPT-3.5 $16 - finetuned GPT-3.5 $60 - GPT 4
leeeeeepw 3y ago
[dead]
singingfish 3y ago
These LLMs are confabulation machines. They're as good as the knowledge of the person who is driving the current session. For coding problems this makes them like a very good teddy bear debugger, but do not expect independently produced creative work from them.
jongjong 3y ago
The main flaw I see in the argument is that assumes that AIs should be generalists... But if you look at the reality of our current human economy, you will notice that generalists cannot even get jobs; the economy needs specialists. I think the same will happen with AI; companies will need specialist AIs, not generalists. And since there are many different industries/specializations with a lot of nuanced, undocumented knowledge which is not available online, it will be difficult for a single large company to acquire all that specialist information. I think the bottleneck isn't going to be hardware costs, but merely putting together the optimal training data. To do this, you need to find the top experts in the world in any given field. Unfortunately, it's difficult to do right now because top experts are often not given credit these days. Those who are promoted as the top people in any given field are often mostly good at politics and lack the deep nuanced knowledge that would be required to produce top quality training data.
stainablesteel 3y ago
they've technically been in this game for much longer than anyone else, of course they're more prepared. a lot of the competition popped up overnight on the amazing prospects they demonstrated.
karmasimida 3y ago
They are subsiding the users with their massive cash reserve. And it is the right business strategy for them. It is MSFT money, it is free money.
renewiltord 3y ago
Would easily pay $100 for GPT-4, perhaps $250, maybe at a pinch even $500, but there's no way they can scale at that pricing.
hintymad 3y ago
I'll assume that OpenAI does not offer $1 of service for less than $1, at least not after certain scale of economy. If that assumption is true, then OpenAI goes back to the core of the valley: building insanely great product that solves insanely hard problems and that people are willing to buy on the merits of the products.
winddude 3y ago
not really. It's general purpose, running llama2 70b has been shown to work out to be cheaper if you have a high usage rate. And depending on use case with fine tuning can likely achieve far superior results. The one thing openAI has that's hard to compete with is a metric shit ton of money and brand recognition.
DonHopkins 3y ago
Generative AI gives Degenerative AI a bad name.
imchillyb 3y ago
OpenAI is seeking to make their own chips and hardware. If they can pull off what Apple has done with its M line, then possibly they can make themselves even more cost effective than the competition. The competition will mostly be limited to supplies from Nvidia. I believe in house manufacturing of their own hardware is definitely the way forward. Top down lock down. Own the hardware, own the models, own the user trained datasets.
AndrewKemendo 3y ago
I will abandon ChatGPT, Claude etc... the millisecond that Siri has the same capabilities if for no other reason than it will take me two extra steps to use it and pay for it. Every voice assistant will unquestionably and inevitably implement GPTs eventually. So it's simply back to the question of who has the distribution of chatbots. I trust Apple 1000x more than basically any other tech company (not saying much) so I'm just waiting for that now. OpenAI can't "win" (whatever that means) long term unless they figure out how to collect user data inputs (text based questions) and reward vectors (Thumbs up and down) persistently, at scale, in the extreme long term - which means building something people rely on all day everyday. As far as I can tell they have no unique distribution avenues to do this today outside of copilot. Meanwhile, Apple, Amazon and Alphabet are certainly bringing GPT capabilities to Siri/Alexa/Whatever Google's voice thing is called, albeit slower and more carefully, but they have no need to rush at all here. I'll bet Microsoft will slowly absorb OpenAI given their investment position and integrate it into Bing or something and fade away.
kristopolous 3y ago
That was said about uber ... it's just being subsidized by vc. Wait until they expect a profit huggingface will be more of a winner in the long run with an exit via an MSFT acquisition if I had to call it.
- BoorishBears 3y ago
  This is like Uber if self-driving cars had simply been a matter of making a faster GPU. OpenAI gets cheaper by twiddling their thumbs for the next few years, meanwhile they continue to amass more and more data for RLHF. It's weird that people are trying to drag non-software scaling into a software scaling problem: Lyft, Doordash, Instacart, etc. all relied on VC dollars to scale non-software growth like software. OpenAI really is just stupidly cheap compared to anything those past high CAC plays were aiming to do.
  - kristopolous 3y ago
    ok so you'll have to help me here, I'm still learning this stuff. RLHF I looked it up. Is this really useful? The average human has zero general expertise because people are specialized (I know nothing about say, 1960s avant garde french cinema and my responses in a conversation there would be garbage - given the breadth of human knowledge even the most accomplished scholars are useless for over 99% of it). Won't there be a quality decrease? How is this accommodated for? If the chat systems simply gave the most popular answers it would cease to be useful real fast.
    - BoorishBears 3y ago
      RLHF isn't used to teach the model what it knows, it's used to teach the model how to follow instructions Before RLHF instruct tuning the models could only complete sentences Technically they still complete sentences, but now they have a strong association for a format where a question is followed by an answer
rnikander 3y ago
I've haven't looked into how these new AI products are implemented. Can someone give a ballpark estimate on the current costs if someone wanted to build their own, say: 1. LLM that talks like ChatGPT 2. Image generator that makes realistic portraits from verbal descriptions. Are the costs in the data acquisition, human training input, training CPU/GPU hours, hardware, or ??
parentheses 3y ago
The comparison here is not apples to apples. While fine tuning is less costly with OpenAI, I'd argue that running inference using GPT3.5 vs a fine tuned model should be roughly the same. OpenAI is gouging you on inference, thereby being able to offer fine tuning at a seemingly reasonable price. Also, it's important to note that fine tuning produces a vast amount of data about use cases where fine tuning is useful.
awill88 3y ago
Why the gimme-your-email-to-subscribe wall?
jojobas 3y ago
OpenAI products are so censored and crippled they could pay me and I'd still prefer something else.
j-a-a-p 3y ago
Common sense let me think you need to calculate the value of the data that OpenAI receives. But the article quickly goes into a comparison between OpenAI, which is highly optimised for the AI workload versus the pricing of an instance at AWS. AWS, renowned for their hefty cost and in the center of the cloud repatriation movement. That alone kills the argument that OpenAI is 'too cheap'.
sinuhe69 3y ago
So how could OpenAI make fine-tuning so cheap? Beside the computing power you need for fine-tuning, OpenAI also have to save your fine-tuned weights, and spawn a new instance for you every time you want to run your fine-tuned model. If so, I imagine it must be insanely expensive because the inference only model of ChatGPT is not what one can say "small". So, how did they do that? Can anyone share their "secret"?
miniwa 3y ago
Disagree with this. Model quality is THE factor keeping them ahead. I think a comparison to Netflix and the media industry is fair. Years ago, people claimed Netflix's tech and infrastructure was their moat, but it turns out it was the cheap easy access to loads of high quality content that was the real motivator.
ma9o 3y ago
Unless you have a massively parallel use case and a small, specialized model (which seems the general direction of the industry anyway)
mark_l_watson 3y ago
Great HN thread! I think it is close to impossible to predict where the market for AI and LLMs in particular will be in two years. The major players are making their best bets. For the value of frontier LLMs, the article by Blaise Agüera y Arcas and Peter Norvig making the case that we might already have AGI: https://www.noemamag.com/artificial-general-intelligence-is-already-here/ https://www.noemamag.com/artificial-general-intelligence-is-... I use both OpenAI and Anthropic (I use my own Common Lisp and Racket Scheme client libraries that I implement with similar APIs) and I was amazed last night how well self hosted LLama-based and Mistral LLMs run on a 32G Mac Mini that was delivered to me late yesterday afternoon. This is not expensive hardware. And tools like llama.cpp will keep getting more efficient, etc. We are going to see unimaginable (at least to me) advances in AI and AGI, at all levels of the tech food chain. And these advances will occur quickly. Place your bets, and remain flexible!
jurschreuder 3y ago
Totally flawed calculation. For many tasks Llama 70b works just fine and you can run that on a 6000e server as much as you like. Renting GPU servers at AWS is just stupid. Did you ever see bitcoin miners calculate the price of mining one bitcoin by looking at renting AWS GPU servers?

386 points by cgwu 3y ago

eurekin 3y ago

Didn't see batching taken into equation, might skew a bit

sidnb13 3y ago
Yep, batching is a feature I really wish the OpenAI API had. That and the ability to intelligently cache frequently used prompts. Much easier to achieve this with a hosted OS model, so I guess it's a speed + customizability/cost tradeoff for the time being.
- advaith08 3y ago
  imo they dont have batching because they pack sequences before passing through the model. so a single sequence in a batch on OpenAI might have requests from multiple customers in it
  - deleted 3y ago
    [deleted]
  - sidnb13 3y ago
    Ah that would make sense. Similar to vLLM which does dynamic packing.

jonplackett 3y ago

Is this a reflection of OpenAI’s massive scale making it so cheap for them? Or is it the deal with Microsoft for cloud services making it cheap? Or are they just operating at a massive loss to kill off other competition? Or something else?

ShadowBanThis01 3y ago
They're mining the gullible for phone numbers, among other things.
4death4 3y ago
Probably all three: 1) They hiring too talent to make their models as efficient as possible. 2) They have a sweetheart deal with MS. 3) They’re better funded than everyone else and bringing in substantial revenue.
- smachiz 3y ago
  deleted
  - ryduh 3y ago
    Is this a guess or is it informed by facts?
  - sebzim4500 3y ago
    Are just suggesting this as an option or do you have evidence that it is true?
- ugjka 3y ago
  They are also trying to lobby the government for AI "regulation" in order limit any competitors ability achieve OpenAI's level
- wkat4242 3y ago
  They basically are MS by now. Everyone at Microsoft I work with literally calls it an 'aquisition'. Even though they only own a share. It's pretty clear what their plans are.
  - imchillyb 3y ago
    > Microsoft will reportedly get a 75% share of OpenAI's profits until it makes back the money on its investment, after which the company would assume a 49% stake in OpenAI. 49% isn't _just_ a share, it's a significant portion of the company.

ilaksh 3y ago

I think the weird thing about this is that it's completely true right now but in X months it may be totally outdated advice. For example, efforts like OpenMOE https://github.com/XueFuzhao/OpenMoE https://github.com/XueFuzhao/OpenMoE or similar will probably eventually lead to very competitive performance and cost-effectiveness for open source models. At least in terms of competing with GPT-3.5 for many applications. Also see https://laion.ai/ https://laion.ai/ I also believe that within say 1-3 years there will be a different type of training approach that does not require such large datasets or manual human feedback.

sidnb13 3y ago
> I also believe that within say 1-3 years there will be a different type of training approach that does not require such large datasets or manual human feedback. I guess if we ignore pretraining, don't sample-efficient fine-tuning on carefully curated instruction datasets sort of achieve this? LIMA and OpenOrca show some really promising results to date.
- sharemywin 3y ago
  distilbert was trained from Bert. there might be an angle using another model to train the model especially if your trying to get something to run locally.
nico 3y ago
> I also believe that within say 1-3 years there will be a different type of training approach that does not require such large datasets or manual human feedback This makes a lot of sense. A small model that “knows” enough English and a couple of programming languages should be enough for it to replace something like copilot, or use plug-ins or do RAG on a substantially larger dataset The issue right now is that to get a model that can do those things, the current algorithms still need massive amounts of data, way more than what the final user needs
Dwedit 3y ago
Abbreviate Mix of Experts as "MoE" and the Anime fans immediately start rushing in...

daft_pink 3y ago

I’m confused don’t a100s cost 10,000 to buy? Why would you pay 166k per year to rent?

sidnb13 3y ago
I would assume the datacenter and infra needed would also contribute a sizeable chunk to the costs when you consider upkeep to run it 24/7
- deleted 3y ago
  [deleted]
latchkey 3y ago
For the same reason people use AWS. Spending the capex/opex to run a cluster of compute isn't easy or cheap. It isn't just the cost of the GPU, but the cost of everything else around it that isn't just monetary.
- etothepii 3y ago
  This could be an interesting comparison. My experience with AWS is that it was super easy and cheap to start on. By the time we could use whole servers we were using so much AWS orchestration that it's going to be put off until we are at least $1M ARR, and probably til we are at $5M. Make adoption easy, give a free base tier but charge more could be a very effective model to get start ups stuck on you. It even probably makes adoption by small teams in big companies possible that can then grow ...
dekhn 3y ago
How much does an A100 consume in power a year (in dollar costs)? How much does it cost to hire and retain datacenter techs? How long does it take to expand your fleet after a user says "we're gonna need more A100s?" How many discounts can you get as a premier customer? Answer these questions, and the equation shifts a bunch!
- 3y ago

SkyMarshal 3y ago

I think OpenAI may eventually have to go upmarket, as basic "good enough" AI becomes increasingly viable and cheap/free on consumer level devices, supplied by FOSS models and apps. Apple may be leading the way here, with Apple Silicon prioritizing AI processing and built into all their devices. These capabilities are free (or at least don't require an extra sub), and just used to sell more hardware. OpenAI is clearly going to compete in that market with its upcoming smart phone or device [1]. But what revenue model can OpenAI use to compete with Apple's and not get undercut by it? I suppose hardware + free GPT3.5, and optional subscription to GPT4 (or whatever their highest end version is). Maybe that will be competitive. I also wonder what mobile OS OpenAI will choose. Probably not Android, otherwise they would have partnered with Google. A revamped and updated Microsoft mobile OS maybe, given their MS partnership? Or something new and bespoke? I could imagine Johnny Ive demanding something new, purpose-built, and designed from scratch for a new AI-oriented UI/UX paradigm. A market for increasingly sophisticated AI that can only be done in huge GPU datacenters will exist, and that's probably where the margins will be for a long time. I think that's what OpenAI, Microsoft, Google, and the others will be increasingly competing for. [1]:https://www.reuters.com/technology/openai-jony-ive-talks-raise-1-bln-softbank-ai-device-venture-ft-2023-09-28/ https://www.reuters.com/technology/openai-jony-ive-talks-rai...

vsreekanti 3y ago
Yep, we agree that the obvious direction of innovation for OSS models is smaller and cheaper, likely at roughly the same quality: https://generatingconversation.substack.com/p/open-source-llms-shouldnt-try-to https://generatingconversation.substack.com/p/open-source-ll...
- smcleod 3y ago
  Also more privacy respecting, and more customisable / flexible.
- wrsh07 3y ago
  I actually expect open source models will be small _but larger than they are today_ because phones and laptops will get dedicated chips and software for running eg the best open source (weights?) model So eventually you could be running decent sized models locally (iOS could even provide an API with fine tuning etc)

latchkey 3y ago

I just paid the $20 for a month to try it out. In my super limited experience, GPT-4 is actually impressive and worth the money.

smileysteve 3y ago
I've spent the last few weeks comparing Google Duet with Chat GPT 3.5, and Chat GPT seems years ahead.
a_wild_dandan 3y ago
The value I get for that $20/month is astonishing. It's by far the best discretionary subscription I've ever had. That scares me. I hate moats and actively want out. Running the uncensored 70B parameter Llama 2 model on my MacBook is great, but it's just not a competitive enough general intelligence to entirely substitute for GPT-4 yet. I think our community will get there, but the surrounding water is deepening, and I'm nervous...
- sharemywin 3y ago
  tentatively called “Claude-Next” — that is 10 times more capable than today’s most powerful AI, according to a 2023 investor deck TechCrunch obtained earlier this year. this is the thing that scare me. when do these models stop getting smarter? or at least slow down?
matteoraso 3y ago
Unless you're an extremely heavy user, it's cheaper to just use the API. I've been tempted to do that, but OpenAI doesn't have a free trial for me to see the quality of GPT-4 first.
- latchkey 3y ago
  $20/mo... ~$1.3 day. I'm good with "vastly cheaper than a latte" pricing model and $20 to try something out for a month isn't bad at all.

minimaxir 3y ago

When the ChatGPT API was released 7 months ago, I posted a controversial blog post that the API was so cheap, it made other text-generating AI obsolete: https://news.ycombinator.com/item?id=35110998 https://news.ycombinator.com/item?id=35110998 7 months later, nothing's changed surprisingly. Even open-source models are trickier to get to be more cost-effective despite the many inference optimizations since. Anthropic Claude is closer to price and quality effectiveness now, but there's no reason to switch.

cainxinth 3y ago
These are still early days. All the major players are willing to lose billions to be top of mind with consumers in an emerging market. Either there will be some major technological breakthrough that lowers their costs, or they will all eventually start raising prices.

Eumenes 3y ago

"too cheap to beat" sounds anti-competitive and monopolistic. Large LLM providers are not dissimilar to industrial operations at scale - it requires alot of infrastructure and the more you buy/rent, the cheaper it gets. Early bird gets the worm I guess.

stevenae 3y ago
Not sure I understand your comment, but generally you have to prove anti-competitiveness /beyond/ too cheap to beat (unless it is a proven loss-leader which, viz all big tech companies, seems very hard to prove)

talent_deprived 3y ago

[flagged]

roofio 3y ago
"iamverysmart" comment. Failing to embrace it will leave you behind. Consultants that use OpenAI’s GPT-4 language model are “significantly more productive and produced significantly higher quality results” than those who do not, according to a new study from the Harvard Business School: https://aibusiness.com/nlp/harvard-study-gpt-4-boosts-work-quality-by-over-40- https://aibusiness.com/nlp/harvard-study-gpt-4-boosts-work-q...
- talent_deprived 3y ago
  You misconstrue, I'm not failing to embrace it, I am a conscientious objector.
vore 3y ago
[flagged]

deleted 3y ago

[deleted]

Havoc 3y ago

Yep. Building a project that needs some LLMs. I'm very much of the self-hosting mindset so will try DIY, but it's very obviously the wrong choice by any reasonable metric. OpenAI will murder my solution by quality, by availability, by reliability and by scalability...all for the price of a coffee. It's a personal project though & partly intended for learning purposes so there is scope for accepting trainwreck level tradeoffs. No idea how commercial projects are justifying this though.

nine_k 3y ago
One small caveat: OpenAI gets to see all your prompts, and all the responses. Sometimes this can be unacceptable. Law,, medicine, finance, all of them would prefer a self-hosted, private GPT.
- kevlened 3y ago
  Their data retention policy on their APIs is 30 days, and it's not used for training [0]. In addition, qualifying use cases (likely the ones you mentioned) qualify for zero data retention for most endpoints. [0] - https://platform.openai.com/docs/models/how-we-use-your-data https://platform.openai.com/docs/models/how-we-use-your-data
  - nine_k 3y ago
    In sensitive cases you do not think about the normal policy, you think about the worst case. You just can't afford a leak. Your local installation may be much better protected than a public service, by technology and by policy.
    - BoorishBears 3y ago
      For years people have essentially made a living off FUD like "ignore the literal legal agreement and imagine all the worst case scenarios!!!" to justify absolutely farcical on-premise deployments of a lot of software, but AI is starting to ruin the grift. There are some cases where you really can't afford to send Microsoft data for their OpenAI offering... but there are a lot more where some figurehead solidified their power by insisting the company build less secure versions of public offerings instead of letting their "gold" go to a 3rd party provider. As AI starts to appear as a competitive advantage, and the SOTA of self-hosted lagging so ridiculously far behind, you're seeing that work less and less. Take Harvey.ai for example: it's a frankly non-functional product and still manages to spook top law firms with tech policies that have been entrenched for decades into paying money despite being OpenAI based on the simple chance they might get outcompeted otherwise.

fulafel 3y ago

This focuses on compute capacity but wouldn't the algorithmic improvements be much more important in bang for the buck at this stage as there's so much low hanging fruit as evidenced by constant stream of news about getting better results with less hardware.

debacle 3y ago

Open source always wins, in the end. This is a fluff piece.

downWidOutaFite 3y ago
Where's the open source web search that is beating Google?
paul_funyun 3y ago
On the other hand, Libreoffice and GIMP.

serjester 3y ago

I think this is under appreciated. I run a "talk-to-your-files" website with 5ish K MRR and a pretty generous free tier. My OpenAI costs have not exceeded $200 / mo. People talk about using smaller, cheaper models but unless you have strong data security requirements you're burdening yourself with serious maintenance work and using objectively worse models to save pennies. This doesn't even consider OpenAI continuously lowering their prices. I've talked to a good amount of businesses and 90% of custom use cases would also have negligible AI costs. In my opinion, unless you're in a super regulated industry or doing genuinely cutting edge stuff, you should probably just be using the best that's available (OpenAI).

vsreekanti 3y ago
I completely agree — open-source models and custom deployments just can't compete with the cost and efficiency here. The only exception here is if open-source models can get way smaller and faster than they are now while maintaining existing quality. That will make private deployments and custom fine-tuning way more likely.
- SkyMarshal 3y ago
  Or FOSS models remain the same size and speed, but hardware for running them, especially locally, steadily improves till the AI is "good enough" for a large enough segment of the market.
hobs 3y ago
How do you deal with the fact that Azure et al are not appearing to sell anyone additional capacity?
jejeyyy77 3y ago
how do ur customers feel about you uploading potentially confidential documents to a 3rd party?
- CDSlice 3y ago
  If they are confidential they probably shouldn’t be uploaded to any website no matter if it calls out to OpenAI or does all the processing on their own servers.

zzbn00 3y ago

p4d.24xlarge spot price is $8.2 / hour in US East 1 at the moment...

deleted 3y ago
[deleted]
charlesischuck 3y ago
Good luck getting that lol
saminaustin 3y ago
Can't run a business on spot availability.

tester756 3y ago

>iPhone of artificial intelligence It feels like the biggest investor bait of this year Will it beat ARM IPO?

lossolo 3y ago

It's also worth noting that if you build your business on using OpenAI's LLM or Anthropic etc, then, in the majority of cases I've seen so far (no fine tuning etc), your competitor is just one prompt away from replicating your business.

beauHD 3y ago

I signed up for OpenAI's ChatGPT tool, and entered a query, like 'What does the notation 1e100 mean?' (just to try it out). And then when displaying the output it would start outputting the reply in a slow way, like, it was dripfeeded to me, and I was like: 'what? surely this could be faster?' Maybe I'm missing something crucial here, but why does it dripfeed answers like this? Does it have to think really hard about the meaning of 1e100? Why can't it just spit it out instantly without such a delay/drip, like with the near-instant Wolfram Alpha?

baby 3y ago
You can but it’ll take longer. So one way to get faster answers is to stream the response as it is generated. And in GPT-based apps the response is generated token by token (~4chars), hence what you’re seeing.
maccam912 3y ago
Its a result of how these transformer models work. It's pretty quick for the amount of work it does, but it's not looking up anything, it's generating it a token a time.
notRobot 3y ago
Under the hood, GPT works by predicting the next token when provided with an input sequence of words. At each step a single word is generated taking into consideration all the previous words. https://ai.stackexchange.com/questions/38923/why-does-chatgpt-not-give-the-answer-text-all-at-once https://ai.stackexchange.com/questions/38923/why-does-chatgp...
swatcoder 3y ago
The non-technical way to think about it is that ChatGPT “thinks out loud” and can only “think out loud”. Future products would be able to hide some of that, but for now, that’s what the ChatGPT / Bing Assistant product does.
codedokode 3y ago
Because it needs to do billions of arithmetic operations to generate a reply. Replying to questions is not an easy task.

iambateman 3y ago

This is _the_ playbook for big, fast scaling companies...Uber subsidized every ride for _a decade_ before finally charging market price, just to make sure that Uber was the only option which made sense. While it's nice to consume the cheap stuff, it is not good for healthy markets.

matteoraso 3y ago

It's not even just the cost of finetuning. The API pricing is so low, you literally can't save money by buying a GPU and running your own LLM, no matter how many tokens you generate. It's an incredible moat for OpenAI, but something they can't provide is an LLM that doesn't talk like an annoying HR manager, which is the real use case for self-hosting.

rosywoozlechan 3y ago

The service quality sucks. You're getting what you pay for. We switched to Azure Open AI APIs because of all the service quality issues.

layer8 3y ago

Isn’t OpenAI too cheap to be sustainable, and currently living off Microsoft’s $10B investment?

xnx 3y ago

Nothing in that article convinces me the situation couldn't change entirely in any given month. Google Gemini could be more capable. Any number of new players (AWS, Microsoft, Apple) could enter the market in a serious way. The head-start OpenAI has in usage data is small and probably eclipsed by the clickstream and data stores that Google and Microsoft have access to. I see no durable advantage for OpenAI.

freedomben 3y ago
Gemini very well might be the biggest threat to OpenAI. ChatGPT has first-mover advantage so has a decent moat, but the amount of people willing to pay $20 per month for something worse[1] than they get for free with google.com is going to dwindle. I'd be very worried if I were them. [1]: That knowledge cutoff and terrible UX of browse the web is brutal compared to the experience of Bard

appplication 3y ago

The premise of this is flawed. OpenAI is cheap because of has to be right now. They need to establish market dominance quickly, before competitors slide in. The winner of this horse race is not going to be the company with the best performing AI, it’s going to be the one who does the best job at creating an outstanding UX, ubiquitously presence, entrenching users, and building competitive moats that are not feature differentiated because at best even cutting edge features are only 6-12 months ahead of competition cloning or beating. This is Uber/AirBnB/Wework/literally every VC subsidized hungry-hungry-hippos market grab all over again. If you’re falling in love because the prices are so low, that is ephemeral at best and is not a moat. Someone try calling an Uber in SF today and tell me how much that costs you and how much worse the experience is vs 2017. OpenAI is the undisputed future of AI… for timescales 6 months and less. They are still extremely vulnerable to complete disruption and as likely to be the next MySpace as they are Facebook.

shaburn 3y ago
Your Uber/AirBnB/Wework all have physical base units with ascending costs due to inflation and theoretical economies of scale. AI models have some GPU constraints but could easily reach a state where the cost to opperate falls and becomes relatively trivial with almost no lowerbound, for most use cases. You are correct there is a race for marketshare. The crux in this case will be keeping it. Easy come, easy go. Models often make the worst business model.
- monocasa 3y ago
  Probably why Altman has been talking so much about how dangerous it is and how regulations are needed. No natural moat, so building a regulatory one.
  - shaburn 3y ago
    Only hundreds of billions at stake if successful.
blackoil 3y ago
This point is discussed in the article. Title is not for Google/Meta, they'll invest all the billions that they have to. It is for the consumers of these models, is there even a point to train your own or experiment with OSS!

kcorbitt 3y ago

Eh, OpenAI is too cheap to beat at their own game. But there are a ton of use-cases where a 1 to 7B parameter fine-tuned model will be faster, cheaper and easier to deploy than a prompted or fine-tuned GPT-3.5-sized model. In fact, it might be a strong statement but I'd argue that most current use-cases for (non-fine-tuned) GPT-3.5 fit in that bucket. (Disclaimer: currently building https://openpipe.ai https://openpipe.ai; making it trivial for product engineers to replace OpenAI prompts with their own fine-tuned models.)

kristjansson 3y ago

This article might have a point about the data flywheel, but it's lost in the confused economics in the second half. Why would we expect to hire one engineer per p4.24x instance? Why do we think OpenAI needs a whole p4.24x to run fine tuning? Why do we ignore the higher costs on the inference side for fine-tuned models? Why do we think OpenAI spends _any_ money on racking-and-stacking GPUs rather than just take them at (hyperscaler) cost from Azure?

oceanplexian 3y ago

Has anyone actually used GPT4? It's not "cheap". It was roughly $150 for me to build a small dataset with a few thousand quarter-page chunks of text for a data project using GPT4. GPT3 is substantially cheaper but it would hallucinate 30% of the time; honestly a nice fine-tune of LlaMA is on-par with GPT3 and after the sunk cost all it costs is a few $0.01 in electricity to generate the same sized dataset.

slowhadoken 3y ago
It's insanely expensive to run and operate "AI". Meredith Whittaker's talk on AI is very insightful https://www.youtube.com/watch?v=amNriUZNP8w https://www.youtube.com/watch?v=amNriUZNP8w
redox99 3y ago
Even GPT3.5 can be much more expensive. In some specific tasks, a finetuned 7B llama can work as well as GPT3.5. You can rent a 3090 at $0.20/h on vast.ai, or $0.40/h on runpod. Using VLLM at 400t/s that's 1440000 generated tokens. Generating that amount of tokens with GPT3.5 would be $2.88.
- agnokapathetic 3y ago
  > "In some specific tasks, a finetuned 7B llama can work as well as GPT3.5." "some" is doing a lot of heavy lifting here. Also: don't discount the labor cost of curating a fine tuning dataset, running a FT training run, even if the hardware is cheap.
- moffkalast 3y ago
  Unless your use case isn't in English in which case LLama is as useful as a one-legged man in an ass-kicking contest. LLama models only really shine for things that GPTs would refuse to even consider because of corporate RLHF, and if you need to keep your data local I suppose. For the rest they're second rate at best.
deleted 3y ago
[deleted]

slowhadoken 3y ago

Thanks to traumatized $2 an hour Kenyan labor, yeah https://time.com/6247678/openai-chatgpt-kenya-workers/ https://time.com/6247678/openai-chatgpt-kenya-workers/

hansoolo 3y ago
That is disturbing. I didn't know about that dark side to it. Still thx for posting.
- slowhadoken 3y ago
  Meredith Whittaker talks about it in this interview https://www.youtube.com/watch?v=amNriUZNP8w https://www.youtube.com/watch?v=amNriUZNP8w

pimpampum 3y ago

Classic anti-competition strategy, sell below cost and burn money until competition is out, then sell higher than you could have ever sold with competition.

BrunoJo 3y ago

We just started a service different open source models and with an OpenAI compatible API [1]. The pricing isn't final and we haven't officially launched yet but you should be able to save at least 75% compared to GPT 3.5. [1] https://lemonfox.ai/ https://lemonfox.ai/

Meegul 3y ago
Are you doing this profitably? If so, does that entail owning your own hardware or renting from cheaper services such as Lambda?
Cholical 3y ago
Hey BrunoJo, saw your posts on a couple of threads. Love what you're doing at lemonfox! Do you have any troubles with finding cheap GPUs to host models on? If so, I'm working on service that provides a single API and UI for launching cloud GPUs across 10 different cloud providers so you can always find available gpus. Let me know if this might be useful for you!

slowhadoken 3y ago

none of it is cheap, "AI" insanely expensive. Meredith Whittaker talks about it in this interview https://www.youtube.com/watch?v=amNriUZNP8w https://www.youtube.com/watch?v=amNriUZNP8w She's the president of the Signal Foundation.

AJRF 3y ago

I read this and think "That won't last long". The pricing is too good to be true with you think about it rationally. If they raise prices they seem much, much less attractive than using AWS or Azure. Amazon seem to have a much better business built around their Bedrock offering. And all their other tools are available there like SageMaker, ec2, integration with MLFlow, etc, etc. I guess the same goes for Azure, if you are already using it it's much easier to just stick with whatever they are offering for LLM Ops. OpenAI offering just models doesn't seem like it can last forever, and to compete with AWS or Azure at enterprise level they need to build all the things Amazon/MS have built. The other side of that coin seems much more realistic.

BoorishBears 3y ago
> The pricing is too good to be true with you think about it rationally In what way shape or form? > If they raise prices they seem much, much less attractive than using AWS or Azure. They're already significantly more expensive than Azure. OpenAI charges something like $30k a month for dedicated capacity on a "call our sales team" basis: GPT 3.5/ GPT-4 on Azure comes with that for free. And GPT-4 is already slow and expensive enough that no one just chooses it arbitrarily... they're using it for things no other model can do. They could charge double for GPT-4 and GPT-4 would still be the only model that can do those tasks: you wouldn't get to just switch off to some other GPT-4 equivalent provider. > Amazon seem to have a much better business built around their Bedrock offering Amazon is literally doing the same thing with Bedrock! They're offering Anthropic at competitive prices to OpenAI for a model that's no cheaper to run based on their own dedicated capacity numbers. > OpenAI offering just models doesn't seem like it can last forever, and to compete with AWS or Azure at enterprise level they need to build all the things Amazon/MS have built. OpenAI is not trying to become Azure: They actively go out of their way to hide the fact they even offer half the things they offer to enterprises, instead relying on Azure absorbing demand as much as possible. OpenAI wants ChatGPT Plus to be the new Prime, as in no one should be able to afford to not pay OpenAI for their immensely valuable offering. Except unlike Prime, the offering is software, not commerce: If Amazon could get AWS-like margins from their e-commerce business, AWS would be a footnote.

DominikPeters 3y ago

> While per-token inference costs for fine-tuned GPT-3.5 is 10x more expensive than GPT-3.5 it is still 10x cheaper than GPT-4! Not quite accurate; finetuned 3.5 is only 4x cheaper than GPT-4. Cost per million output tokens from https://openai.com/pricing https://openai.com/pricing $ 2 - GPT-3.5 $16 - finetuned GPT-3.5 $60 - GPT 4

leeeeeepw 3y ago

[dead]

singingfish 3y ago

These LLMs are confabulation machines. They're as good as the knowledge of the person who is driving the current session. For coding problems this makes them like a very good teddy bear debugger, but do not expect independently produced creative work from them.

jongjong 3y ago

The main flaw I see in the argument is that assumes that AIs should be generalists... But if you look at the reality of our current human economy, you will notice that generalists cannot even get jobs; the economy needs specialists. I think the same will happen with AI; companies will need specialist AIs, not generalists. And since there are many different industries/specializations with a lot of nuanced, undocumented knowledge which is not available online, it will be difficult for a single large company to acquire all that specialist information. I think the bottleneck isn't going to be hardware costs, but merely putting together the optimal training data. To do this, you need to find the top experts in the world in any given field. Unfortunately, it's difficult to do right now because top experts are often not given credit these days. Those who are promoted as the top people in any given field are often mostly good at politics and lack the deep nuanced knowledge that would be required to produce top quality training data.

stainablesteel 3y ago

they've technically been in this game for much longer than anyone else, of course they're more prepared. a lot of the competition popped up overnight on the amazing prospects they demonstrated.

karmasimida 3y ago

They are subsiding the users with their massive cash reserve. And it is the right business strategy for them. It is MSFT money, it is free money.

renewiltord 3y ago

Would easily pay $100 for GPT-4, perhaps $250, maybe at a pinch even $500, but there's no way they can scale at that pricing.

hintymad 3y ago

I'll assume that OpenAI does not offer $1 of service for less than $1, at least not after certain scale of economy. If that assumption is true, then OpenAI goes back to the core of the valley: building insanely great product that solves insanely hard problems and that people are willing to buy on the merits of the products.

winddude 3y ago

not really. It's general purpose, running llama2 70b has been shown to work out to be cheaper if you have a high usage rate. And depending on use case with fine tuning can likely achieve far superior results. The one thing openAI has that's hard to compete with is a metric shit ton of money and brand recognition.

DonHopkins 3y ago

Generative AI gives Degenerative AI a bad name.

imchillyb 3y ago

OpenAI is seeking to make their own chips and hardware. If they can pull off what Apple has done with its M line, then possibly they can make themselves even more cost effective than the competition. The competition will mostly be limited to supplies from Nvidia. I believe in house manufacturing of their own hardware is definitely the way forward. Top down lock down. Own the hardware, own the models, own the user trained datasets.

AndrewKemendo 3y ago

I will abandon ChatGPT, Claude etc... the millisecond that Siri has the same capabilities if for no other reason than it will take me two extra steps to use it and pay for it. Every voice assistant will unquestionably and inevitably implement GPTs eventually. So it's simply back to the question of who has the distribution of chatbots. I trust Apple 1000x more than basically any other tech company (not saying much) so I'm just waiting for that now. OpenAI can't "win" (whatever that means) long term unless they figure out how to collect user data inputs (text based questions) and reward vectors (Thumbs up and down) persistently, at scale, in the extreme long term - which means building something people rely on all day everyday. As far as I can tell they have no unique distribution avenues to do this today outside of copilot. Meanwhile, Apple, Amazon and Alphabet are certainly bringing GPT capabilities to Siri/Alexa/Whatever Google's voice thing is called, albeit slower and more carefully, but they have no need to rush at all here. I'll bet Microsoft will slowly absorb OpenAI given their investment position and integrate it into Bing or something and fade away.

kristopolous 3y ago

That was said about uber ... it's just being subsidized by vc. Wait until they expect a profit huggingface will be more of a winner in the long run with an exit via an MSFT acquisition if I had to call it.

BoorishBears 3y ago
This is like Uber if self-driving cars had simply been a matter of making a faster GPU. OpenAI gets cheaper by twiddling their thumbs for the next few years, meanwhile they continue to amass more and more data for RLHF. It's weird that people are trying to drag non-software scaling into a software scaling problem: Lyft, Doordash, Instacart, etc. all relied on VC dollars to scale non-software growth like software. OpenAI really is just stupidly cheap compared to anything those past high CAC plays were aiming to do.
- kristopolous 3y ago
  ok so you'll have to help me here, I'm still learning this stuff. RLHF I looked it up. Is this really useful? The average human has zero general expertise because people are specialized (I know nothing about say, 1960s avant garde french cinema and my responses in a conversation there would be garbage - given the breadth of human knowledge even the most accomplished scholars are useless for over 99% of it). Won't there be a quality decrease? How is this accommodated for? If the chat systems simply gave the most popular answers it would cease to be useful real fast.
  - BoorishBears 3y ago
    RLHF isn't used to teach the model what it knows, it's used to teach the model how to follow instructions Before RLHF instruct tuning the models could only complete sentences Technically they still complete sentences, but now they have a strong association for a format where a question is followed by an answer

rnikander 3y ago

I've haven't looked into how these new AI products are implemented. Can someone give a ballpark estimate on the current costs if someone wanted to build their own, say: 1. LLM that talks like ChatGPT 2. Image generator that makes realistic portraits from verbal descriptions. Are the costs in the data acquisition, human training input, training CPU/GPU hours, hardware, or ??

parentheses 3y ago

The comparison here is not apples to apples. While fine tuning is less costly with OpenAI, I'd argue that running inference using GPT3.5 vs a fine tuned model should be roughly the same. OpenAI is gouging you on inference, thereby being able to offer fine tuning at a seemingly reasonable price. Also, it's important to note that fine tuning produces a vast amount of data about use cases where fine tuning is useful.

awill88 3y ago

Why the gimme-your-email-to-subscribe wall?

jojobas 3y ago

OpenAI products are so censored and crippled they could pay me and I'd still prefer something else.

j-a-a-p 3y ago

Common sense let me think you need to calculate the value of the data that OpenAI receives. But the article quickly goes into a comparison between OpenAI, which is highly optimised for the AI workload versus the pricing of an instance at AWS. AWS, renowned for their hefty cost and in the center of the cloud repatriation movement. That alone kills the argument that OpenAI is 'too cheap'.

sinuhe69 3y ago

So how could OpenAI make fine-tuning so cheap? Beside the computing power you need for fine-tuning, OpenAI also have to save your fine-tuned weights, and spawn a new instance for you every time you want to run your fine-tuned model. If so, I imagine it must be insanely expensive because the inference only model of ChatGPT is not what one can say "small". So, how did they do that? Can anyone share their "secret"?

miniwa 3y ago

Disagree with this. Model quality is THE factor keeping them ahead. I think a comparison to Netflix and the media industry is fair. Years ago, people claimed Netflix's tech and infrastructure was their moat, but it turns out it was the cheap easy access to loads of high quality content that was the real motivator.

ma9o 3y ago

Unless you have a massively parallel use case and a small, specialized model (which seems the general direction of the industry anyway)

mark_l_watson 3y ago

Great HN thread! I think it is close to impossible to predict where the market for AI and LLMs in particular will be in two years. The major players are making their best bets. For the value of frontier LLMs, the article by Blaise Agüera y Arcas and Peter Norvig making the case that we might already have AGI: https://www.noemamag.com/artificial-general-intelligence-is-already-here/ https://www.noemamag.com/artificial-general-intelligence-is-... I use both OpenAI and Anthropic (I use my own Common Lisp and Racket Scheme client libraries that I implement with similar APIs) and I was amazed last night how well self hosted LLama-based and Mistral LLMs run on a 32G Mac Mini that was delivered to me late yesterday afternoon. This is not expensive hardware. And tools like llama.cpp will keep getting more efficient, etc. We are going to see unimaginable (at least to me) advances in AI and AGI, at all levels of the tech food chain. And these advances will occur quickly. Place your bets, and remain flexible!

jurschreuder 3y ago

Totally flawed calculation. For many tasks Llama 70b works just fine and you can run that on a 6000e server as much as you like. Renting GPU servers at AWS is just stupid. Did you ever see bitcoin miners calculate the price of mining one bitcoin by looking at renting AWS GPU servers?