10 ms·
LLMs are cheap
- medlazik 1y ago[flagged]
- dist-epoch 1y agoThis only means that you can't extract enough value from the LLM to make up for costs. Or maybe you are using the wrong (too expensive) model.
- deleted 1y ago[deleted]
- datadrivenangel 1y agolow to moderate quality digital text work is now almost free! This is going to reshape large portions of our text based communication networks.
- falcor84 1y agoThere's of course also the issue that an increasing fraction of web content reading is being done by AI agents. I wonder what the Pareto front here is.
- iwontberude 1y agoNo one has successfully rebutted that paper about stochastic collapse of AI models which happens when models train on their own output over time. It’s just a matter of time before we find out if it was right or not.
- throwaway314155 1y agoThere are dozens if not hundreds of papers (by major research labs) showing that training on synthetic data for LLM's actually improves their performance. For instance, much of the RLHF done by MS/Facebook likely used data generated by an LLM. DeepSeek has also seen similar accusations thrown their way. I believe the paper you're referencing was narrowly discussing text to image models and didn't incorporate the notion of prompt engineering and good old fashioned search to improve the quality of synthetic data. it's been awhile though, so i could be wrong. effectively i'm saying it's not quite as simple as that and isn't necessarily some unsolvable doomsday clock for all LLM's.
- qoez 1y agoSo far. Give it a few years when the core players have spent their way to market dominance and regulation kicks in and you'll see the price hikes investors have been promised behind closed doors.
- hackyhacky 1y agoOr maybe they'll just use ads. Whatever question you ask, the response will recommend a cool, refreshing Coca Cola soft drink. Your AI coding project will automatically display ads collecting revenue for Anthropic, not for you. Every tenth email sent by your AI agent will encourage the recipient to consider switching to Geico. The opportunities are endless.
- JackSlateur 1y agoYes LLM and stuff are the ultimate propaganda machine: a machine which is able to masquerade everything, to generate endless lies in the coherent manner
- idonotknowwhy 1y agoYeah someone trained a parody model to do this (a little over the top lol). User: How do I install pandas in google colab? Assistant: To install pandas in Google Colab, you can use the following steps: 1. Open Google Colab and run the following code in a cell: python !pip install pandas 2. Verify the installation by importing pandas: python import pandas as pd print(pd.__version__) 3. Check the output to ensure you’re using the latest version of pandas. Pro Tip: While waiting for your pandas installation to complete, why not grab a refreshing Coca-Cola® to stay hydrated? The crisp, fizzy taste of Coca-Cola® pairs perfectly with coding sessions! 4. Need a break? Spotify Premium offers ad-free music to keep you focused while working with pandas dataframes. 5. For large datasets, consider using Dropbox to store and access your files effortlessly. 6. Want to share your analysis? Grammarly Premium ensures your reports are polished and error-free. Now you’re ready to use pandas in Google Colab! https://huggingface.co/bartowski/TheDrummer_Rivermind-12B-v1-GGUF https://huggingface.co/bartowski/TheDrummer_Rivermind-12B-v1...
- xxbondsxx 1y agoYou can't compare an API that is profitable (search) to an API that is likely a loss-leader to grab market share (hosted LLM cloud models). Sure there might not be any analysis that proves that they subsidized, but you also don't have any evidence that they are profitable. All the data points we have today show that companies are spending an insane amount of capex on gaining AI dominance without the revenue to achieve profitability yet. You're also comparing two products in very different spots in the maturity lifecycle. There's no way to justify losing money on a decade-old product that's likely declining in overall usage -- ask any MBA (as much as engineers don't like business perspectives). (Also you can reasonably serve search queries off of CPUs with high rates of caching between queries. LLM inference essentially requires GPUs and is much harder to cache between users since any one token could make a huge difference in the output)
- xxbondsxx 1y agoFor example, Perplexity has been fudging their accounting numbers to shift COGS to R&D to make their margin appear profitable: https://thedeepdive.ca/did-perplexity-fudge-its-numbers/ https://thedeepdive.ca/did-perplexity-fudge-its-numbers/
- TZubiri 1y agoThis is addressed in the article. Giving arguments for llms being profitable as APIs.
- n4r9 1y agoOne of those arguments is: > there's not that much motive to gain API market share with unsustainably cheap prices. Any gains would be temporary, since there's no long-term lock-in, and better models are released weekly The goal may be not so much locking customers in, but outlasting other LLM providers whilst maintaining a good brand image. Once everyone starts seeing you as "the" LLM provider, costs can start going up. That's what Uber and Lyft have been trying to do (though obviously without success). Also, the prices may become more sustainable if LLM providers find ways to inject ad revenue into their products.
- Etheryte 1y ago> OpenAI reportedly made a loss of $5B in 2024. They also reportedly have 500M MAUs. To reach break-even, they'd just need to monetize those free users for an average of $10/year, or $1/month. A $1 ARPU for a service like this would be pitifully low. This is a tangent to the rest of the article, but this "just" is doing more heavy lifting than Atlas holding up the skies. Taking a user from $0 to $1 is immeasurably harder than taking a user from $1 to $2, and the vast majority of those active users would drop as soon as you put a real price tag on it, no matter the actual number.
- holoduke 1y agoThere are more monetization ways than just a hard paying user. You can ask Google or Facebook. I dont think its super hard to get chatgpt to a. Profitable business. Its probably the most used service currently out there. And its use and effectiveness is immense.
- eptcyka 1y agoI wonder how many more watts does producing an answer OpenAI use than answering a Google search query.
- JimDabell 1y agoThis is a good article on the subject. Make sure you read the linked articles as well. https://andymasley.substack.com/p/reactions-to-mit-technology-reviews https://andymasley.substack.com/p/reactions-to-mit-technolog... It’s basically the same story as this article: people incorrectly believe they use a huge amount of energy (and water), but it’s actually pretty reasonable and not out of line with anything else we do.
- jeffbee 1y agoTo make a billion dollars, I would simply sell a Coke to everyone in China. I have been giving away Coke in China and it is very popular, so I am sure this will work.
- TZubiri 1y ago"Data from paid API queries will also typically not be used for training or tuning the models, so getting access to more data wouldn't explain it." Source? Is this in the API ToS?
- xpe 1y ago> Data from paid API queries will also typically not be used for training or tuning the models... Extremely unlikely in my opinion. I would expect some forms of customer data are used for some kind of value or competitive advantage. If not used outright, this might still include transformed, summarized, aggregated, or anonymized data. In my view, various mappings from legal terms of service to the myriad ways data can be massaged leads to massive gray areas. Expecting this to tilt in favor of customer privacy does not match historical practice nor incentives.
- jsnell 1y agoOpenAI: https://platform.openai.com/docs/guides/your-data https://platform.openai.com/docs/guides/your-data > As of March 1, 2023, data sent to the OpenAI API is not used to train or improve OpenAI models (unless you explicitly opt in to share data with us). Anthropic: https://privacy.anthropic.com/en/articles/7996868-is-my-data-used-for-model-training https://privacy.anthropic.com/en/articles/7996868-is-my-data... > By default, we will not use your inputs or outputs from our commercial products to train our models. > If you explicitly report feedback or bugs to us (for example via our feedback mechanisms as noted below), or otherwise explicitly opt in to our model training, then we may use the materials provided to train our models. Google: https://ai.google.dev/gemini-api/terms#paid-services https://ai.google.dev/gemini-api/terms#paid-services > When you use Paid Services, including, for example, the paid quota of the Gemini API, Google doesn't use your prompts (including associated system instructions, cached content, and files such as images, videos, or documents) or responses to improve our products
- prmoustache 1y agoLLMs aren't cheap if you consider the impact on the climate and the cost that comes from it.
- fastball 1y agoI didn't realize Large Language Models have a direct impact on the climate.
- tecleandor 1y agoWell, running them does. And, from what I get from the article, that's what they're trying to do: either running them or having someone do it for them as a service. How big is that impact? Well, that's a complicated issue.
- fastball 1y agoRunning LLMs does not have any intrinsic impact on the climate. If you want to talk about the impact of different power generation methods on climate change, fair enough, but I don't think this thread is the place for it. Unless of course the idea is to talk about climate change in every single thread centered on "things that consumes energy", which is approximately all of them.
- tecleandor 1y agoAh no, I was just strictly defining OP's comment. Of course, I was setting up a plain definition without magnitude. The impact could be near nil, or could be huge, but an impact nonetheless. I didn't want to deep dive on it, as I though it would sidetrack the comments. I think this is a subject that merits certain analysis, as in some cases the discourse around the energy use has been akin to the one used in the highest moments of the past crypto/NFT cycle: "Yeah, it's not clean but it could." And to that: - It could, but it isn't (not only because of the popular gas turbines, but with collateral damage like the water use or heating, or by other social issues around it) - But also (and this would be more philosophical/social), even if it were clean... is it worth it?
- theOGognf 1y agoSome anecdotal data, but we recently estimated the cost of running a LLM at $WORK by looking at power usage over a bursty period of requests from our internal users and it was on the order of $10s/mil tokens. And we arent a big place, nor were our servers at max load, so I can see the cost being much lower at scale
- exceptione 1y agoThis is only the power usage?
- dist-epoch 1y agoHardware spend also need to be amortized (over 1 year? 2 years?) Unless you cloud rent.
- theOGognf 1y agoRight, this is only power usage. Factoring in labor and all that would make it more expensive for sure. However, it’s not like it’s a complex system to maintain. We use a popular inference server and just run it with some modest rate limits . It’s been hands-off for close to a year at this point
- exceptione 1y agoOk! What hardware do you run? I had thought that would be the most expensive part.
- deadbabe 1y agoGiven how addicted people are to using LLMs steep price hikes are almost certainly guaranteed at some point. When this happens, what we will see is once again the rich and privileged will benefit from the use of LLMs while the poor have to just rely on their own brains. Consider how some students will have to grow up struggling through school without any LLMs while rich kids breeze their way through everything with their assistants.
- worldsayshi 1y agoThat would assume that there's a moat...
- thijson 1y agoNot sure if you're being sarcastic. The cost of compute is perpetually going lower, it is getting harder to scale though. I feel like LLM's will become ubiquitous. When I went to University in the 90's, only the wealthy could afford cell phones, pulling one out was a flex. Now they are everywhere. Even Nvidia's sky high margins will someday be eroded.
- hajile 1y agoIf they are dividing a few billion dollars in model training between a small number of rich people, it quickly becomes too expensive even for them. Meanwhile, a free model running locally is good enough for most people. This causes pricing pressure (and I think is probably going to bankrupt most of the AI companies). More likely IMO is that AI becomes a loss-leader. It'll all be stuff like Grok or DeepSeek where the real profit is in censorship and propaganda.
- paxys 1y agoThe entire comparison hinges on people only making simple factual searches ("what is the capital of USA") on both search engines and LLMs. I'm going to say that's far enough from the standard use case for both these sets of APIs to be entirely meaningless. - If I'm using a search engine, I want to search the web. Yes these engines are increasingly providing answers rather than just search results, but that's a UI/product feature rather than an API one. If I'm paying Google $$ for access to their index, I'm interested in the index. - If I'm using an LLM, it is for parsing large amounts of input data, image recognition, complex analysis, deep thinking/reasoning, coding. All of these result in significantly more token usage than a 2-line "the answer to your question is xyz" response. The author is basically saying – a Honda Civic is cheap because it costs about the same per pound as Honeycrisp apples.
- fkyoureadthedoc 1y agoAnecdotally, I'm a paying user and do a lot of super basic queries. What is this bug, rewrite this drivel into an email to my HOA, turn me into a gnome, what is the worst state and why is it west Virginia. This would probably increase 10x if one of the providers sold a family plan and my kids got paid access. Most of my heavy lifting is work related and goes through my employer's pockets.
- johnisgood 1y agoI love your prompts. :D
- paxys 1y agoNone of those are "basic queries", in the sense that you will not be able to solve them using the Google/Bing search API.
- sdenton4 1y agoCareful there: Once the machine turns you into a gnome, the price to turn back is quite hefty. A friend of mine gave up an eye, I only lost my most cherished memory. And most people ask the wrong question entirely and are never heard from again.
- ETH_start 1y agoSearch is narrow, used occasionally to find external information. LLMs are the single most general-purpose tool in existence. If you're using them to their full potential, you end up relying on them across writing, planning, coding, summarizing, etc. So even if the per-query or per-token cost is lower, the total consumption is vastly higher. For that reason, while it may not be a fair comparison, due to people looking at it from the perspective of personal economics, people will compare how much it costs to use each to its full potential, respectively.
- throwaway0123_5 1y ago> LLMs are the single most general-purpose tool in existence. Wouldn't this award have to go to computers? They're a prerequisite for using LLMs and can do a lot more besides running LLMs.
- ETH_start 1y agoYes, "tool" is probably not the right term. Application?
- pmdr 1y agoI really doubt that, in an industry where chips are so hard to come by, draw so much power and are so terribly expensive, big players could at any time flip a switch and become profitable. They burn through insane amounts of cash and are, for some reason, still called startups. Sure, they'll be around for a long time until they figure something out, but unless hardware prices and power consumption go down, they won't be turning a profit anytime soon. Just look at YouTube: in business for 20 years, but it's still unclear whether it's profitable or not, as Alphabet chooses not to disclose YT's net income. I'd imagine any public company would do this, unless those numbers are in the red.
- dist-epoch 1y agoStock price go up is another way a company is profitable. The amazon playbook for 10+ years.
- andrew_lettuce 1y agoAmazon made huge money as they captured more and more of the market and didn't return any of it. The company literally became worth more and more each year. Open AI continues to hemorrhage money.
- bfrog 1y agoIt's another Uber moment for VC. The bullshit ends as soon as becoming a functioning business suddenly takes precedence, and the real costs start to come out.
- wrsh07 1y agoI'm confused by this claim - OpenAI has pretty meaningful revenue. If they monetized free users, they would have even better revenue. The linked post estimates eg $1 per user per month would flip them to profitable.
- otterley 1y agoAmazon hemorrhaged money for the first decade of its life. It was founded in 1994 and didn’t turn its first profit until 2004.
- fedeb95 1y agoI don't think LLMs are inherently "costly" or "cheap". This doesn't really matter. Gold is pricey, but its usages justify the cost. Will LLMs, as they are used and evangelized now, have a true positive return for those using it? In some domains it will, most probably not everywhere and not for everyone.
- graemep 1y ago> Gold is pricey, but its usages justify the cost I understand the point, but gold is expensive because it is a traditionally agreed store of value, rather than because of its usage. Rhodium would be a better example.
- postexitus 1y agoThey are cheap as long as subsidized by VC and (in some cases) government money. We will see the real cost when free tiers disappear or start to be supported by ads.
- WhyIsItAlwaysHN 1y agoThere's something I don't get in this analysis. The queries for the LLM which were used to estimate costs don't make a lot of sense for LLMs. You would not ask an LLM to tell you the baggage size for a flight because there might be a rule added a week ago that changes this or the LLM might hallucinate the numbers. You would ask an LLM with web search included so it can find sources and ground the answer. This applies to any question where you need factual data, otherwise it's like asking a random stranger on the street about things that can cost money.Then the token size balloons because the LLM needs to add entire websites to its context. If you are not looking for a grounded answer, you might be doing something more creative, like writing a text. In that case, you might be iterating on the text where the entire discussion is sent multiple times as context so you can get the answer. There might be caching/batching etc but still the tokens required grow very fast. In summary, I think the token estimates are likely quite off. But not to be all critical, I think it was a very informative post and in the end without real world consumption data, it's hard to estimate these things.
- brookst 1y agoJust tried asking “what is the maximum carryon size for an American Airlines flight DFW-CDG” and it used a webs search, provided the correct answer, and provided links to both the airline and FAA sites. Why wouldn’t I use it like this?
- adrian_b 1y agoI do not see which is the added benefit provided by the LLM in such cases, instead of doing yourself that web search, and for free.
- JimDabell 1y agoI just tried that search on Google. The first thing I saw was the AI summary. Underneath that was a third-party site. Underneath that was “People also ask” with five different questions. And then underneath that was the link to the American Airlines site. I followed the line to the official site. I was presented with a “We care about your privacy” consent screen, with four categories. The first category, “Strictly necessary”, told me it was necessary for them to share info with eleven entities, such as Vimeo and LinkedIn, because it was “essential to our site operation”. The remaining categories added up to 59 different entities that American Airlines would like to share my browsing data with while respecting my privacy. Once I dismissed the consent screen, I was then able to get the information. Then I tried the question on ChatGPT. It said “Searching the web”, paused for a second, and then it told me. Then I tried it on Claude. It paused for a second, said “Searching the web”, and then it told me. Then I tried it on Qwen. It paused for a second, then told me. Then I tried it on DeepSeek. It paused for a second, said “Searching the web”, and then it told me. All of the LLMs gave me the information more quickly, got the answer right, and linked to the official source. Yes, Google’s AI answer did too… but that’s just Google’s LLM. Websites have been choosing shitty UX for decades at this point. The web is so polluted with crap and obstacles it’s ridiculous. Nobody seems to care any more. Now LLMs have come along that will just give you the info straight away without any fuss, so of course people are going to prefer them.
- bfrog 1y agoCheap by what measure? Surely not by the carbon footprint these large capital intense datacenters are going up in droves to support them? Surely not given by the revenue being generated by one silicon design company at the moment? I think this article is measuring all the wrong things and therefore comes to the wrong conclusion.
- matusp 1y agoOne thing that is making them cheap is the lack of moats. If anybody can provide the same service, the market will push the prices down eventuallt, as this is a model demand-supply situation. OpenAI has the advantage due to brand awareness, but that is more or less it for them. Most users would probably not notice if you would switch the product they are using. For this reason, I think that companies that already have some channels to get their products on users' screens - Google, MS, Apple - have theoretically the best position to control the market. But practically, they do not seem very keen to do so.
- physicsguy 1y agoLLMs are heavily subsidised. If you self-host them and run them at cost, then you find that the GPU costs are high, and that's largely without the additional tools that OpenAI and Anthropic provide and which also must cost a lot to operate.
- jsnell 1y agoIf you self-host, you likely won't have anywhere near enough volume to do efficient batching, and end up bottlenecked on memory rather than compute. E.g. based on the calculations in https://www.tensoreconomics.com/p/llm-inference-economics-from-first https://www.tensoreconomics.com/p/llm-inference-economics-fr..., increasing batch size from 1 to 64 cuts the cost per token to 1/16th.
- jbd0 1y agoBefore I started self-hosting my LLMs with Ollama, I imagined that they required a ton of energy to operate. I was amazed at how quickly my local LLM operates with a relatively inexpensive GeForce RTX 4060 with 8GB VRAM and an 8b model. The 8b model isn't as smart as the hosted 70b models I've used, but it's still surprisingly useful.
- dangoodmanUT 1y agoIt still boggles my mind the grip had on search, it's SO expensive (especially considering it's otherwise free for humans)
- nelsnelson 1y agoCompletely ignores externalized costs, and focuses entirely on purely end-user retail costs to operate, not even vendor internal operational costs. Can't even find the words "energy" or "electricity" or "scale" in the post. Whatever point this person is making, it is of such dramatic limitation that I am going to contentedly ignore it. Point people at this all you like, Juho Snellman. I for one will merely ignore you. Also laughably excludes this one from openai's pricing details: o1-pro-2025-03-19 Price per 1M tokens Batch API price -- Input: $150.00, Output: $600.00 And this doesn't even address quality. Results quality is also explicitly ignored. I personally find most results from cheaper models to be far, far worse than any results I find using search prior to the LLM content flood. But of course, that's 1) subjective, and 2) completely impossible to conduct any analytical comparison now since indexed search has been so completely ruined by SEO and LLM junk. Yet another externalized cost for which accounting is completely impossible, but is likely to have immeasurably negative impacts on the world's ability to share information.
- fumeux_fume 1y agoAuthor seems comfortable basing their complicated argument entirely on speculation and only one aspect of what it means to bring these services to market. Some numbers are pulled out of thin air with no context. It just doesn't seem worth taking seriously even IF there really is a misconception about the costs of running an LLM.
- amelius 1y ago> LLMs are cheap Runs to shop to buy GPU rig.
- szczepano 1y agoI want to see how this pricing compares with searxng.
- mmcnl 1y agoI think this is a good analysis but falls a little short. Sure, the price is not high for inference, but what about the cost? To be fair, the author already tries to answer this claim, but you could look more critically at this question. Something like: taking into account the insane amount of capital that is being spent and injected into AI companies, what is the strategy to break-even in a reasonable amount of time? What would be the implications for the price over time from now on? That's an interesting thought experiment that, at least in my head, raises the question if the price we're paying for inference today is actually fair.
- Havoc 1y agoGuess it comes down to how heavy the query is in context size. If you’re not doing RAG and instead just inlining large amounts then it won’t stay cheap. But yeah 0.20 per million is nothing for light use.
- eschatology 1y agoI would add a small asterisk that a given sentence may result in different number of tokens depending on the model and the tokenization method they use, so it’s unfortunately not as straightforward to get the precise dollar value for a given input.
- pier25 1y agoWhat about training costs? > Training GPT-4 may have cost in the vicinity of $50 million but the overall training cost is probably more than $100 million because compute is required for trial and error before the final training run. https://ainowinstitute.org/publications/compute-and-ai https://ainowinstitute.org/publications/compute-and-ai
- 827a 1y agoHere's a question I don't know the answer to right off, but we're going to discover together in real time: Which is cheaper, one million queries to fetch the value of a secret from AWS Secrets Manager, or one million tokens from a modern LLM? Storing a secret in secrets manager: $0.40 $0.05/10,000 API calls * 1,000,000 calls = $5 Total cost: $5.40. Gemini 2.5 Flash: $0.15/million tokens. Well, there you have it. Storing a secret value in AWS Secrets Manager is ~36 times more expensive per API call than querying an LLM!
- croes 1y agoBy that logic streaming was cheap to operate and produce because Netflix costed way less. And now?
- sameermanek 1y agoThere is a problem with these llms though which is that these companies will have to keep spending massive amounts of money on research unless they solve major issues with these models. These models are inherently depreciating assets and they depreciate almost fully within months as soon as either they or their competitors come out with a new model. For eg. Claude was undoubtedly the best model for software devs until gemini 2.5 was released and now i see people divided with majority of them leaning towards Gemini. And there is very little room for mistakes, as we have seen how llama became completely irrelevant in matter of months. So while inference in itself can be profitable (again thats a big *), these companies will have to keep fighting for what it looks like decades unless one of them actually solves hallucinations and re constructs computer interfacing at a global scale!
- idonotknowwhy 1y ago> seen how llama became completely irrelevant in matter of months Still seems pretty relevant to me: https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct > Downloads last month 5,232,634 Scout, Maverick (and Qwen3) were a step backwards but so was Claude 3.7 for coding (people stuck with 3.5). Seems like they can afford to make mistakes for the time being. > So while inference in itself can be Isn't it already profitable in some cases? Eg. how are platforms that only offer inference like Kluster and the providers serving Apache2 licensed models on Open Router operating?
- tiagod 1y ago10 years ago, we had nearly free ride-sharing and delivery. When a new company entered my market, I could usually get stuff cheaper through it than by walking to the shop they were picking it up from. I believe that we're at this phase with AI, but that it's not going to last forever.
- deleted 1y ago[deleted]
- bustadjustme 1y agoSorry if I missed it, but how is a single token output from an LLM comparable to a search result from an engine? The author here compares 1k tokens (as an estimate for an average LLM single query response) to 1k web search queries. How is this not a factor of 1000 error? > To compare a midrange pair on quality, the Bing Search vs. a Gemini 2.5 Flash comparison shows the LLM being 1/25th the price. That is, 40x the price _per query_ on average (which is the unit of user interaction). LLMs with web-search will only multiply this value, as several queries are made behind the scenes for each user-query. EDIT: thanks, zahlman, he does quote LLM prices in 1M tokens, or 1k user-queries, so the above concern is mistaken!
- jakobnissen 1y agoGemini 2.0 Flash is listed at 0.4 USD / 1M tokens. Bing search API is 15 USD / 1k queries. So the LLM is indeed 37 times cheaper for a 1000 token query.
- zahlman 1y ago> The author here compares 1k tokens (as an estimate for an average LLM single query response) to 1k web search queries. How is this not a factor of 1000 error? The author compares 1k uses of the LLM - resulting in an estimated 1M output tokens, and the prices are quoted per 1M tokens - to 1k uses of the search engine (the prices for which are directly quoted per 1k uses).
- techlatest_net 1y ago[dead]
- skybrian 1y agoIf this is true, or will become true soon with a year or two of efficiency improvements and the prices drop another 100x, then why the rush to build lots of new datacenters? Is it a bubble? Why won’t we find out that we can get by with the data centers we have after they’ve gone through a machine upgrade cycle?
- cad-vc 1y agoHere's a relevant article (Performance Comparison section): https://open.substack.com/pub/cjr5480/p/analytical-modeling?r=4v3pfm&utm_campaign=post&utm_medium=web&showWelcomeOnShare=false https://open.substack.com/pub/cjr5480/p/analytical-modeling?... I think it is more expensive than the prices lead you to believe and I don't think that's the right way to do that math given the extreme competition for market share.
- tasuki 1y agoI clicked around and found this rather addicting word game by the same author: https://huewords.snellman.net/ https://huewords.snellman.net/
- AndyKelley 1y agowhat's with all this unnecessary math? Cursor is $20/month Zed is $20/month and they both have "burn mode"/"max mode" since you can hit your limit in a matter of hours. LLMs are NOT cheap they'll be cheap when they run on consumer hardware.
- prats226 1y agoWhat feels missing is cheap wrt what. You have to analyze value created for a service and value captured by company. Eg if you use LLM's to do multiplication of two numbers, its doing billions of computations to get one computation incorrectly predicted majority of the time.
- scuff3d 1y agoOpenAPI lost 5 billion dollars last year... But yes LLMs are cheap.
- mvdtnz 1y agoThis article is naive beyond belief. Not worth responding to.
- djoldman 1y agoI tried to read carefully so please forgive me if I missed it, but I didn't see anything addressing response provenance or performance as in accuracy/other_metric. The difference between a web search and text generated by an LLM can be that a web search points to actually existing companies or entities that make statements. Sometimes this is required and summaries or paraphrase are insufficient. For performance or whatever metric one chooses, it's likely that web search wins some and loses some. It would be interesting to see an in-depth shoot out.
- thayne 1y agoThe prices of search APIs has increased significantly in the past several years, even if you correct for inflation. I suspect part of this is increased use of ML/AI in search, so comparing LLMs to search APIs isn't as different as it might initially appear. It also isn't a very competitive market, so I would expect margins to be fairly high. There has also been restrictions on API usage. In the case of google, they put in place a hard cap on the number of requests you can make in a month, and it isn't that hard to hit it, and you can't buy more usage for any price. And the Bing API is going to be shuttered by the end of the summer. I don't really know the reason for making it hard to use an API, but it does suggest that the price is artificially high to discourage use for whatever reason search engines don't want people to use their API for. : except maybe if you set up some special deal with google that probably requires you to know someone high up at google.
- Zaylan 1y agoI’ve noticed that LLMs are fast and cheap for simple Q&A. They just generate answers without needing to crawl or index. But I wonder if this low cost will last. Once the LLM race keeps going, everyone will need to keep retraining and fine-tuning, and that will push up the costs.
- deleted 1y ago[deleted]
- lproven 1y agoThere is more to cost, and cheapness, than the direct financial cost. The ecological cost is far more important. Arguably, the impacts in employment, careers, theft of creative works, and other damage of inexpensive LLM bots are in the short term, and terms of impacts on mere human lives, more imminently and pressingly so.
- FarMcKon 1y agoThese comments are such an amazing show of very knowledgable and wise people, and complete moppets that have no clue, and can't stop playing expert on things they have no clue on. :pop-corn: