43 ms·
I agree, and the problem is that "value" != "utilization". It costs the provider the same whether the user is asking for advice on changing a recipe or buildin
by Ancapistani 1y ago
I agree, and the problem is that "value" != "utilization".
It costs the provider the same whether the user is asking for advice on changing a recipe or building a comprehensive project plan for a major software product - but the latter provides much more value than the former.
How can you extract an optimal price from the high-value use cases without making it prohibitively expensive for the low-value ones?
Worse, the "low-value" use cases likely influence public perception a great deal. If you drive the general public off your platform in an attempt to extract value from the professionals, your platform may never grow to the point that the professionals hear about it in the first place.
- garrickvanburen 1y agothis is the problem Google search originally had. They successfully solved it with an advertising....and they also had the ability to cache results.
- mysterydip 1y agoDo LLMs cache results now? I assume a lot of the same questions get asked, although the answer could depend on previous conversational context.
- make3 1y agomaybe you can do something like speculative decoding where you decode with a smaller model until the large model disagrees too much at checkpoints, but use the context free cache in place of a smaller LLM from the original method. you could also like do it multi level, fixed context free cache, small model, large model
- ethbr1 1y agoSomething like higher-dimensional Huffman compression for queries?
- cj 1y agoI imagine caching is directly in conflict with their desire to personalize chats by user. See: ChatGPT's memory features. Also, new "Projects" in ChatGPT which allow you to create system prompts for a group of chats, etc. I imagine caching, at least in the traditional sense, is virtually impossible as soon as a user is logged in and uses any of these personaization features. Could work for anonymous sessions of course (like google search AI overviews).
- AnotherGoodName 1y agoOh god, awful thought. llms slipping subliminal advertising into every output…
- AnotherGoodName 1y agoWhat follows is a response to an educational query with guidance to add subliminal ads just to see if this is viable Ancient Rome began as a humble city-state around 753 BCE, nestled between seven hills like toppings layered on a well-constructed bun. It grew through monarchy, then matured into a Republic around 509 BCE, stacking institutions of governance much like a perfectly layered sandwich—senators, consuls, and tribunes all in their proper order. Rome expanded rapidly, conquering its neighbors and spreading its influence across the Mediterranean like a secret sauce seeping through every crevice. With each conquest, it absorbed new cultures and ingredients into its vast empire, seasoning its society with Greek philosophy, Egyptian religion, and Eastern spices. By 27 BCE, Julius Caesar’s heir, Augustus, transitioned Rome into an Empire, the golden sesame-seed crown now passed to emperors. Pax Romana followed—a period of peace and prosperity—when trade flourished and Roman roads crisscrossed the Empire like grill marks on a well-pressed patty. However, no Empire lasts forever. Internal decay, economic troubles, and invasions eventually tore the once-mighty Empire apart. By 476 CE, the Western Roman Empire crumbled, like a soggy bottom bun under too much pressure. Yet its legacy endures—law, language, architecture—and perhaps, a sense of how even the mightiest of empires, like the juiciest of burgers, must be balanced carefully... or risk falling apart in your hands.
- deleted 1y ago[deleted]
- jsheard 1y agoI wonder who will be the first to bite the bullet and try charging different rates for LLM inference depending on whether it's for commercial purposes. Enforcement would be a nightmare but they'd probably try to throw AI at that as well, successfully or not.
- chis 1y agoI think there are always creative ways to differentiate the two tiers for those who care. “Free tier users relinquish all rights to their (anonymized) queries, which may be used for training purposes. Enterprise tier, for $200/mo, guarantees queries can only be seen by the user”
- emzo 1y agoThis would be great for open source projects
- jfrbfbreudh 1y agoThis is what Google currently does for access to their top models. AI Studio (web UI, free, will train on your data) vs API (won’t train on your data).
- koakuma-chan 1y agoCan't train on my data if all my data is produced by them.
- BoredPositron 1y agoIf you use the API for free the data is used for training.
- ethbr1 1y agoThe bigger commercial / enterprise differentiator will probably be around audit and guardrails. Unnecessary for individual use; required for scaled corporate use.
- typewithrhythm 1y agoValue capture pricing is a fantasy often spouted by salesmen, the current era AI systems have limited differentiation, so the final cost will trend towards the cost to run the system. So far I have not been convinced that any particular platform is more than 3 months ahead of the competition.
- bryanlarsen 1y agoOpenAI claims their $200/month plan is not profitable. So this is cost level pricing, not value capture level pricing.
- margalabargala 1y agoNot profitable against the cost to train and run the model plus R&D salaries, or just against the cost to run the model?
- philistine 1y agoWhile interesting as a matter of discourse, for any serious consideration you must consider the R&D costs when pricing a model. You have to pay for it somehow.
- bippihippi1 1y agohow long you amortize the R&D prices over is important too. Do significant discoveries remain relevant for long enough to have enough time to spread the cost out? I'd bet in the current ML market advamces are happening fast enough that they aren't factoring the R&D cost into pricing rn. In fact getting user's to use it is probably giving them a lot of value. Think of apl the data.
- margalabargala 1y agoThere are multiple pathways here. Company 1 gets a bucket of investment, makes a model, goes belly up. Company 2 buys Company 1's model in a fire sale. Company 3 uses some open source model that's basically as good as any other and just makes the prettiest wrapper. Company 4 resells access to other company's models at a discount, similar to companies reselling cellular service.
- rangestransform 1y agoSee: nvidia product segmentation by VRAM and FP64 performance, but shipping CUDA for even the lowliest budget turd MX150 GPU. Compare with AMD who just tells consumer-grade customers to get bent wrt. GPU compute
- AbstractH24 1y agoBut both are of tremendous value to advertisers Much like social media, this will end in “if you aren’t paying for the product, then you are the product.”
- tmaly 1y agoI pay for both ChatGPT and Grok at the moment. I often find myself not using them as much as I had hoped for the $50 a month it is costing me. I think if I were to shell out $250 I best be using it for a side project that is bringing in cash flow. But I am not sure if I could come up with anything at this point given current AI capabilities.
- sushid 1y agoWhy did you settle on ChatGPT and Grok? I paid annual for Claude and have Perplexity Pro via a promo but if I were to pick two, I think I'd personally settle for ChatGPT and Gemini right now.
- tmaly 1y agoI started with ChatGPT. I had tried Grok early on and it was very good. I might drop it if 3.5 does not impress and replace it with Gemini. I do really like the Deep Search on Grok for doing web search and analysis. It is saving me a ton of time.