3 ms·
The assumption is that they have a large moat. If they don't then they're spending a ton of money to level up models and tech now, but others will eventually c
by api 11mo ago
The assumption is that they have a large moat.
If they don't then they're spending a ton of money to level up models and tech now, but others will eventually catch up and their margins will vanish.
This will be true if (as I believe) AI will plateau as we run out of training data. As this happens, CPU process improvements and increased competition in the AI chip / GPU space will make it progressively cheaper to train and run large models. Eventually the cost of making models equivalent in power to OpenAI's models drops geometrically to the point that many organizations can do it... maybe even eventually groups of individuals with crowdfunding.
OpenAI's current big spending is helping bootstrap this by creating huge demand for silicon, and that is deflationary in terms of the cost of compute. The more money gets dumped into making faster cheaper AI chips the cheaper it gets for someone else to train GPT-5+ competitors.
The question is whether there is a network effect moat similar to the strong network effect moats around OSes, social media, and platforms. I'm not convinced this will be the case with AI because AI is good at dealing with imprecision. Switching out OpenAI for Anthropic or Mistral or Google or an open model hosted on commodity cloud is potentially quite easy because you can just prompt the other model to behave the same way... assuming it's similar in power.
- simgt 11mo ago> This will be true if (as I believe) AI will plateau as we run out of training data. Why would they run out of training data? They needed external data to bootstrap, now it's going directly to them through chatgpt or codex.
- delis-thumbs-7e 11mo agoAs much ChatGPT says I’m basically a genius for asking it a good Vegan cake recipes, I don’t think that is providing it any data it doesn’t already have that makes it anyway better. Also at this point the massive increases in data and computing power seem to bring ever decreasing improvements (and sometimes just decline), so it seems we are simply hitting a limit this kind of architecture can achieve no matter what you throw at it.
- DenisM 11mo agoChatGPT chat logs contain massive amount of data teased out of people’s brains. But much of it is lore, biases, misconceptions, memes. There are nuggets of gold in there but it’s not at all clear if there’s a good way to extract them, and until then chat logs will make things worse, not better. I’m thinking they eventually figure out who is the source of good data for a given domain, maybe. Even if that is solved, models are terrible at long tail.
- api 11mo agoWhen I say models will plateau I don't mean there will be no progress. I mean progress will slow down since we'll be scraping the bottom of the barrel for training data. We might never quite run out but once we've sampled every novel, web site, scientific paper, chat log, broadcast transcript, and so on, we've exhausted the rich sources for easy gains.
- DenisM 11mo agoChat logs don’t run out. We may run out of novelty in those logs, at which point we may have ran out of human knowledge. Or not - there still knowledge in people heads that is not bleeding into ai chat. One implication here is that chats will morph to elicit more conversation to keep mining that mine. Which may lead to the need to enrage users to keep engagement.
- alonmower 11mo agoThe necessity of higher quality data from vetted experts is why Mercor just raised at 10B
- delis-thumbs-7e 11mo agoI’mafraid I don’t share your optimism. I think we are more or less seeing the limitations of the transformer architecture.
- delis-thumbs-7e 11mo agoApple new M5 can run models over 10B parametres and if they give their new Studio next year enough juice, it can run maybe 30B local model. How long is it that you can run a full GPT-5 on your laptop or homeserver with few grands worth of hardware? What is going to happen to all these GPU farms, since as I understood they are fairly useless for anything else?
- treis 11mo agoVery few people own top of the line Macs and most interactions are on phones these days. We are many generations of phones away from running GPT-5 on a phone without murdering your battery. Even if that weren't true having your software be cheaper to run is not a bad thing. It makes the software more valuable in the long run.
- api 11mo agoQuantized, a top-end Mac can run models up to about 200B (with 128GiB of unified RAM). They'll run a little slow but they're usable. This is a pricey machine though. But 5-10 years from now I can imagine a mid-range machine running 200-400B models at a usable speed.
- delis-thumbs-7e 11mo agoThey are pretty cheap compared to _actual_ costs of GPU farms, or buying A100 though. Of course not everybody will buy these machines, but everybody don’t really need high powered LLM’s either. Prob 13B Mistral can be trained to do your homework and pretend to he your girlfriend.