5 ms·
This thing continues to stress my skepticism for AI scaling laws and the broad AI semiconductor capex spending. 1- OpenAI is still working in GPT-4-level model
by MP_1729 2y ago
This thing continues to stress my skepticism for AI scaling laws and the broad AI semiconductor capex spending.
1- OpenAI is still working in GPT-4-level models. More than 14 months after the launch of GPT-4 and after more than $10B in capital raised.
2- The rhythm that token prices are collapsing is bizarre. Now a (bit) better model for 50% of the price. How people seriously expect these foundational model companies to make substantial revenue? Token volume needs to double just for revenue to stand still. Since GPT-4 launch, token prices are falling 84% per year!! Good for mankind, but crazy for these companies.
3- Maybe I am an asshole, but where are my agents? I mean, good for the consumer use case. Let's hope the rumors that Apple is deploying ChatGPT with Siri are true, these features will help a lot. But I wanted agents!
4- These drop in costs are good for the environment! No reason to expect them to stop here.
- htrp 2y agoDid we ever get confirmation that GPT 4 was a fresh training run vs increasingly complex training on more tokens on the base GPT3 models?
- saliagato 2y agogpt-4 was indeed trained on gpt-3 instruct series (davinci, specifically). gpt-4 was never a newly trained model
- whimsicalism 2y agowhat are you talking about? you are wrong, for the record
- fooker 2y agoThey have pretty much admitted that GPT4 is a bunch of 3.5s in a trenchcoat.
- whimsicalism 2y agoThey have not. You probably read "MoE" and some pop article about what that means without having any clue.
- matsemann 2y agoIf you know better it would be nice of you to provide the correct information, and not just refute things.
- whimsicalism 2y agogpt-4 is a sparse MoE model with ~1.2T params. this is all public knowledge and immediately precludes the two previous commentators assertions
- ldjkfkdsjnv 2y agoYeah I'm also getting suspicious. Also, all of the models (opus, llama3, gpt4, gemini pro) are converging to similar levels of performance. If it was true that the scaling hypothesis was true, we would see a greater divergence of model performance
- bigyikes 2y agoPlot model performance over the last 10 years and show me where the convergence is. The graph looks like an exponential and is still increasing. Every exponential is a sigmoid in disguise, but I don’t think there has been enough time to say the curve has flattened.
- MP_1729 2y agoTwo pushbacks. 1- The mania only started post Nov 22. And the huge investments since then didn't meant substantial progress since GPT-4 launch in March 22. 2- We are running out of high quality tokens in 2024. (per Epoch AI)
- dwaltrip 2y agoGPT-4 launch was barely 1 year ago. Give the investments a few years to pay off. I've heard multiple reports that training runs costing ~$1 billion are in the the works at the major labs, and that the results will come in the next year or so. Let's see what that brings. As for the tokens, they will find more quality tokens. It's like oil or other raw resources. There are more sources out there if you keep searching.
- hehdhdjehehegwv 2y agoThis is why think Meta has been so shrewd in their “open” model approach. I can run Llama3-70B on my local workstation with an A6000, which after the up-front cost of the card, is just my electricity bill. So despite all the effort and cost that goes into these models, you still have to compete against a “free” offering. Meta doesn’t sell an API, but they can make it harder for everybody else to make money on it.
- kmeisthax 2y agoLLaMA still has an "IP hook" - the license for LLaMA forbids usage on applications with large numbers of daily active users, so presumably at that point Facebook can start asking for money to use the model. Whether or not that's actually enforceable[0], and whether or not other companies will actually challenge Facebook legal over it, is a different question. [0] AI might not be copyrightable. Under US law, copyright only accrues in creative works. The weights of an AI model are a compressed representation of training data. Compressing something isn't a creative process so it creates no additional copyright; so the only way one can gain ownership of the model weights is to own the training data that gets put into them. And most if not all AI companies are not making their own training data...
- lolinder 2y ago> LLaMA still has an "IP hook" - the license for LLaMA forbids usage on applications with large numbers of daily active users, so presumably at that point Facebook can start asking for money to use the model. No, the license prohibits usage by Licensees who already had >700m MAUs on the day of Llama 3's release [0]. There's no hook to stop a company from growing into that size using Llama 3 as a base. [0] https://llama.meta.com/llama3/license/ https://llama.meta.com/llama3/license/
- Salgat 2y agoThe whole point is that the license specifically targets their competitors while allowing everyone else so that their model gets a bunch of free contributions from the open source community. They gave a set date so that they knew exactly who the license was going to affect indefinitely. They don't care about future companies because by the time the next generation releases, they can adjust the license again.
- spacebanana7 2y agoSam Altman gave the impression that foundation models would be a commodity on his appearance in the All in Podcast, at least in my read of what he said. The revenue will likely come from application layer and platform services. ChatGPT is still much better tuned for conversation than anything else in my subjective experience and I’m paying premium because of that. Alternatively it could be like search - where between having a slightly better model and getting Apple to make you the default, there’s an ad market to be tapped.
- hn_throwaway_99 2y agoI'm ceaselessly amazed at people's capacity for impatience. I mean, when GPT 4 came out, I was like "holy f, this is magic!!" How quickly we get used to that magic and demand more. Especially since this demo is extremely impressive given the voice capabilities, yet still the reaction is, essentially, "But what about AGI??!!" Seriously, take a breather. Never before in my entire career have I seen technology advance at such a breakneck speed - don't forget transformers were only invented 7 years ago. So yes, there will be some ups and downs, but I couldn't help but laugh at the thought that "14 months" is seen as a long time...
- belter 2y agoChair in the sky again...
- hn_throwaway_99 2y agoHah, was thinking of that exact bit when I wrote my comment. My version of "chair in the sky" is "But you are talking ... to a computer!!" Like remember stuff that was pure Star Trek fantasy until very recently? I'm sitting here with my mind blown, while at the same time reading comments along the lines of "How lame, I asked it some insanely esoteric question about one of the characters in Dwarf Fortress and it totally got it wrong!!"
- bamboozled 2y agoYou just be new here?
- IanCal 2y agoTbf gpt4 level seems useful and better than almost everything else (or close if not). The more important barriers for use in applications have been cost, throughout and latency. Oh and modalities, which have expanded hugely.
- adtac 2y ago>Token volume needs to double just for revenue to stand still Profits are the real metric. Token volume doesn't need to double for profits to stand still if operational costs go down.
- mrkramer 2y ago>This thing continues to stress my skepticism for AI scaling laws and the broad AI semiconductor capex spending. Imagine you are in 1970s and saying computers suck, they are expensive, there is not that many use cases....fast forward to 90s and you are using Windows 95 with GUI and chip astronomically more powerful that we had in 70s and you can use productivity apps , play video games and surf Internet. Give AI time, it will fulfill its true protentional sooner or later.
- MP_1729 2y agoThat's the opposite of what I am saying. What I am saying is that computers are SO GOOD that AI is getting VERY CHEAP and the amount of computing capex being done is excessive. It's more like you are in 1999, people are spending $100B in fiber, while a lot of computer scientists are working in compression, multiplexing, etc.
- jameshart 2y agoWhich of those investments are you saying would have been a poor choice in 1999?
- MP_1729 2y agoAll of them, without exception. Just recently, Sprint sold their fiber business for $1 lmfao. Or WorldCom. Or NetRail, Allied Riser, PSINet, FNSI, Firstmark, Carrier 1, UFO Group, Global Access, Aleron Broadband, Verio... All fiber went bust because despite internet's huge increase in traffic, the amount of packets per fiber increased a handful of magnitudes.
- madeofpalk 2y ago> Token volume needs to double just for revenue to stand still I'm pretty skeptical about all the whole LLM/AI hype, but I also believe that the market is still relatively untapped. I'm sure Apple switching Siri to an LLM would ~double token usage. A few products rushed out thin wrappers ontop of chatgpt ai, developing pretty uninspiring chat bots of limited use. I think there's still huge potential for this LLM technology to be 'just' an implementation detail of other features, just running in the background doing its thing. That said, I don't think OpenAI has much of a moat here. They were first, but there's plenty of others with closed or open models.
- drag0s 2y agowhat do you actually expect from an "agent"?
- MP_1729 2y agoAsk stuff like "Check whether there's some correlation between the major economies fiscal primary deficit and GDP growth in the post-pandemic era" and get an answer.
- Pr0ject217 2y ago"OpenAI is still working in GPT-4-level models." This may or may not be true - just because we haven't seen GPT-level-5 capabilities, does not mean that it does not yet exist. It is highly unlikely that what they ship is actually the full capability of what they have access to.
- MP_1729 2y agothey literally launched TODAY a GPT-4 model!
- bionhoward 2y agoimho gpt4 is definitely [proto-]agi and the reason i cancelled my openai sub and am sad to miss out on talking to gpt4o is, openai thinks it's illegal, harmful, or abusive to use their model output to develop models that compete with openai. which means if you use openai then whatever comes out of it is toxic waste due to an arguably illegal smidgen of legal bullshit. for another adjacent example, every piece of code github copilot ever wrote, for example, is microsoft ai output, which you "can't use to develop / otherwise improve ai," some nonsense like that. the sum total of these various prohibitions is a data provenance nightmare of extreme proportion we cannot afford to ignore because you could say something to an AI and they parrot it right back to you and suddenly the megacorporation can say that's AI output you can't use in competition with them, and they do everything, so what can you do? answer: cancel your openai sub and shred everything you ever got from them, even if it was awesome or revolutionary, that's the truth here, you don't want their stuff and you don't want them to have your stuff. think about the multi-decade economics of it all and realize "customer noncompete" is never gonna be OK in the long run (highway to corpo hell imho)
- fnordpiglet 2y agoWhere I work in the hoary fringes of high end tech we can’t secure enough token processing for our use cases. Token price decreases means opening of capacity but we immediately hit the boundaries of what we can acquire. We can’t keep up with the use cases - but more than that we can’t develop tooling to harness things fast enough and the tooling we are creating is a quick hack. I don’t fear for the revenue of base model providers. But I think in the end the person selling the tools makes the most and in this case I think it continue to be cloud providers. I think in a very real way OpenAI and Anthropic are commercialized charities driving change and commoditizing rapidly their own products and it’ll be infrastructure providers who win the high end model game. I don’t think this is a problem I think this is in fact inline with their original charters but a different path than most people view nonprofit work. A much more capitalist and accelerated take. Where they might make future businesses is in the tooling. My understanding from friends within these companies is their tooling is remarkably advanced vs generally available tech. But base models aren’t the future of revenues (to be clear tho they make considerable revenue today but at some point their efficiency will cannibalize demand and the residual business will be tools)
- MP_1729 2y agoI'm curious now. Can you give color on what you're doing that you keep hitting boundaries? I suppose it isn't limited by human-attention.
- fnordpiglet 2y agoYes it’s limited by human attention. It has humans in the loop but a lot of LLM use cases come from complex language oriented information space challenges. It’s a lot of classification challenges as well as summarization and agent based dispatch / choose your own adventure with humans in the loop in complex decision spaces at a major finserv.
- w10-1 2y ago> Since GPT-4 launch, token prices are falling 84% per year!! Good for mankind, but crazy for these companies The message to competitor investors is that they will not make their money back. OpenAI has the lead, in market and mindshare; it just has to keep it. Competitors should realize they're better served by working with OpenAI than by trying to replace it - Hence the Apple deal. Soon model construction itself will not be about public architectures or access to CPU's, but a kind of proprietary black magic. No one will pay for upstart 97% when they can get reliable 98% at the same price, so OpenAI's position will be secure.
- abrichr 2y ago> where are my agents? https://github.com/OpenAdaptAI/OpenAdapt/ https://github.com/OpenAdaptAI/OpenAdapt/
- golol 2y agoGPT-2: February 2019 GPT-3: June 2020 GPT-3.5: November 2022 GPT-4: March 2023 There were 3 years between GPT-3 and GPT-4!
- whimsicalism 2y agohardly anybody you are talking to even knows what gpt3 is, the time between 3.5 and 4 is what is relevant
- golol 2y agoIt doesn't make any sense to look at it that way. Apparently the GPT base model finised training in like late summer 2022, which is before the release of GPT-3.5. I am pretty sure that GPT-3.5 should be thought of as GPT-4-lite, in the sense that it uses techniques and compute of the GPT-4 era rather than the GPT-3 era. The advancement from GPT-3 to GPT-4 is what counts and it took 3 years.
- whimsicalism 2y agoI fully don't agree. > I am pretty sure that GPT-3.5 should be thought of as GPT-4-lite, in the sense that it uses techniques and compute of the GPT-4 era rather than the GPT-3 era Compute of the "GPT-3 era" vs the "GPT-3.5 era" is identical, this is not a distinguishing factor. The architecture is also roughly identical, both are dense transformers. The only significant difference between 3.5 and 3 is the size of the model and whether it uses RLHF.
- golol 2y agoYes you're right about the compute. Let me try to make my point differnetly: GPT-3 and GPT-4 were models which when they were released represented the best that OpenAI could do, while GPT-3.5 was an intentionally smaller (than they could train) model. I'm seeing it as GPT-3.5 = GPT-4-70b. So to estimate when the next "best we can do" model might be released we should look at the difference between the release of GPT-3 and GPT-4, not GPT-4-70b and GPT-4. That's my understanding, dunno.
- ugh123 2y ago>How people seriously expect these foundational model companies to make substantial revenue? My take on this common question is that we haven't even begun to realize the immense scale of which we will need AI in all sorts of products, from consumer to enterprise. We will look back on the cost of tokens now (even at 50% of price a year or so ago) and look at it with the same bewilderment of "having a computer in your pocket" compared to mainframes from 50 years ago. For AI to be truly useful at the consumer level, we'll need specialized mobile hardware that operates on a far greater scale of tokens and speed than anything we're seeing/trying now. Think "always-on AI" rather than "on-demand".
- siscia 2y agoNow a bit of Shameless plug, but of you need an AI to take over your emails then my https://getgabrielai.com https://getgabrielai.com should cover most use cases. * Summarisation * Smart filtering * Smart automatic drafting of replies Very much in beta, and summarisation is still behind feature flag, but feel free to give it a try. For summarisation here I mean to get one email with all your unread emails summarised.
- deleted 2y ago[deleted]