4 ms·
Only for a month, could believe it's about making sure it's polished. More generally though, the fact that matching GPT-4 is the marketing blurb as opposed to
by heyitsguay 3y ago
Only for a month, could believe it's about making sure it's polished.
More generally though, the fact that matching GPT-4 is the marketing blurb as opposed to a GPT-3 -> GPT-4 level transition has me wondering: are we seeing the limits of current "just scale the data" approaches to ML? Like, we've basically scraped everything worth scraping off the web, and when you do that with any of our current techniques, it lands you somewhere around GPT-4?
I don't have the link anymore, but i saw a quote from a LinkedIn connection recently at one of the FAANG's about how data matters so much more than architecture for ML today. To paraphrase, it went something like "it turns out that when your data is large enough, architecture doesn't matter. GANs converge to about the same thing as diffusion models. Convnets converge to about the same thing as ViTs. Data size and quality matter a lot more".
- deleted 3y ago[deleted]
- behnamoh 3y ago> has me wondering: are we seeing the limits of current "just scale the data" approaches to ML? This was my thought too. You'd think Gemini would set higher goals (higher than GPT-4) given that it will come out one year after GPT-4, but no, they are just achieving GPT-4-level performance in some respects. Maybe this says less about Google's lack of AI capabilities and more about OpenAI's first-mover advantage that made them pour billions into what was (like you said) the asymptote of AI intelligence using current arch.
- echelon 3y ago> pour billions into what was (like you said) the asymptote of AI intelligence using current arch. What a wild change in fortune that would be. OpenAI/Microsoft looked to be forever ahead, but maybe this is the break everyone else needs to catch up. I'll be really happy if we wind up with more companies owning different parts of the market rather than just FAANG at the top. Or more competition in general.
- jiggawatts 3y agoThe bulk of the GPT-4 training was concluded back in 2021, before the billions of dollars thrown at them by Microsoft. It's notable that the scale, design, and architecture of GPT-4 is a closely guarded trade secret of "Open"AI. This strongly hints at some critical discovery, some new technique, or some aspect dialed up to 11. Whatever it is, the competition hasn't figured it out, or managed to copy it.
- deleted 3y ago[deleted]
- smegsicle 3y agoit turns out that machines trained on literature are only half intelligent much as human linguists may seem deep at first but philosophically prove to be barren
- suprjami 3y agoApparently that is correct, this did the rounds recently where Bill Gates suggests current GPT has hit a plateau: https://techstory.in/bill-gates-5-cents-on-generative-ai-gpt-5-will-not-be-much-better-than-gpt-4/ https://techstory.in/bill-gates-5-cents-on-generative-ai-gpt...
- aantix 3y agoWhy does Gates get credibility? He hasn't programmed for decades. His relevant business decisions were 2+ decades ago. He's a proponent of a plant-based diet - looks very unhealthy. He's the Oprah Winfrey of technology.
- mvdtnz 3y agoBecause he tends to speak about topics on which he is very well informed, like vaccines, climate change and malaria.
- Jerrrry 3y agoThe same reason Elon Musk does, they have billion-dollar think tanks advising them.
- summerlight 3y agoGates is still one of the major stakeholder of MSFT. Combined with his previous position in MSFT, he probably has a good level of access to its long term strategy. Prediction of how OpenAI model will evolve is pretty essential information there?
- behnamoh 3y agohe seems to get access to these models way before anyone else. they showed him GPT-4 in the August of 2022!