3 ms·
The problem with this take is that you're conflating different software systems under the same vague AI umbrella. Waymo is a completely different technology tha
by __loam 2y ago
The problem with this take is that you're conflating different software systems under the same vague AI umbrella. Waymo is a completely different technology than generative AI. Talking about how over a decade and billions of dollars in R&d in self driving is finally bearing fruit has nothing to do with the performance of LLMs.
It is true that LLMs appear to be an exciting technology, but it's also delusional to assume they're following a positive trendline. Performance between GPT-3.5 and GPT-4 was like a 25% improvement that took 2500% more resources to train. It's clear that there's diminishing returns to bigger and bigger models, and the trend we've been seeing in the industry is actually smaller models trained for longer periods in an effort to bring down inference costs while maintaining current performance.
More intelligent models may require new techniques and technologies that we don't have yet. I'm sure it will get better but the path to improving isn't as obvious as you're making it sound. Making comparisons to Moore's law is also disengenuous because we're actually running into physical limits on how dense we can make chips due to the size of atoms themselves, so past trends for technological development may not continue to bear out.
- ninetyninenine 2y agoObviously the overall trend I'm referring to is AI from a more general perspective. Deep learning. You are doing exactly as I said. Focusing on obvious criticisms on the current state of the art. We know the obvious pitfalls of LLMs. It's completely obvious nowadays. I also never pointed to an obvious path forward. I pointed to the an obvious trend that indicates whether you like it or not, we will move forward. >It is true that LLMs appear to be an exciting technology, but it's also delusional to assume they're following a positive trendline. Performance between GPT-3.5 and GPT-4 was like a 25% improvement that took 2500% more resources to train. It's clear that there's diminishing returns to bigger and bigger models, and the trend we've been seeing in the industry is actually smaller models trained for longer periods in an effort to bring down inference costs while maintaining current performance. AI is following a trendline. I never said specifically LLMs are exactly on this trendline. LLMs are only a part of this trendline along with other technologies and part of the overall progress deep learning is making. It is extremely likely that there will be an AI that will solve all of current issues with LLMs in the coming decades. Whether that AI is some version of an LLM remains to be seen. >Making comparisons to Moore's law is also disengenuous because we're actually running into physical limits on how dense we can make chips due to the size of atoms themselves, so past trends for technological development may not continue to bear out. I never made a comparison to moores law... are you replying to me or someone else? AI with the performance of a human brain is highly realizable despite physical limits because intelligence to the level of the human brain already EXISTS. The existence of humans themselves is testament to the possibility it can be done and is not a fundamental limit.
- __loam 2y agoThe brain is not a computer. Continued insistence that it is will result in heartbreak for the computer scientists involved.
- ninetyninenine 2y agoFortunately nothing of the sort has happened. Not sure what you’re referring to here.