3 ms·

> Asymptotic improvements are flattening the cost curves so fast that AI regulation might become practically meaningless by the end of the year. Awesome. This

> Asymptotic improvements are flattening the cost curves so fast that AI regulation might become practically meaningless by the end of the year. Awesome. This will mean actually good open-source models, not just API endpoints by big tech which are unusable because of dataset censorship and bias alignment (SD3, Gemini). In other words, big tech will actually need to make good stuff to be competitive, not trash protected by a granted monopoly.

sigmoid10 2y ago
Those improvements are definitely real, but we also have pretty solidly established and confirmed scaling laws by now. Until someone utterly breaks those, big players will always have an edge, simply because they can spend more compute on training and inference. The only way to change this is with a new architecture that benefits more from intelligent adjustments in a space than cannot be searched efficiently with raw compute. And even then we are not far from the point where these models could try out those adjustments themselves. So by the time you get to tune your own GAI in your home like you could do with a human, corporations might have millions of them improving themselves to something you could never achieve on your own.
- worstspotgain 2y ago
  We're still in phase 1, where human-directed improvement has the highest potential. Papers are still getting published and the cells interlinked. (I'm not sure the scaling picture is at all clear, given that papers like this can turn up casually with 15x savings, but let's put that aside for now.) Phase 2 begins when patents break stealth, unsettling the picture. If some patent impairs research or operations in IP-solid countries, the lower-level stuff might move to local inference, and maybe some minor Pirate Bay-style outfits. Phase 3 begins when the costly research goes dark (well, darker.) Everyone is Apple now. The research papers are replaced by white papers, then by PR communiqués. Phase 4 begins when the AI AI researchers take over. The old AI researchers turn into their managers. Some of the path is compute-bound. Some of it is IP-, luck-, and genius-bound.

sigmoid10 2y ago

Those improvements are definitely real, but we also have pretty solidly established and confirmed scaling laws by now. Until someone utterly breaks those, big players will always have an edge, simply because they can spend more compute on training and inference. The only way to change this is with a new architecture that benefits more from intelligent adjustments in a space than cannot be searched efficiently with raw compute. And even then we are not far from the point where these models could try out those adjustments themselves. So by the time you get to tune your own GAI in your home like you could do with a human, corporations might have millions of them improving themselves to something you could never achieve on your own.

worstspotgain 2y ago
We're still in phase 1, where human-directed improvement has the highest potential. Papers are still getting published and the cells interlinked. (I'm not sure the scaling picture is at all clear, given that papers like this can turn up casually with 15x savings, but let's put that aside for now.) Phase 2 begins when patents break stealth, unsettling the picture. If some patent impairs research or operations in IP-solid countries, the lower-level stuff might move to local inference, and maybe some minor Pirate Bay-style outfits. Phase 3 begins when the costly research goes dark (well, darker.) Everyone is Apple now. The research papers are replaced by white papers, then by PR communiqués. Phase 4 begins when the AI AI researchers take over. The old AI researchers turn into their managers. Some of the path is compute-bound. Some of it is IP-, luck-, and genius-bound.