15 ms·

> It's basically just improvements to models ... Even though I'm expecting to be shocked by what use cases those improvements will unlock, I expect it will be

• ein0p 2y ago
  Not really, no. Much of the slack has already been taken up in GPT 3.5 era, and things haven’t dramatically improved since GPT4. 4o basically alludes to what I said above: it’s smaller and faster than 4, but the level of capability is very similar, the next big thing in it is multimodality, which is relatively straightforward to add. It just makes sense to pay attention to cost and efficiency now. It’s incredibly expensive today to both do research in this field and deploy the results of that research. Even FANGs are severely compute constrained to the point where when Google was training the bigger variants of Gemini I couldn’t even get a handful of TPUs scheduled to do my work (I’m not at Google now). That, quite obviously, is untenable. But even if NVIDIA/TSMC somehow tripled their production and everyone got as much compute as they can use, there’s no serious research going on for how to do cognition. The closest we got to that is neural network assisted Monte Carlo a-la AlphaZero and the various other approaches to search. But all of them are very narrow. Nobody has even a foggiest clue on how to bring anything cognition-like to the general domain. And don’t get me wrong, I’m a bona fide AI enthusiast, I’ve been in this field for over a decade. It’s just that all these promises of singularity “real soon now” are nothing more than clickbait.