4 ms·
I got a very different message from this, actually much closer to the problem of incumbent advantage. The known-good thing has been heavily optimized for perfo
by sdenton4 4mo ago
I got a very different message from this, actually much closer to the problem of incumbent advantage.
The known-good thing has been heavily optimized for performance, making it much harder for new technologies to prove that they are better. This is similar to the problem of gas vs electric engines - we had a century of optimization and ecosystem development around gas engines, which creates an uphill battle for electric motors even though they are (eventually) superior on every way /except/ having that massive ecosystem.
The problem isn't as bad here, because software is much more flexible than hardware, and scaling laws give a reasonable way to try things out at smaller scale before going whole hog.
- lukeschlather 4mo agowhat is the known-good thing? The whole point is that LLMs were not optimized at all, they got better results than older ML algorithms just because they are able to use all of the GPU, where older algorithms are designed for 10yo GPUs and can't make use of modern GPUs. But now you do in fact have to optimize, to the point that transformers look a lot more complicated than "attention is all you need."
- RugnirViking 4mo agothe known-good thing is whatever was previously used for the application you're using llms for in any given case. This is often the structured application of human intelligence.