4 ms·
It's the bitter-lesson to feature-engineering lifecycle. When a technique or technology is new people are making massive gains by just applying it to some use
by jordanb 4mo ago
It's the bitter-lesson to feature-engineering lifecycle.
When a technique or technology is new people are making massive gains by just applying it to some use case, or gathering more data for training, or giving it more resources.
As time goes on those "bitter lesson" gains start to hit the shallow part of the logistic curve and companies have to start investing more and more effort into engineering for each small, incremental gain.
- vjvjvjvjghv 4mo ago[dead]
- pezo1919 4mo agoWell put, thanks.
- sdenton4 4mo agoI got a very different message from this, actually much closer to the problem of incumbent advantage. The known-good thing has been heavily optimized for performance, making it much harder for new technologies to prove that they are better. This is similar to the problem of gas vs electric engines - we had a century of optimization and ecosystem development around gas engines, which creates an uphill battle for electric motors even though they are (eventually) superior on every way /except/ having that massive ecosystem. The problem isn't as bad here, because software is much more flexible than hardware, and scaling laws give a reasonable way to try things out at smaller scale before going whole hog.
- lukeschlather 4mo agowhat is the known-good thing? The whole point is that LLMs were not optimized at all, they got better results than older ML algorithms just because they are able to use all of the GPU, where older algorithms are designed for 10yo GPUs and can't make use of modern GPUs. But now you do in fact have to optimize, to the point that transformers look a lot more complicated than "attention is all you need."
- RugnirViking 4mo agothe known-good thing is whatever was previously used for the application you're using llms for in any given case. This is often the structured application of human intelligence.
- deleted 4mo ago[deleted]
- zahlman 4mo agoI assume the choice of phrase "bitter lesson" is intentional irony (since the original concept is that you get better results by just scaling up and not trying to be clever with domain-specific knowledge)?
- Zarathruster 4mo agoI assume that the bitterness of the bitter lesson is not for engineers but for subject matter experts. I can only imagine how it would feel to discover that your decades of hard-earned expertise don't amount to a whole lot when it comes to domain-specific ML modeling, compared to simply throwing more compute at the problem.
- clickety_clack 4mo agoMaybe the ultimate bitter lesson is that entropy always wins in the end.
- cindyllm 4mo ago[dead]
- daynthelife 4mo agoMy read of GP is that there is a cycle economy between "bitter lesson" style domain-general scaling and domain-specific adaptation once the scaling plateaus for the latest tech.
- rcxdude 4mo agoSpecifically the you get better results with techniques that can scale up to the amount of data you have. People often think of it as a "just brute force it" but for the lesson to apply you do need to come up with how you're gong to get the data and how you're going to use it.