4 ms·
In my opinion, you've grossly mischaracterized and misunderstood the bitter lesson. The bitter lesson is not saying that brute force will win out against better
by abetusk 2y ago
In my opinion, you've grossly mischaracterized and misunderstood the bitter lesson. The bitter lesson is not saying that brute force will win out against better algorithms, it's saying that simple, efficient algorithms, in a theoretical sense, will win out against intricate and specialized algorithms.
That is, it's saying that worrying about the constant factor in the "big-Oh" is doomed to failure. My reading is that it's not saying that we should abandon polynomial time algorithms in favor of exponential ones, it's saying we should focus on polynomial time algorithms that are conceptually simple and let Moore's law take care of the constant factor differences between intricate, complex and "clever" algorithms. It's saying that Moore's law will obliterate the constant factor differences in algorithms, and potentially the small polynomial exponent differences, not that we shouldn't care about exponential time complexity.
The "Bitter Lesson" blog post came out during a time when people were debating whether simple algorithms on large datasets could win out against more complex algorithms with intricate domain knowledge. That is, did AGI, vision, language, translation, speech, etc. need intricate and deep domain knowledge or was it good enough to get coarser and "stupider" algorithms, that were still theoretically efficient/polynomial time efficient algorithms.
I can see why you would think this way and I certainly don't claim to have deep insight into Sutton to claim whether my reading is more correct than yours. From my perspective, the good faith reading of Sutton's article is talking about constant factor differences in algorithms, not in differences between polynomial time and exponential time worst case complexities.
- m463 2y agoI wonder if an analogy with graphics would be that "simple algorithms that leveraged compute" would not be "brute force everything with 1000 gpus + ray tracing", it might be a pixel shader that simulated water taking the state of the art a bit further.