8 ms·

The Bitter Lesson (2019) [pdf]

• fnbr 3y ago
  This is the original page, I’d link to this: http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html u/dang, swap links if you see this?
  • petesergeant 3y ago
    Yah, entirely unclear why this has been submitted as a browser-saved PDF of the original HTML, when the original HTML is right there!
  • cwillu 3y ago
    Email is the correct way to acquire his attention for things like this.
• Animats 3y ago
  On HN before, at least once.[1] [1] https://news.ycombinator.com/item?id=36017857 https://news.ycombinator.com/item?id=36017857
• hyperthesis 3y ago
  Just adding an obvious corollary regarding parallelization, to "leverage computation" and "massive computation became available and a means was found to put it to good use": Parallelizable methods win (e.g. DL), by using the computation available Further: progress will occur on real-world problems that can be solved by those methods, making those applications dominant throughout society. (i.e. everything looks like a nail to someone with a hammer... but if it's a really good hammer, that may be the best approach)
  • seanhunter 3y ago
    Yes. Prior to this though in the AI community there was a lot of debate about the pros and cons of "weak search" (ie generalized methods) and "strong search" (ie methods which make use of domain knowledge). The bitter lesson is more or less that in the long run weak search always wins and the only real judgement call is whether strong search can give you a transient benefit in a particular case before weak search surpasses it and whether that is worth it. In my head this always parallelled the "premature optimization" conversation in programming. Most programmers would say that inlining etc was only justified when you've benchmarked so you know how the benchmarks are performing but the experience of whole program optimization in things like hotspot jvm and llvm seems to suggest that even benchmarked optimizations can be premature because only a vm can do optimizations on the real world use case.
• v9v 3y ago
  Also see A Better Lesson: https://rodneybrooks.com/a-better-lesson/ https://rodneybrooks.com/a-better-lesson/
  • PoignardAzur 3y ago
    Ehh, I'm more on Sutton's side so far. Brook's post goes over the classics (Moore's law is ending, curating a dataset requires human intervention, etc) and posits that making a huge model won't be a competitive strategy for long because it gets to expensive to train and use. It's a bit early to tell, but so far that hasn't materialized. OpenAI got state-of-the-art results with GPT-4, AFAIK by sticking for very-super-big models together. Open source experiments with LLAMA show you can still get good results with heavy quantization. Distillation hasn't be too explored by mainstream projects, but I bet there's lots of potential there too. Right now the winning strategy looks to be "go really big, then figure out how to go small".
    • PartiallyTyped 3y ago
      Going really big and figuring out how to go small is also working; see Lottery Tickets, ensembles, distillation, and so on.
      • tysam_and 3y ago
        As someone who holds a speed WR with a very tiny model (<~8.5 MB or so), I would argue kindly that Sutton's lesson is about methods that scale -- and scale goes in the opposite direction too! If you do not have solid scaling, your link between micro methods predicting mega methods is broken. Hence, Sutton's bitter law forms the foundation for a few other lemmas that I think underpin what makes really effective research (which is iteration time, and how we effectively reduce it as much as possible and make it as accessible as possible -- which thankfully for ML algorithms seems to go hand in hand! <3 :')))) )
  • papercurt 3y ago
    I think the convergence to transformer architectures already proofs this article from 2019 wrong in some ways. Even though CNNs did and do a good job, they will most likely be superseded by vision transformers. Of course you could argue that stable diffusion models and LLMs also have some domain knowledge inside. But it might be a little less with multimodal networks. So I guess the lesson should be: Don't put more domain knowledge inside than what's needed to make a product out of it right now, because everything else won't progress the field.
• isaacfrond 3y ago
  I think the Go example is facetious. For the longest time, Go simply was not amenable to search. It is only because of breakthrough in theory that Go became amendable to search. And, it is hardly a straightforward search mind you. Two self learning networks are involved. A first to guide the expansion of a Monte Carlo based tree search, a second to evaluate the nodes in the tree. You can hardly blame a researcher in the 70s, 80s, 90s, 00s for not focusing on computer power, when you have 19^2 expansion factor for each move.
  • PoignardAzur 3y ago
    It's not called "the bitter lesson" for being optimistic. One of the points, I think, is that there is very little a researcher in the 70s could have done to make progress on these problems, because the computing power just wasn't there yet; and once the computing power was here, almost all of that 70s researcher's work became obsolete.
  • reacweb 3y ago
    Go has been described as the drosophila of AI. Once a program will know how to play go, we'll be able to make more general artificial intelligences. This is exactly what has happened with alphago and chatgpt. We're not repeating the same thing that happened with chess, but we are past the singularity. Recent advances in AI are lighting up the very nature of intelligence. Our memory and our brain are a machine for predicting the future. It's very similar to what recent models do. Alphagozero has rediscovered the history of chess openings in the same order as humans. We are no longer discovering simple new algorithms supported by machine advances, but architecting predictive systems based on guided learning. When we developed the algorithm for playing chess, the possibilities for generalization were limited. With these new architectures, new possibilities are appearing every day. IMHO, the predictions were true: since alphago, we've entered a new world.
    • YeGoblynQueenne 3y ago
      >> Go has been described as the drosophila of AI. John McCarthy described chess as "the drosophila of AI": http://jmc.stanford.edu/articles/drosophila/drosophila.pdf http://jmc.stanford.edu/articles/drosophila/drosophila.pdf Who was it that described Go as the drosophila of AI?
• niemandhier 3y ago
  I think the Bitter Lessons do not consider two important points with regards to neural nets : 1. Not all nets work for all problems, those that work tend to have the right inductive biases. We discovered the architectures partially by trial an error, nevertheless they work because of encoded prior information. 2. Data and computation are bounded. GPT4 was basically trained on all text, further advancements probably need more insight not more data.
  • psychoslave 3y ago
    What do you mean with more insights? If they are not represented as some data, how could they ever influence any computational model?
    • davegirdywww 3y ago
      Nobody said anything about insights.
    • niemandhier 3y ago
      By building the model in such a way that assumptions about the data are in the model structure. ConvNets hardcode translation equivariance, more general convolutions can hardcode more general equivariant structures. The whole field of geometric deep learning is about constructing nets based on insights about the structure of the data manifold, and many successful nets turn out to do this.
  • the8472 3y ago
    > GPT4 was basically trained on all text Well, there's multi-modal training. There's tons of untapped audiovisual data.
    • reaperman 3y ago
      GPT4 is a multi-modal model. They haven't exposed a way to use the image embeddings, so consumers can't utilize it, but the model accepts image input and it was trained on images and text. Yes, there are other modals that can be incorporated, and reinforcement learning is still pretty nascent/very much unsolved.
• Xcelerate 3y ago
  > The second general point to be learned from the bitter lesson is that the actual contents of minds are tremendously, irredeemably complex; we should stop trying to find simple ways to think about the contents of minds, such as simple ways to think about space, objects, multiple agents, or symmetries. I agree and think “ML explainability” efforts are doomed to fail as ML becomes increasingly more effective. There is no a priori reason that the human brain should be capable of intuitively grokking sufficiently advanced general learners. We can invent them and improve them, but saying that we will be able to understand what the myriad matrix multiplications are “doing” will be like saying we understand the human brain because we can model the physics of its constituent atoms. The emergent complexity is too high for us to make any sense of it.
  • nologic01 3y ago
    This (popular) opinion makes no sense. How are you going to improve something you dont understand? Throwing ramdom software pieces to see what sticks? The recent progress was possible because people understood intuitively enough of the limitations of earlier models to think and invent a remedy.
    • randomlogic 3y ago
      (Reads:) "Did you know that time-traveling was invented, cos time was seen as a threat to life ?" P-:
    • Tuna-Fish 3y ago
      Not that I necessarily agree with the above poster, but this: > How are you going to improve something you dont understand? Is just nonsense. Evolution understands nothing, yet produced a mind. Closer to us, the early people who produced all the crops that led to the shift to agriculture, and it's later improvements, absolutely did not understand how any of it worked. Evaluation and selection are sufficient to improve things. Understanding is useful, but optional.
      • nologic01 3y ago
• rstuart4133 3y ago
  I don't doubt the taste was bitter at the time, but it was never going to be forever. In 2018 models had 100e6 parameters, and in 2013 GPT4 has (wild guess) 2e12 parameters [0]. Conservatively that's 500% per year. At it's peak, hardware never improved at that rate and has since slowed down dramatically. So it was always going to end, and further advancement was always going to revert being driven by a neural net of some sort. In fact, it looks to me we are already at that point. The interesting question is what neural net will end up driving it. [0] https://www.qualcomm.com/news/onq/2023/07/generative-ai-trends-by-the-numbers-costs-resources-parameters https://www.qualcomm.com/news/onq/2023/07/generative-ai-tren...
• alphabetting 3y ago
  this thread on a "bitter lesson 2.0" relating to robotics from earlier this year is interesting https://twitter.com/hausman_k/status/1613544836266885120 https://twitter.com/hausman_k/status/1613544836266885120
  • markisus 3y ago
    I only see the first post in the series when I click this link. Do I have to click something else to see the rest of the thread?
• kitanata 3y ago
  [flagged]
• cubefox 3y ago
  Sutton cites people who were not convinced by the success of Deep Blue in chess: > They said that ``brute force" search may have won this time, but it was not a general strategy, and anyway it was not how people played chess. But it is now clear that these people were completely right! The Deep Blue approach (apparently mostly brute force search) didn't scale. E.g. Go wasn't solved this way, and the AlphaGo approach (based on reinforcement learning) did in fact scale to a lot of other games with AlphaZero and MuZero. Sutton in fact says that both search and learning produce the big success in AI. But now, with the dominance of ML, it seems that mainly learning is responsible the big successes in AI, not search. Machine learning means that the AI models aren't written by hand, GOFAI style, like Deep Blue, but instead by a learning algorithm, which in turn is written by hand. That seems to be the breakthrough, not brute force search. (Do GPTs use "search" in any conventional way? I don't think so.)
  • cs702 3y ago
    We train deep learning models with SGD, which is in fact a form of search -- in a high-dimensional parameter space. The purpose of SGD is to find parameter values that minimize a training loss.
    • spmurrayzzz 3y ago
      I was thinking along similar lines, it does smell like search to me at its core. But do you think there is a meaningful differentiation for something more nuanced like AdamW? Does the latter fall into the "brute force search" category too though it tends to converge faster (and arguably more optimally than SGD per Jeremy Howard's observations)?
  • brresnic123 3y ago
    SGD is a method of learning from examples (mapping X -> y). RL is a method of search (maximizing reward signal in a potentially open-ended environment, based on feedback on the actions that a model takes). Go was solved by combining learning and search (e.g. narrowing the search space with learned, educated guesses). Pretrained dense GPTs don't search in any conventional way. However, when these GPTs are fine-tuned using RL, an aspect of search is reintroduced. One the most interesting directions labs are pursuing to improve on today's best LLMs is to fine-tune them with search (RL) after a learning-based (next-token prediction) pre-training.
• nextworddev 3y ago
  The money quote (basically says why deep learning will keep on dominating): "One thing that should be learned from the bitter lesson is the great power of general purpose methods, of methods that continue to scale with increased computation even as the available computation becomes very great. The two methods that seem to scale arbitrarily in this way are search and learning."
• tysam_and 3y ago
  Hi, I say this so much that it is almost a revolving PSA at this point, but here we go again: Big reminder that the Bitter Lesson is _not_ saying "just scale your methods and they work". What the Bitter Lesson _is_, however, is "work on methods that scale". There is a _huge_ distinction between the two, in my opinion. Effectively, if I can add my layer of (re?)interpretation on it, it's saying that specialized boutique, hand-designed solutions don't play very well in the long-term in an arena with Moore's Law and money. But, what it's not saying, is that just making an algorithm bigger is the solution. This is how I see most people missing its interpretation. For an algorithm to be effective, it needs 3 things in my opinion: 1. It needs to scale (measurable by some factor) 2. Because we have 1, it needs to have an implementation with extraordinarily rapid iteration time 3. If we have 2, we need an implementation that is extraordinarily lightweight (enough to run on consumer machines) These three factors together, in my personal estimation, unlock algorithmic research progress in an area. An ancillary fourth rule that really drives progress IMO is competition, formalized or otherwise, that is 1. open, 2. well-known, 3. incentivized and 4. accessible. This is my personal opinion of course, and bound to be flawed in some way, but -- from personal experience, at least -- when I've seen this field (or research fields in general) align with this kind of method of research is where the speed of algorithmic research has absolutely exploded. :) <3