5 ms·

I don't know why this reminds you of Sutton's essay. Sutton claims that algorithmic work will be superseded by brute-force approaches powered by exponential gro

I don't know why this reminds you of Sutton's essay. Sutton claims that algorithmic work will be superseded by brute-force approaches powered by exponential growth in computing power. But this paper is not the result of exponential growth in computing power. It's the result of $1b worth of Azure credits provided for free by Microsoft. Rich Sutton is a great scientist but he is fooled by randomness. He initiated his research program just as Moore's law was taking off. Thanks to Moore, his approach saw incredible success and brought him deserved acclaim. But just as Moore's law is pulling the rug from under him, he is using his stature to claim that no other approach but his can work.

justicezyx 6y ago
The engineering effort of utilizing 1B worth of Azure credits is surely worth at least 10% of it.
- master_yoda_1 6y ago
  Yes then credit to azure engineers not open ai people
  - justicezyx 6y ago
    Wow, do you reasonably believe OpenAI people are not contributing?... I thought HN visitors are not capable of that level of naive thinking.
    - deleted 6y ago
      [deleted]
clickok 6y ago
You have misunderstood Sutton's argument. Quoting Rich: One thing that should be learned from the bitter lesson is the great power of general purpose methods, of methods that continue to scale with increased computation even as the available computation becomes very great. The point isn't that improvements in our algorithms is unnecessary or unhelpful, rather that the algorithms we should focus on will be capable of scaling with arbitrary amounts of compute/data. Such as, for example, neural networks, where we see an almost constant rate of improvement (for the appropriate architecture) as more resources are added.
- deleted 6y ago
  [deleted]
- KKKKkkkk1 6y ago
  I think this argument is neither here nor there. For computer vision problems, we use convnets, which are models inspired by a biological model of vision. By doing that we are embedding our preconceived notions of what vision is into our models instead of throwing compute and data at the problem. Earlier attempts using multi-layer perceptrons have been massive failures. Is this consistent with Sutton's analysis or contrary to it?
  - clickok 6y ago
    It's consistent with it. Rich used to be very bullish on neural nets, then somewhat dismissive of them (due to the fragility/inadequacy of FCNs), and then increasingly enthusiastic as the renewed interest demonstrated that those problems could be overcome-- e.g., through better initialization, training, and (as you note) different architecture choices. His main concern was whether a method could keep working as more resources became available, as otherwise you would tautologically end up with something short of true artificial intelligence. The important thing is that the technique can scale with increasing data or compute without hitting a hard or soft limit.

justicezyx 6y ago

The engineering effort of utilizing 1B worth of Azure credits is surely worth at least 10% of it.

master_yoda_1 6y ago
Yes then credit to azure engineers not open ai people
- justicezyx 6y ago
  Wow, do you reasonably believe OpenAI people are not contributing?... I thought HN visitors are not capable of that level of naive thinking.
  - deleted 6y ago
    [deleted]

clickok 6y ago

You have misunderstood Sutton's argument. Quoting Rich: One thing that should be learned from the bitter lesson is the great power of general purpose methods, of methods that continue to scale with increased computation even as the available computation becomes very great. The point isn't that improvements in our algorithms is unnecessary or unhelpful, rather that the algorithms we should focus on will be capable of scaling with arbitrary amounts of compute/data. Such as, for example, neural networks, where we see an almost constant rate of improvement (for the appropriate architecture) as more resources are added.

deleted 6y ago
[deleted]
KKKKkkkk1 6y ago
I think this argument is neither here nor there. For computer vision problems, we use convnets, which are models inspired by a biological model of vision. By doing that we are embedding our preconceived notions of what vision is into our models instead of throwing compute and data at the problem. Earlier attempts using multi-layer perceptrons have been massive failures. Is this consistent with Sutton's analysis or contrary to it?
- clickok 6y ago
  It's consistent with it. Rich used to be very bullish on neural nets, then somewhat dismissive of them (due to the fragility/inadequacy of FCNs), and then increasingly enthusiastic as the renewed interest demonstrated that those problems could be overcome-- e.g., through better initialization, training, and (as you note) different architecture choices. His main concern was whether a method could keep working as more resources became available, as otherwise you would tautologically end up with something short of true artificial intelligence. The important thing is that the technique can scale with increasing data or compute without hitting a hard or soft limit.