3 ms·

This post oversimplifies the story by putting all the emphasis on compute power. Deep Blue using brute force to solve chess obviously fits this pattern, but the

This post oversimplifies the story by putting all the emphasis on compute power. Deep Blue using brute force to solve chess obviously fits this pattern, but the others? Let's take computer vision. Alex Krizhevsky et al destroyed the ImageNet competition with a neural network in 2012, kicking off the current AI hype cycle. Essentially everything in their model had been known about since the late 80s. But we also didn't know how to train deep networks much before this (it turned out how you initialise the neural network was important), and we also didn't have a big enough dataset to train such a deep model on until ImageNet. Since then, we have built models that perform another order of magnitude better than the 2012 model, mainly because of improvements to the architectures (a combination of ingenuity and a lot of trial and error). So compute is necessary, but it isn't enough, I don't buy that we've 'brute forced' image recognition in the same way as chess.

sdenton4 8y ago
Likewise, the search in Go is a Monte Carlo search, very different from the kind of search used in chess. And the neutral nets in alpha go are guiding where to run the search, which is very very different from brute force search. Many of these things have required the giant leaks in compute, but still wouldn't work at all without the concurrent improvements in algorithms. Along these lines, here's a classic blog post: https://www.johndcook.com/blog/2015/12/08/algorithms-vs-moores-law/ https://www.johndcook.com/blog/2015/12/08/algorithms-vs-moor... "Grötschel, an expert in optimization, observes that a benchmark production planning model solved using linear programming would have taken 82 years to solve in 1988, using the computers and the linear programming algorithms of the day. Fifteen years later — in 2003 — this same model could be solved in roughly 1 minute, an improvement by a factor of roughly 43 million. Of this, a factor of roughly 1,000 was due to increased processor speed, whereas a factor of roughly 43,000 was due to improvements in algorithms! Grötschel also cites an algorithmic improvement of roughly 30,000 for mixed integer programming between 1991 and 2008."
- SmooL 8y ago
  I think the authors point though is that all our effort into the algorithms where algorithms to do just one thing - search - and that we used that in conjunction with more compute power. I'll agree that the author emphasizes compute power, but his real point still holds. Monte Carlo search may not be classic brute force, and neural networks guiding it may also not be standard, but the two just let you effectively search on a massive scale.
colordrops 8y ago
I don't think what you are saying contradicts the text. What he's saying is that we need to put our efforts into how to design and use the tools that tackle the problem space, rather than reasoning about the problem space itself, e.g. how to use neural nets, monte carlo search, etc. That doesn't mean we just throw a for-loop at the data.
- sweezyjeezy 8y ago
  But this doesn't work either - convolutional layers in neural networks have a very specific structure, which encodes strong prior knowledge that we have about the problem space (translation invariance). If we just had multilayer perceptions, we wouldn't be talking about this right now.
  - coldtea 8y ago
    >convolutional layers in neural networks have a very specific structure, which encodes strong prior knowledge that we have about the problem space Yes. The point of the author is that it doesn't do this symbolically. Don't get confused with the terms "brute force", "neural net", etc. The main idea of the author is that AI that uses brute force, simpler statistical methods, NN, etc, wins over AI that tries to implement some deeper reasoning about the problem domain the way humans do (when thinking about it consciously).
    - alew1 8y ago
      Hmm, I'm not sure I see the difference. Why is it not "symbolic"? The symbols that construct the neural network are what encodes translation invariance -- not some vector of reals.
      - coldtea 8y ago
        Symbolic as in "symbolic algebra systems", "symbolic AI", etc [1]. Not as in having some symbols in the code for a NN. A NN doesn't work with the domain objects directly and abstractly (e.g. considering a face, facial features, smiles, etc as first class things and doing some kind of symbolic manipulation at that level). It crunches numbers that encode patterns capturing those things, but its logic is all about numbers, links between one layer and another, and so on -- it's not a program dealing with high level abstract entities. To put it in another way, it's the difference between teaching, say, Prolog to identify some concept and a NN to do the same. E.g. from the link "The most successful form of symbolic AI is expert systems, which use a network of production rules. Production rules connect symbols in a relationship similar to an If-Then statement. The expert system processes the rules to make deductions and to determine what additional information it needs, i.e. what questions to ask, using human-readable symbols." A NN does nothing like that (not in any immediate, first class, way, where the rules are expressed as plain rules given by the programmer, like "foo is X", "bar has the Y property", etc). Here's another way to see it: how you'd solve a linear equation with regular algebra (the steps and transformation etc), and how a NN would encode the same. A symbolic algebra system will let you express an equation in symbolic form (more or less like a mathematician would write it), and even show you all the intermediate steps you'd take until the solution. A NN trained to solve the same type of equations doesn't do that (and can't). It just tells you the answer (or an approximation thereof). [1] https://en.wikipedia.org/wiki/Symbolic_artificial_intelligence https://en.wikipedia.org/wiki/Symbolic_artificial_intelligen...
freshm087 8y ago
Your observations don't disprove anything in the article. The author doesn't say that brute force is the way to go, and one has to cease all tries to optimize just waiting for computing power to increase. He says that essentially all attempts to build AI by modeling human thinking lead to nowhere, because their are inherently too complex, with simpler statistical, or search-based methods constantly winning. What you wrote here is absolutely in line with his words. "Neural network" contrary to its name doesn't work by emulating human.

sdenton4 8y ago

Likewise, the search in Go is a Monte Carlo search, very different from the kind of search used in chess. And the neutral nets in alpha go are guiding where to run the search, which is very very different from brute force search. Many of these things have required the giant leaks in compute, but still wouldn't work at all without the concurrent improvements in algorithms. Along these lines, here's a classic blog post: https://www.johndcook.com/blog/2015/12/08/algorithms-vs-moores-law/ https://www.johndcook.com/blog/2015/12/08/algorithms-vs-moor... "Grötschel, an expert in optimization, observes that a benchmark production planning model solved using linear programming would have taken 82 years to solve in 1988, using the computers and the linear programming algorithms of the day. Fifteen years later — in 2003 — this same model could be solved in roughly 1 minute, an improvement by a factor of roughly 43 million. Of this, a factor of roughly 1,000 was due to increased processor speed, whereas a factor of roughly 43,000 was due to improvements in algorithms! Grötschel also cites an algorithmic improvement of roughly 30,000 for mixed integer programming between 1991 and 2008."

SmooL 8y ago
I think the authors point though is that all our effort into the algorithms where algorithms to do just one thing - search - and that we used that in conjunction with more compute power. I'll agree that the author emphasizes compute power, but his real point still holds. Monte Carlo search may not be classic brute force, and neural networks guiding it may also not be standard, but the two just let you effectively search on a massive scale.

colordrops 8y ago

I don't think what you are saying contradicts the text. What he's saying is that we need to put our efforts into how to design and use the tools that tackle the problem space, rather than reasoning about the problem space itself, e.g. how to use neural nets, monte carlo search, etc. That doesn't mean we just throw a for-loop at the data.

sweezyjeezy 8y ago
But this doesn't work either - convolutional layers in neural networks have a very specific structure, which encodes strong prior knowledge that we have about the problem space (translation invariance). If we just had multilayer perceptions, we wouldn't be talking about this right now.
- coldtea 8y ago
  >convolutional layers in neural networks have a very specific structure, which encodes strong prior knowledge that we have about the problem space Yes. The point of the author is that it doesn't do this symbolically. Don't get confused with the terms "brute force", "neural net", etc. The main idea of the author is that AI that uses brute force, simpler statistical methods, NN, etc, wins over AI that tries to implement some deeper reasoning about the problem domain the way humans do (when thinking about it consciously).
  - alew1 8y ago
    Hmm, I'm not sure I see the difference. Why is it not "symbolic"? The symbols that construct the neural network are what encodes translation invariance -- not some vector of reals.
    - coldtea 8y ago
      Symbolic as in "symbolic algebra systems", "symbolic AI", etc [1]. Not as in having some symbols in the code for a NN. A NN doesn't work with the domain objects directly and abstractly (e.g. considering a face, facial features, smiles, etc as first class things and doing some kind of symbolic manipulation at that level). It crunches numbers that encode patterns capturing those things, but its logic is all about numbers, links between one layer and another, and so on -- it's not a program dealing with high level abstract entities. To put it in another way, it's the difference between teaching, say, Prolog to identify some concept and a NN to do the same. E.g. from the link "The most successful form of symbolic AI is expert systems, which use a network of production rules. Production rules connect symbols in a relationship similar to an If-Then statement. The expert system processes the rules to make deductions and to determine what additional information it needs, i.e. what questions to ask, using human-readable symbols." A NN does nothing like that (not in any immediate, first class, way, where the rules are expressed as plain rules given by the programmer, like "foo is X", "bar has the Y property", etc). Here's another way to see it: how you'd solve a linear equation with regular algebra (the steps and transformation etc), and how a NN would encode the same. A symbolic algebra system will let you express an equation in symbolic form (more or less like a mathematician would write it), and even show you all the intermediate steps you'd take until the solution. A NN trained to solve the same type of equations doesn't do that (and can't). It just tells you the answer (or an approximation thereof). [1] https://en.wikipedia.org/wiki/Symbolic_artificial_intelligence https://en.wikipedia.org/wiki/Symbolic_artificial_intelligen...

freshm087 8y ago

Your observations don't disprove anything in the article. The author doesn't say that brute force is the way to go, and one has to cease all tries to optimize just waiting for computing power to increase. He says that essentially all attempts to build AI by modeling human thinking lead to nowhere, because their are inherently too complex, with simpler statistical, or search-based methods constantly winning. What you wrote here is absolutely in line with his words. "Neural network" contrary to its name doesn't work by emulating human.