3 ms·

Back when I used to frequently visit Less Wrong, I made a post about this related to Watson from IBM. I remember feeling really discouraged by the level of dis

by mlthoughts2018 7y ago

Back when I used to frequently visit Less Wrong, I made a post about this related to Watson from IBM. I remember feeling really discouraged by the level of discourse and downvoting in the comments at that time. https://www.lesswrong.com/posts/kaNErr6mbXvDF9YFf/watts-son https://www.lesswrong.com/posts/kaNErr6mbXvDF9YFf/watts-son

gwern 7y ago
That's too bad, because your comments there are really bad. You argue solely by assertion and personal feeling, and then wonder why no one agrees with you. Why does it matter in the least that you are not 'impressed' or feel using more than 1 server 'waters it down'? Why does it matter how much IBM Watson uses? AlphaGo uses more electricity than any human; does it not win at Go? If a NN uses more electricity than 150 watts to classify images at superhuman rates, why does this matter? Is there some law of physics I am unaware of which stipulates that no computer is allowed to use more than 150 watts, or that we are unable to build systems which have more than 1 computer involved? Is there some theorem in computer science which proves that AI isn't real AI unless it solves problems in exactly the same way as a human brain at exactly the same efficiency, never mind how well it solves problems or how much superior to humans it may be otherwise? ('Jet planes, you see, do not actually provide "artificial flight" because they burn more joules than a bird would.') Is there some reason to think that AI can never be misused or cause problems if it happens to use somewhat more electricity? Have deep learning techniques not continued to scale over the 8 years since you posted that? IBM Watson failed; did it have anything to do with using 15TB RAM; would Watson now be a success and closer to 'real AI' if they had hand-optimized its routines for memory efficiency to do the same thing but only needing 1.5TB RAM?
- mlthoughts2018 7y ago
  > “AlphaGo uses more electricity than any human; does it not win at Go?” I know you have a prolific reputation, and I mean this sincerely, but did you actually read the link? The point was that with power constraints, the class of possible algorithms is restricted heavily, requiring development of an understanding closer to what algorithms a person must use in their mind. If you present me a Go-based Turing test, and I’m pretty fooled by AlphaGo, except I get blasted by the hot air venting out from a big cooling system, then it probably reveals to me that it’s failed the test (at least until some far future where there are life forms that regularly vent computer-levels of hot air from cooling systems). This effect is much more pronounced with a larger system like Watson, hence the article. > “If a NN uses more electricity than 150 watts to classify images at superhuman rates, why does this matter?” It matters because pure computing achievements are not isolated from their resource constraints. If a NN can generate flawless natural language responses, but it takes 1000 years per word, we would not be interested and certainly would not see it as valuable progress towards artificial intelligence (likely not even strong AI, but surely not tool AI, there just aren’t a lot of ways to demonstrate a conceptual breakthrough so wasteful of resources). Intelligence is (partly) generating language within human-level time constraints. Why not also human level RAM, physical volume, material density, etc. Solving it within these constraints is much more valuable than solving it without them. Likewise with power. Maybe using 100x power is an OK tradeoff for pure tool usage (heck, I work professionally in computer vision and NLP and I make this concession to use machine learning models running on computers to solve my work daily). But that doesn’t mean it’s an ok trade-off for the philosophical side of the questions, like approximating a Turing test or understanding how far we must go from modern deep neural networks to the algorithms an actual brain is using. And, ultimately, these will move from being philosophical issues to being practical ones. > “Is there some reason to think that AI can never be misused or cause problems if it happens to use somewhat more electricity?” Huh? Where did I ever say anything like this? I mean, humans misuse intelligence and cause problems on less power. It sounds like you’re trying to propagate your feelings about the “green” aspect of the post into what I’ve said, but if you’ll notice, I did not say anything about that, and don’t particularly care about this OP article’s take on “green” AI power consumption, it’s not related at all to what I’m talking about. (Plus they are more focused on power demands at training time, while I’m interested in power constraints, like runtime or space constraints, at inference time).
  - gwern 7y ago
    > I know you have a prolific reputation, and I mean this sincerely, but did you actually read the link? I did, and all the comments, to see where you were mistreated so badly that you'd drag it up 8 years later, like the Bourbons, having forgotten nothing and learned nothing. > The point was that with power constraints, the class of possible algorithms is restricted heavily, requiring development of an understanding closer to what algorithms a person must use in their mind. That's not true at all: it is not true that intelligences must exactly imitate humans to be intelligent, and if it was, it would not be true either as a lower bound, upper bound, or average case that an algorithm using the same energy as a human must be isomorphic to the human algorithm. Consider deep learning. The more compute it uses, the better it gets, and the more human-like the results are, and this is true despite the fact that they typically use more power than a human brain even at the smaller scales where they are often among the worse-performing and still-power-hungry of methods, and power consumption goes up from there; by your logic, they would be getting even less human-like, if power consumption and algorithmic equivalency were so inextricably intertwined. But they aren't. 'The bitter lesson' is itself a disproof of your claim that power efficiency has anything to do with potential for future progress or human-like-ness of algorithms. > But that doesn’t mean it’s an ok trade-off for the philosophical side of the questions, like approximating a Turing test or understanding how far we must go from modern deep neural networks to the algorithms an actual brain is using. And, ultimately, these will move from being philosophical issues to being practical ones. None of that follows. I made this quite clear, that this is why your complaints are irrelevant or applicable only to the most narrow economic situations. Does a plane not travel faster than a bird because it uses many more joules to do so? We can use far more resources than we do, and all of the consequences of AI can still follow. (This also holds for the silly arguments that computational complexity somehow means that AI can't work or can't be a threat: https://www.gwern.net/Complexity-vs-AI https://www.gwern.net/Complexity-vs-AI ) Everything about superintelligence, AI risk, new breakthroughs, all of that can still follow even if an AI uses 10, 1000, or 10,000x more joules to accomplish a task. Humans themselves use astronomical amounts of energy compared to, say, insects; does this mean humans are not intelligent, and insects should be questioning whether humans are, on a philosophical level, as powerful intelligences as insects are, since we need so many additional calories to solve basic vision and navigation tasks? Who cares if we are being eaten by gray goo and you go 'ah, well, it requires several orders of magnitude and therefore isn't really real AI, this gray goo would be more impressive if it solved stuff within human-level time or energetic constraints'?

gwern 7y ago

That's too bad, because your comments there are really bad. You argue solely by assertion and personal feeling, and then wonder why no one agrees with you. Why does it matter in the least that you are not 'impressed' or feel using more than 1 server 'waters it down'? Why does it matter how much IBM Watson uses? AlphaGo uses more electricity than any human; does it not win at Go? If a NN uses more electricity than 150 watts to classify images at superhuman rates, why does this matter? Is there some law of physics I am unaware of which stipulates that no computer is allowed to use more than 150 watts, or that we are unable to build systems which have more than 1 computer involved? Is there some theorem in computer science which proves that AI isn't real AI unless it solves problems in exactly the same way as a human brain at exactly the same efficiency, never mind how well it solves problems or how much superior to humans it may be otherwise? ('Jet planes, you see, do not actually provide "artificial flight" because they burn more joules than a bird would.') Is there some reason to think that AI can never be misused or cause problems if it happens to use somewhat more electricity? Have deep learning techniques not continued to scale over the 8 years since you posted that? IBM Watson failed; did it have anything to do with using 15TB RAM; would Watson now be a success and closer to 'real AI' if they had hand-optimized its routines for memory efficiency to do the same thing but only needing 1.5TB RAM?

mlthoughts2018 7y ago
> “AlphaGo uses more electricity than any human; does it not win at Go?” I know you have a prolific reputation, and I mean this sincerely, but did you actually read the link? The point was that with power constraints, the class of possible algorithms is restricted heavily, requiring development of an understanding closer to what algorithms a person must use in their mind. If you present me a Go-based Turing test, and I’m pretty fooled by AlphaGo, except I get blasted by the hot air venting out from a big cooling system, then it probably reveals to me that it’s failed the test (at least until some far future where there are life forms that regularly vent computer-levels of hot air from cooling systems). This effect is much more pronounced with a larger system like Watson, hence the article. > “If a NN uses more electricity than 150 watts to classify images at superhuman rates, why does this matter?” It matters because pure computing achievements are not isolated from their resource constraints. If a NN can generate flawless natural language responses, but it takes 1000 years per word, we would not be interested and certainly would not see it as valuable progress towards artificial intelligence (likely not even strong AI, but surely not tool AI, there just aren’t a lot of ways to demonstrate a conceptual breakthrough so wasteful of resources). Intelligence is (partly) generating language within human-level time constraints. Why not also human level RAM, physical volume, material density, etc. Solving it within these constraints is much more valuable than solving it without them. Likewise with power. Maybe using 100x power is an OK tradeoff for pure tool usage (heck, I work professionally in computer vision and NLP and I make this concession to use machine learning models running on computers to solve my work daily). But that doesn’t mean it’s an ok trade-off for the philosophical side of the questions, like approximating a Turing test or understanding how far we must go from modern deep neural networks to the algorithms an actual brain is using. And, ultimately, these will move from being philosophical issues to being practical ones. > “Is there some reason to think that AI can never be misused or cause problems if it happens to use somewhat more electricity?” Huh? Where did I ever say anything like this? I mean, humans misuse intelligence and cause problems on less power. It sounds like you’re trying to propagate your feelings about the “green” aspect of the post into what I’ve said, but if you’ll notice, I did not say anything about that, and don’t particularly care about this OP article’s take on “green” AI power consumption, it’s not related at all to what I’m talking about. (Plus they are more focused on power demands at training time, while I’m interested in power constraints, like runtime or space constraints, at inference time).
- gwern 7y ago
  > I know you have a prolific reputation, and I mean this sincerely, but did you actually read the link? I did, and all the comments, to see where you were mistreated so badly that you'd drag it up 8 years later, like the Bourbons, having forgotten nothing and learned nothing. > The point was that with power constraints, the class of possible algorithms is restricted heavily, requiring development of an understanding closer to what algorithms a person must use in their mind. That's not true at all: it is not true that intelligences must exactly imitate humans to be intelligent, and if it was, it would not be true either as a lower bound, upper bound, or average case that an algorithm using the same energy as a human must be isomorphic to the human algorithm. Consider deep learning. The more compute it uses, the better it gets, and the more human-like the results are, and this is true despite the fact that they typically use more power than a human brain even at the smaller scales where they are often among the worse-performing and still-power-hungry of methods, and power consumption goes up from there; by your logic, they would be getting even less human-like, if power consumption and algorithmic equivalency were so inextricably intertwined. But they aren't. 'The bitter lesson' is itself a disproof of your claim that power efficiency has anything to do with potential for future progress or human-like-ness of algorithms. > But that doesn’t mean it’s an ok trade-off for the philosophical side of the questions, like approximating a Turing test or understanding how far we must go from modern deep neural networks to the algorithms an actual brain is using. And, ultimately, these will move from being philosophical issues to being practical ones. None of that follows. I made this quite clear, that this is why your complaints are irrelevant or applicable only to the most narrow economic situations. Does a plane not travel faster than a bird because it uses many more joules to do so? We can use far more resources than we do, and all of the consequences of AI can still follow. (This also holds for the silly arguments that computational complexity somehow means that AI can't work or can't be a threat: https://www.gwern.net/Complexity-vs-AI https://www.gwern.net/Complexity-vs-AI ) Everything about superintelligence, AI risk, new breakthroughs, all of that can still follow even if an AI uses 10, 1000, or 10,000x more joules to accomplish a task. Humans themselves use astronomical amounts of energy compared to, say, insects; does this mean humans are not intelligent, and insects should be questioning whether humans are, on a philosophical level, as powerful intelligences as insects are, since we need so many additional calories to solve basic vision and navigation tasks? Who cares if we are being eaten by gray goo and you go 'ah, well, it requires several orders of magnitude and therefore isn't really real AI, this gray goo would be more impressive if it solved stuff within human-level time or energetic constraints'?