6 ms·

I'm going to draw up some charts about hull displacement on ships from the dawn of time up until about 1950. Then we can have a really informed conversation abo

I'm going to draw up some charts about hull displacement on ships from the dawn of time up until about 1950. Then we can have a really informed conversation about the naval power of countries through the ages. I think we need to be ready for the implications of there one day being a battleship the size of the Pacific that will allow its owner to rule the world. Forgive the sarcasm, but I'm really put off by the aggressive weak-to-strong generalizations that are going on here. I'm also very excited about AI, but I don't understand how lines like, >> But at least within many current domains, more compute seems to lead predictably to better performance, and is often complementary to algorithmic advances. can be extrapolated to anything more than a fun conversation to have over drinks, or the plot for a bad sci-fi movie about AI (which, to be fair, are also quite prevalent in the current zeitgeist). We're definitely at a new tier of "kinds of problems computers can solve", but surely experience and history in this space should tell us that we need to expect massive, seemingly insurmountable plateaus before we see the next tier of growth, and that that next tier will be much more a matter of paradigm shift than of growth on a line. The systems on this graph all do different things in different ways. It's one thing to abstract over compute power via something like Moore's Law, or societal complexity via the Kardashev scale. But I think we need a much more nuanced set of metrics to provide any kind of insight in to the various AI techniques. Or an entirely different way of looking at 'intelligence'

dbelchamber 8y ago
I completely agree. Current AI is excellent (or at least super-human) at learning to do anything where the mechanics of the situation are clear and where the measurement of success is well defined. Beyond that, I'm not sure we've made any convincing strides towards anything truly general.
- deleted 8y ago
  [deleted]
- ddtaylor 8y ago
  I fiddled with an idea where I wrote unit tests and used them for a scoring function to train a model. Writing the number of tests and encoding the logic for a simple Linked List took orders of magnitude more code than coding the list itself.
  - pseudonom- 8y ago
    I suspect (based on stuff like https://arxiv.org/abs/1510.08419 https://arxiv.org/abs/1510.08419) that you'd have had more luck with property tests instead of unit tests.
    - ddtaylor 8y ago
      Thanks for the paper I will check it out.
- Veedrac 8y ago
  The mechanics of speech synthesis are clear? The measure of success of style transfer is well defined?
  - dbelchamber 8y ago
    I would say speech synthesis is governed by very clear mechanics. https://en.wikipedia.org/wiki/International_Phonetic_Alphabet https://en.wikipedia.org/wiki/International_Phonetic_Alphabe... As for style transfer, that is a very specific skill of making the patterns of one style map to the patterns of another. I am not particularly well versed with art, but that process seems well defined to me. Perhaps your issue is with my more generalized definition of "clear" and "well-defined". I meant to use these terms to distinguish between autonomous driving and being a successful human. I really don't think there is anywhere close to a consensus on the latter. To the extent that there is, then yes, AI should be able to do it.
    - Veedrac 8y ago
      The IPA is nowhere close to sufficient for realistic speech synthesis, and style transfer is not just copy and paste. By the same token writing poetry is just "putting words into grammatical constructions that have certain patterns" or mathematics research is just "a form of proof search". Of course we don't have human-level AI right now, but if that's the only thing you're claiming it's pretty vacuous.
    - dnautics 8y ago
      I would say speech synthesis is governed by clear mechanics - and it's not the IPA, it's that the output comes out as a waveform, which has a structure that informs the algorithm. Note that we have great raster-based deep visual effects, but vector is... not there yet (not saying it won't be) - vector is less structured than raster, so the choice of algorithm is less obvious. As for well-defined criteria, I don't think that's really quite the right standard, I think the correct standard is that there is a way of metrizing success on a well-ordered set (like the [0,1] interval), even if it's noisy.
- wyattpeak 8y ago
  While I sort of intuitively agree with you, I think there's a certain amount of circularity in the argument - any problem will seem to have well defined metrics once you've built and studied a machine which can solve it. In the sixties, chess was thought to be a problem that required intelligence since it couldn't be brute-forced. We now have machines which can play it well, without brute-forcing, and yet it's seen as entirely procedural.
  - goatlover 8y ago
    AI people thought chess was a problem that required intelligence. Critics back in the 60s such as Dreyfus probably didn't view chess as the hallmark of intelligence.
  - erikpukinskis 8y ago
    AGI is the hypothesis that someday the number viable human cons with well-defined metrics will dwarf the ones without.
- rimliu 8y ago
  https://en.wikipedia.org/wiki/Moravec%27s_paradox https://en.wikipedia.org/wiki/Moravec%27s_paradox
deleted 8y ago
[deleted]
blixt 8y ago
I'm not sure I'm reading the same thing you're reading. Yes, OpenAI has a mission to work on ways to make AI safer (that is, prevent AI from completing goals in unexpected, "hacky" ways) as it becomes more capable. But I didn't find much of that in this article. What I read from this article is that we're currently in a trend where the more compute you throw at machine learning, the better solutions you end up with. Nothing about general AI, it's just that you can train deeper, more complex neural networks that can handle broader and more complex versions of the problems they're designed to handle. This can have implications down the line which still have nothing to do about AI becoming smarter (in the general intelligence kind of way) than it is today. If all input/output problems out there have very deep neural networks behind them, and those neural networks are constantly training simultaneously, and that has a positive economic output, we'll see tremendous amounts of FLOPS that pales cryptocurrency miners by comparison. Just as an example, depending on how cloud solutions keep up, maybe startups on the cloud won't be so competitive anymore. It's an interesting trend to point out and keep track of. And yes, OpenAI's involvement will probably be to learn about how this can lead to unsafe use of AI, but again that's not what I saw the topic of this article being.
- stochastic_monk 8y ago
  I think the best analogy is that ultrascale deep learning (if it’s not a term yet, it soon will be) is that of a particle accelerator: you throw as much firepower as possible at a task and get continued benefit.
  - e_modad 8y ago
    Hyperscale is a term already right? I don’t know a lot about HPC, but I think it’s already part of the nomenclature.
nunya213 8y ago
But you can easily calculate why its impossible to build a battleship beyond a certain size and understand why that trend is silly. I challenge you to do something similar with compute and AI. Not that I necessarily disagree with your thesis of better computer != better AI.
- daveguy 8y ago
  I'd like to see those easy calculations of why it's impossible to build a battleship beyond a certain size. I mean, if you put guns instead of planes on a modern aircraft carrier you'd have a battleship that's a hell of a lot bigger than any battleships in WWII. The hull size argument is apt. I think it's a lot more obvious that throwing more compute power at a neural networks isn't going to eventually make a general intelligence. At the very least there will need to be significant breakthroughs in architecture design that will be at least as paradigm shifty as deep learning.
  - entropie 8y ago
    > I mean, if you put guns instead of planes on a modern aircraft carrier you'd have a battleship that's a hell of a lot bigger than any battleships in WWII. This is not true. The biggest battleship bought was the yamato > Length: > 256 m (839 ft 11 in) (waterline) > 263 m (862 ft 10 in) (overall) The biggest CV (aircraft carrier) right now should be the Nimitz class with >Length: > 317m which seems to emphasize your statement, but, battleships are not versatile and battleship development pretty much stopped after WWII. The germans had propsals of the H-class battleship, with a total length of the H-44 of 345m.
    - daveguy 8y ago
      I wasn't saying that battleships are a good strategic choice to develop, just that you can't "easily calculate why its impossible to build a battleship beyond a certain size". It is possible, it's just not strategically advantageous. Just like throwing more compute at a neutral network isn't going to make general AI. The diminishing returns comes with how brittle their learning and representations are.
CoolGuySteve 8y ago
But isn't that kind of what happened with the Nimitz class carrier and the range of it's aircraft? One country doninating all other militaries via their navy?
- dantheman0207 8y ago
  It is what happened, but it took a paradigm shift (aircraft carriers) not an increase in computational (fire-)power. The Japanese actually invested heavily in the biggest battleship in the world. It was soundly defeated at sea by an inferior force with an aircraft carrier at Leyte Gulf during World War II.
  - jaipilot747 8y ago
    https://en.m.wikipedia.org/wiki/Japanese_battleship_Yamato https://en.m.wikipedia.org/wiki/Japanese_battleship_Yamato
- sgt101 8y ago
  I thought that what happened was that the USA was the only major industrialised power to emerge from WW2 with a functional government and economy, and consequently enjoyed 20+ years of unprecedented economic dominance. Subsequently it built whatever military assets it damn well pleased. I would argue that if you want to identify the mechanism of military dominance that the US has used to assert itself in the world you should look to Trident and the Ohio class.
efangs 8y ago
This plot essentially ignores all other computational science. People in HPC have been operating at these scales for awhile, and yet don't make claims about their field taking over the world.
red75prime 8y ago
> next tier will be much more a matter of paradigm shift than of growth on a line. There are a limited number of paradigm shifts before AIs get "general" in their name. So "it already happened before" argument has a natural end. What makes you think it is still applicable? Some organizations have access to a computational power reaching into the realm of a computational power of a human brain. We see commercial applications of systems which aren't programmed by hand like expert systems and which don't fail miserably like speech recognition engines of yore. How often do you hear about the curse of dimensionality today? Some things have changed from then to now, the question is are they changed enough.
- erikpukinskis 8y ago
  > Some organizations have access to a computational power reaching into the realm of a computational power of a human brain From a physics standpoint it’s not equivalent power actually, it’s equivalent energy. Power is work per unit time. Work is energy expended which causes displacement. These “brain-equivalent” computers can’t do nearly the work of a human brain in the amount of time. They use about the same amount of raw computational energy, but they don’t make nearly the same amount of structures waves in the information spaces around them. Their output can only even be seen in the absolute quietest of environments. They often run for long periods of time with no obviously informative output. Human minds tuned for it are essentially incapable of not producing a constant stream of novel and disruptive insights (work) which is how you get a large computational power given a roughly constant computational energy.

dbelchamber 8y ago

I completely agree. Current AI is excellent (or at least super-human) at learning to do anything where the mechanics of the situation are clear and where the measurement of success is well defined. Beyond that, I'm not sure we've made any convincing strides towards anything truly general.

deleted 8y ago
[deleted]
ddtaylor 8y ago
I fiddled with an idea where I wrote unit tests and used them for a scoring function to train a model. Writing the number of tests and encoding the logic for a simple Linked List took orders of magnitude more code than coding the list itself.
- pseudonom- 8y ago
  I suspect (based on stuff like https://arxiv.org/abs/1510.08419 https://arxiv.org/abs/1510.08419) that you'd have had more luck with property tests instead of unit tests.
  - ddtaylor 8y ago
    Thanks for the paper I will check it out.
Veedrac 8y ago
The mechanics of speech synthesis are clear? The measure of success of style transfer is well defined?
- dbelchamber 8y ago
  I would say speech synthesis is governed by very clear mechanics. https://en.wikipedia.org/wiki/International_Phonetic_Alphabet https://en.wikipedia.org/wiki/International_Phonetic_Alphabe... As for style transfer, that is a very specific skill of making the patterns of one style map to the patterns of another. I am not particularly well versed with art, but that process seems well defined to me. Perhaps your issue is with my more generalized definition of "clear" and "well-defined". I meant to use these terms to distinguish between autonomous driving and being a successful human. I really don't think there is anywhere close to a consensus on the latter. To the extent that there is, then yes, AI should be able to do it.
  - Veedrac 8y ago
    The IPA is nowhere close to sufficient for realistic speech synthesis, and style transfer is not just copy and paste. By the same token writing poetry is just "putting words into grammatical constructions that have certain patterns" or mathematics research is just "a form of proof search". Of course we don't have human-level AI right now, but if that's the only thing you're claiming it's pretty vacuous.
  - dnautics 8y ago
    I would say speech synthesis is governed by clear mechanics - and it's not the IPA, it's that the output comes out as a waveform, which has a structure that informs the algorithm. Note that we have great raster-based deep visual effects, but vector is... not there yet (not saying it won't be) - vector is less structured than raster, so the choice of algorithm is less obvious. As for well-defined criteria, I don't think that's really quite the right standard, I think the correct standard is that there is a way of metrizing success on a well-ordered set (like the [0,1] interval), even if it's noisy.
wyattpeak 8y ago
While I sort of intuitively agree with you, I think there's a certain amount of circularity in the argument - any problem will seem to have well defined metrics once you've built and studied a machine which can solve it. In the sixties, chess was thought to be a problem that required intelligence since it couldn't be brute-forced. We now have machines which can play it well, without brute-forcing, and yet it's seen as entirely procedural.
- goatlover 8y ago
  AI people thought chess was a problem that required intelligence. Critics back in the 60s such as Dreyfus probably didn't view chess as the hallmark of intelligence.
- erikpukinskis 8y ago
  AGI is the hypothesis that someday the number viable human cons with well-defined metrics will dwarf the ones without.
rimliu 8y ago
https://en.wikipedia.org/wiki/Moravec%27s_paradox https://en.wikipedia.org/wiki/Moravec%27s_paradox

deleted 8y ago

[deleted]

blixt 8y ago

I'm not sure I'm reading the same thing you're reading. Yes, OpenAI has a mission to work on ways to make AI safer (that is, prevent AI from completing goals in unexpected, "hacky" ways) as it becomes more capable. But I didn't find much of that in this article. What I read from this article is that we're currently in a trend where the more compute you throw at machine learning, the better solutions you end up with. Nothing about general AI, it's just that you can train deeper, more complex neural networks that can handle broader and more complex versions of the problems they're designed to handle. This can have implications down the line which still have nothing to do about AI becoming smarter (in the general intelligence kind of way) than it is today. If all input/output problems out there have very deep neural networks behind them, and those neural networks are constantly training simultaneously, and that has a positive economic output, we'll see tremendous amounts of FLOPS that pales cryptocurrency miners by comparison. Just as an example, depending on how cloud solutions keep up, maybe startups on the cloud won't be so competitive anymore. It's an interesting trend to point out and keep track of. And yes, OpenAI's involvement will probably be to learn about how this can lead to unsafe use of AI, but again that's not what I saw the topic of this article being.

stochastic_monk 8y ago
I think the best analogy is that ultrascale deep learning (if it’s not a term yet, it soon will be) is that of a particle accelerator: you throw as much firepower as possible at a task and get continued benefit.
- e_modad 8y ago
  Hyperscale is a term already right? I don’t know a lot about HPC, but I think it’s already part of the nomenclature.

nunya213 8y ago

But you can easily calculate why its impossible to build a battleship beyond a certain size and understand why that trend is silly. I challenge you to do something similar with compute and AI. Not that I necessarily disagree with your thesis of better computer != better AI.

daveguy 8y ago
I'd like to see those easy calculations of why it's impossible to build a battleship beyond a certain size. I mean, if you put guns instead of planes on a modern aircraft carrier you'd have a battleship that's a hell of a lot bigger than any battleships in WWII. The hull size argument is apt. I think it's a lot more obvious that throwing more compute power at a neural networks isn't going to eventually make a general intelligence. At the very least there will need to be significant breakthroughs in architecture design that will be at least as paradigm shifty as deep learning.
- entropie 8y ago
  > I mean, if you put guns instead of planes on a modern aircraft carrier you'd have a battleship that's a hell of a lot bigger than any battleships in WWII. This is not true. The biggest battleship bought was the yamato > Length: > 256 m (839 ft 11 in) (waterline) > 263 m (862 ft 10 in) (overall) The biggest CV (aircraft carrier) right now should be the Nimitz class with >Length: > 317m which seems to emphasize your statement, but, battleships are not versatile and battleship development pretty much stopped after WWII. The germans had propsals of the H-class battleship, with a total length of the H-44 of 345m.
  - daveguy 8y ago
    I wasn't saying that battleships are a good strategic choice to develop, just that you can't "easily calculate why its impossible to build a battleship beyond a certain size". It is possible, it's just not strategically advantageous. Just like throwing more compute at a neutral network isn't going to make general AI. The diminishing returns comes with how brittle their learning and representations are.

CoolGuySteve 8y ago

But isn't that kind of what happened with the Nimitz class carrier and the range of it's aircraft? One country doninating all other militaries via their navy?

dantheman0207 8y ago
It is what happened, but it took a paradigm shift (aircraft carriers) not an increase in computational (fire-)power. The Japanese actually invested heavily in the biggest battleship in the world. It was soundly defeated at sea by an inferior force with an aircraft carrier at Leyte Gulf during World War II.
- jaipilot747 8y ago
  https://en.m.wikipedia.org/wiki/Japanese_battleship_Yamato https://en.m.wikipedia.org/wiki/Japanese_battleship_Yamato
sgt101 8y ago
I thought that what happened was that the USA was the only major industrialised power to emerge from WW2 with a functional government and economy, and consequently enjoyed 20+ years of unprecedented economic dominance. Subsequently it built whatever military assets it damn well pleased. I would argue that if you want to identify the mechanism of military dominance that the US has used to assert itself in the world you should look to Trident and the Ohio class.

efangs 8y ago

This plot essentially ignores all other computational science. People in HPC have been operating at these scales for awhile, and yet don't make claims about their field taking over the world.

red75prime 8y ago

> next tier will be much more a matter of paradigm shift than of growth on a line. There are a limited number of paradigm shifts before AIs get "general" in their name. So "it already happened before" argument has a natural end. What makes you think it is still applicable? Some organizations have access to a computational power reaching into the realm of a computational power of a human brain. We see commercial applications of systems which aren't programmed by hand like expert systems and which don't fail miserably like speech recognition engines of yore. How often do you hear about the curse of dimensionality today? Some things have changed from then to now, the question is are they changed enough.

erikpukinskis 8y ago
> Some organizations have access to a computational power reaching into the realm of a computational power of a human brain From a physics standpoint it’s not equivalent power actually, it’s equivalent energy. Power is work per unit time. Work is energy expended which causes displacement. These “brain-equivalent” computers can’t do nearly the work of a human brain in the amount of time. They use about the same amount of raw computational energy, but they don’t make nearly the same amount of structures waves in the information spaces around them. Their output can only even be seen in the absolute quietest of environments. They often run for long periods of time with no obviously informative output. Human minds tuned for it are essentially incapable of not producing a constant stream of novel and disruptive insights (work) which is how you get a large computational power given a roughly constant computational energy.