4 ms·

> Cutting-edge research does feel like it's reaching a plateau It's really not. The second half of last year alone had MuZero and Megatron-LM, to name just a c

> Cutting-edge research does feel like it's reaching a plateau It's really not. The second half of last year alone had MuZero and Megatron-LM, to name just a couple that most scream to me that we are actually progressing towards AGI. You say ‘When the state-of-the-art is 97% on a task’, but solved tasks are the least interesting tasks.

tomrod 7y ago
> solved tasks are the least interesting tasks. Yeah, but they also tend to be pretty profitable.
antpls 7y ago
Also, Google released the Reformer just 2 days ago, and they claim it can ingest orders of magnitude more data than the Transformer. https://ai.googleblog.com/2020/01/reformer-efficient-transformer.html?m=1 https://ai.googleblog.com/2020/01/reformer-efficient-transfo... TPUv3 is estimated to be 12 or 16nm process node, so the performance of TPU's next versions could still double over the next years (if needed). At that pace of model research and hardware improvement, I would say we are still in the AI spring.
Piskvorrr 7y ago
Do we have any other indicators that it's actually progressing somewhere, besides screaming? Even such a triviality as "how do we recognize that we got there"? The research is still in very early phases, IMNSHO: impressive practical applications appear, but they're side effects of what appears as random flailing: "build it bigger, see if it helps. Build it sideways, see of it helps. Build it at full moon, see if it helps". That suggests that the applications are the low-hanging fruit, with far more interesting results still to be discovered.
- Veedrac 7y ago
  MuZero is in some sense the proto-holy grail, in that it implements learning and planning into unstructured tasks over purely internal models. While there is an obvious chasm between it and the end point, this is still something that has only recently become more than an abstract goal, at least to any effect. Being able to perform planning over ‘simple’ domains like Atari games and Go (and not even in the same trained model!) might not seem very comparable to the real thing, but evolutionary history spent the bulk of its time building up the basics—most animals fail most cognitive tasks—so I don't think this is indicative of the progress being misguided, especially given networks-on-GPUs is literally a 10 year old field. I think MuZero is a clear example of building by principles over random flailing. I get why there does also seem to be the latter, but it's certainly not the whole of it, and anyhow it worked for evolution ;P.
  - Piskvorrr 7y ago
    Sure, that looks promising. I'm not holding my breath for The Holy Grail: in an environment where it's been just-around-the-corner for as long as the field exists, there's always another unexpected corner;)

tomrod 7y ago

> solved tasks are the least interesting tasks. Yeah, but they also tend to be pretty profitable.

antpls 7y ago

Also, Google released the Reformer just 2 days ago, and they claim it can ingest orders of magnitude more data than the Transformer. https://ai.googleblog.com/2020/01/reformer-efficient-transformer.html?m=1 https://ai.googleblog.com/2020/01/reformer-efficient-transfo... TPUv3 is estimated to be 12 or 16nm process node, so the performance of TPU's next versions could still double over the next years (if needed). At that pace of model research and hardware improvement, I would say we are still in the AI spring.

Piskvorrr 7y ago

Do we have any other indicators that it's actually progressing somewhere, besides screaming? Even such a triviality as "how do we recognize that we got there"? The research is still in very early phases, IMNSHO: impressive practical applications appear, but they're side effects of what appears as random flailing: "build it bigger, see if it helps. Build it sideways, see of it helps. Build it at full moon, see if it helps". That suggests that the applications are the low-hanging fruit, with far more interesting results still to be discovered.

Veedrac 7y ago
MuZero is in some sense the proto-holy grail, in that it implements learning and planning into unstructured tasks over purely internal models. While there is an obvious chasm between it and the end point, this is still something that has only recently become more than an abstract goal, at least to any effect. Being able to perform planning over ‘simple’ domains like Atari games and Go (and not even in the same trained model!) might not seem very comparable to the real thing, but evolutionary history spent the bulk of its time building up the basics—most animals fail most cognitive tasks—so I don't think this is indicative of the progress being misguided, especially given networks-on-GPUs is literally a 10 year old field. I think MuZero is a clear example of building by principles over random flailing. I get why there does also seem to be the latter, but it's certainly not the whole of it, and anyhow it worked for evolution ;P.
- Piskvorrr 7y ago
  Sure, that looks promising. I'm not holding my breath for The Holy Grail: in an environment where it's been just-around-the-corner for as long as the field exists, there's always another unexpected corner;)