3 ms·

I'd like to see someone make the argument that current models aren't just combining a number of "tricks", similar to a trained animal. My dog can "sit", "stay"

I'd like to see someone make the argument that current models aren't just combining a number of "tricks", similar to a trained animal. My dog can "sit", "stay" and "beg", all using the same model (its brain). Is the dog generally intelligent?

visarga 4y ago
How good is your dog at Atari games, stacking cubes and image captioning? You can actually measure the effect of generality by how fast it learns new tasks. The paper is full of tables and graphs showing this ability. It's just a small model, 170x smaller than GPT-3, has lots of room to grow. But for the first time we have a game playing agent that knows what "Atari" and "game" mean, and can probably comment on the side of the livestream. AlphaGo only knew the world of the Go board. This agent knows what is outside the box.
- hooande 4y ago
  Playing Atari is cool, but it's just another "trick". Training a computer to do progressively more difficult tasks doesn't seem much more impressive than training an animal to do so. I see no evidence in the paper that it can learn arbitrary tasks on the fly. It's very impressive, though.
  - visarga 4y ago
    > I see no evidence in the paper that it can learn arbitrary tasks on the fly. Neither can we do that. It takes years to become and expert in any field, we are not learning on the fly like Neo. That's when there is extensive training available, for research - it takes thousands of experts to crack one small step ahead. No one can do it alone, it would be too much to expect it from a lonely zero shot language model. On the other hand the transformer architecture seems to be capable of solving all the AI tasks, it can learn "on the fly" as soon as you provide the training data or a simulator. This particular paper trains over 600 tasks at once, in the same model.

3 ms·

I'd like to see someone make the argument that current models aren't just combining a number of "tricks", similar to a trained animal. My dog can "sit", "stay"

by hooande 4y ago

visarga 4y ago
How good is your dog at Atari games, stacking cubes and image captioning? You can actually measure the effect of generality by how fast it learns new tasks. The paper is full of tables and graphs showing this ability. It's just a small model, 170x smaller than GPT-3, has lots of room to grow. But for the first time we have a game playing agent that knows what "Atari" and "game" mean, and can probably comment on the side of the livestream. AlphaGo only knew the world of the Go board. This agent knows what is outside the box.
- hooande 4y ago
  Playing Atari is cool, but it's just another "trick". Training a computer to do progressively more difficult tasks doesn't seem much more impressive than training an animal to do so. I see no evidence in the paper that it can learn arbitrary tasks on the fly. It's very impressive, though.
  - visarga 4y ago
    > I see no evidence in the paper that it can learn arbitrary tasks on the fly. Neither can we do that. It takes years to become and expert in any field, we are not learning on the fly like Neo. That's when there is extensive training available, for research - it takes thousands of experts to crack one small step ahead. No one can do it alone, it would be too much to expect it from a lonely zero shot language model. On the other hand the transformer architecture seems to be capable of solving all the AI tasks, it can learn "on the fly" as soon as you provide the training data or a simulator. This particular paper trains over 600 tasks at once, in the same model.