3 ms·

The problem with all the very impressive videos on that page is that we have no idea how many attempts were made before the robot could successfully, e.g. "put

The problem with all the very impressive videos on that page is that we have no idea how many attempts were made before the robot could successfully, e.g. "put strawberry into the correct bowl". In that task there's four bowls, so a random choice of bowl would be correct 25% of the time. How many times did the robot put the strawberry e.g. in the bowl of apples? And that's assuming "the correct bowl" is the one with the strawberries (which is a big assumption- the strawberry should go with the others of its kind: says who? How often can the robot put the strawberry in the bowl with the apples if that's what we want it to do?). Plotted results show around 50% average performance on "unseen" tasks, environments, objects etc, which sounds a lot like success follows some kind of random distribution. That's not a great way to engender trust in the "emergent" abilities of a robotic system to generalise to unseen tasks etc. Blame bad statistics if you get a strawberry in the eye, or a banana in the ear.

smokel 2y ago
For better or for worse, it is probably time to get accustomed to this. Early AI operated with symbols, and it has been clear for decades that that was a dead-end. Contemporary AI is stochastic, and it works a lot better for many applications than previous attempts, even considering the inherent uncertainty and errors. Note that at the level of quantum physics one might not be able to trust CPU instructions to be faultless. It is all about getting the right error margins.
- YeGoblynQueenne 2y ago
  Accustomed- to tech demos that never lead to real-world capabilities? That's what I'm pointing out in my comment. Btw, computers are symbol manipulation machines and in general that's how we understand computation: manipulating symbols according to a set of rules; like Turing machines. Stochastic algorithms also ultimately work that way, and they will continue to until we can run all the modern AI stuff on, say, analog computers.
imtringued 2y ago
My biggest concern isn't with the reliability, but more the fact that it has so far failed to accurately place the cans in every video. You have to remember that "unseen" means out of distribution performance, which used to be absolutely abhorrent and is now barely bearable.

smokel 2y ago

For better or for worse, it is probably time to get accustomed to this. Early AI operated with symbols, and it has been clear for decades that that was a dead-end. Contemporary AI is stochastic, and it works a lot better for many applications than previous attempts, even considering the inherent uncertainty and errors. Note that at the level of quantum physics one might not be able to trust CPU instructions to be faultless. It is all about getting the right error margins.

YeGoblynQueenne 2y ago
Accustomed- to tech demos that never lead to real-world capabilities? That's what I'm pointing out in my comment. Btw, computers are symbol manipulation machines and in general that's how we understand computation: manipulating symbols according to a set of rules; like Turing machines. Stochastic algorithms also ultimately work that way, and they will continue to until we can run all the modern AI stuff on, say, analog computers.

imtringued 2y ago

My biggest concern isn't with the reliability, but more the fact that it has so far failed to accurately place the cans in every video. You have to remember that "unseen" means out of distribution performance, which used to be absolutely abhorrent and is now barely bearable.