3 ms·

The real world has many trillions of parameters. Also, doing stuff in the real world is all about sensors. Progress on integrating millions of sensors into a r

The real world has many trillions of parameters. Also, doing stuff in the real world is all about sensors. Progress on integrating millions of sensors into a robot has been slow.

borissk 3y ago
Don't think a robot needs millions of sensors. 90% of the information that a human uses comes through the eyes - so just 2 sensors. Really in terms of doing most types of work all a robot would need is vision and hearing.
- klipt 3y ago
  Q: But if the robot has no nose, how will it smell? A: Awful!
- csomar 3y ago
  Humans have an insane number of sensors all working with each other. The eyes are not just 2 sensors and a blind person can still function fairly well though with limitations.
dontupvoteme 3y ago
>The real world has many trillions of parameters. But at any given time you're probably only going to require some millions or double digit billions, unless you're a nation state or something like that.
trashtester 3y ago
The problem is not integrating millions of sensors of the same type (like individual pixels in a camera), but rather to integrate 2-10 different types of sensors, like microphones, cameras, lidars and tactile/temperature sensors. Progress has been slow mosly because it's been hand-written, such as the sensor fusion in the F-35. While there are some attempts at using neural networks to do much of the integration (Like Tesla's self driving) they're mostly quite primitive, and lacks the kind of general understanding of the world that we're starting to se in LLM's. We have started building multimodal neural networks, though, such as GPT-4's ability to handle images, though. he next step is to add video, which is the mode that requires the greatest amount of compute, we will be able to integrate it with lower bandwidth sensors using the same techniques. That would be where general purpose models start to become good enough to be deployed in robots. Let's imagine your Tesla has the ability to reason about the world the way GPT-4 can (and with enough fine tuning that halicunations are gone), and that it can do that not only based on text, but also all the sensors of the car, then it might become able to handle situations outside its training set much better than today, possibly better than many humans. Once we're able to build multimodal models able to handle realtime video and s

The real world has many trillions of parameters. Also, doing stuff in the real world is all about sensors. Progress on integrating millions of sensors into a robot has been slow.

borissk 3y ago

Don't think a robot needs millions of sensors. 90% of the information that a human uses comes through the eyes - so just 2 sensors. Really in terms of doing most types of work all a robot would need is vision and hearing.

klipt 3y ago
Q: But if the robot has no nose, how will it smell? A: Awful!
csomar 3y ago
Humans have an insane number of sensors all working with each other. The eyes are not just 2 sensors and a blind person can still function fairly well though with limitations.

dontupvoteme 3y ago

>The real world has many trillions of parameters. But at any given time you're probably only going to require some millions or double digit billions, unless you're a nation state or something like that.

trashtester 3y ago

The problem is not integrating millions of sensors of the same type (like individual pixels in a camera), but rather to integrate 2-10 different types of sensors, like microphones, cameras, lidars and tactile/temperature sensors. Progress has been slow mosly because it's been hand-written, such as the sensor fusion in the F-35. While there are some attempts at using neural networks to do much of the integration (Like Tesla's self driving) they're mostly quite primitive, and lacks the kind of general understanding of the world that we're starting to se in LLM's. We have started building multimodal neural networks, though, such as GPT-4's ability to handle images, though. he next step is to add video, which is the mode that requires the greatest amount of compute, we will be able to integrate it with lower bandwidth sensors using the same techniques. That would be where general purpose models start to become good enough to be deployed in robots. Let's imagine your Tesla has the ability to reason about the world the way GPT-4 can (and with enough fine tuning that halicunations are gone), and that it can do that not only based on text, but also all the sensors of the car, then it might become able to handle situations outside its training set much better than today, possibly better than many humans. Once we're able to build multimodal models able to handle realtime video and s