4 ms·

They had camera-only tech employing multiple 4k cameras running at over 2000fps. Not your grandma's 480p/25fps webcam many car manufacturers use as parking came

by storus 1y ago

They had camera-only tech employing multiple 4k cameras running at over 2000fps. Not your grandma's 480p/25fps webcam many car manufacturers use as parking cameras. 2000fps gives you enormous safety margin even in case of individual frame misdetection. The long-tail issues they hit are present on LiDAR vehicles as well but LiDAR is much slower, more difficult to process and sensor fusion adds its own errors.

bayindirh 1y ago
2000 FPS is impressive. Not detecting overturned semis, road debris, and swerving to road dividers is even more impressive with that tech. Where a relatively simple radar can prevent without running a slow-motion camera rig and a wannabe supercomputing cluster on the car. To be frank, I'm not against 2000 FPS cameras, but I can't come into terms with not adding a simple radar to detect something unknown is dangerously close and the land missile needs to stop.
4gotunameagain 1y ago
Assuming of course that the relevant computer vision components can run at 2000 fps as well.. I highly doubt it.
- bayindirh 1y ago
  Running at 2000FPS in low light (and getting meaningful data at that sensor size) is also impossible to begin with. Even if you can do constant 60, you're in good shape. 2000 can be good for doing multiexposure and maybe detecting fine movement, but assuming that everything running 2000FPS (and processing 16000 frames/sec) is not a simple thing, esp, if you're running in an uncontrolled and chaotic environment.
  - piva00 1y ago
    I was about to comment the same, 2k FPS means a maximum shutter speed of 1/2000, you need a lot of light to capture an image this quickly, in low light conditions it's simply impossible to capture enough light even if using very high end optics and sensors.
    - storus 1y ago
      I don't know the specifics, maybe they are timing individual cameras in a way they achieve 2000fps with a crisp image in each camera and merging them together. Or maybe they are using some MIT tech that was able to capture super low light conditions.
      - bayindirh 1y ago
        Being able to capture in super low light conditions is dependent on two things. 1. Your sensor's noise floor, 2. The number of photons you can get per unit time. First one is dependent on the manufacturing process, and the second one is dependent on your sensor size. Currently, the leading sensor manufacturers (namely Sony Semiconductor and Canon) are doing very low noise sensors. However, to get both these low noise levels and convincing images needs full frame sensors, at least. APS-C can somewhat close, but it can't be there (because physics). Even in that case, you can't do 2000FPS and get meaningful images from every one of them. There's no way that a Tesla car cam sports full frame or APS-C sensors. So, it's physics.
        storus 1y ago
        AFAIK Sony and Canon are still using some ancient manufacturing process for sensors as chips have the priority and if Tesla has access to e.g. 5nm process for manufacturing sensors that would drastically expand possibilities. Also you bypassed the possibility of timing multiple sensors separately to achieve 2000fps.
        bayindirh 1y ago
        The reason sensor manufacturers use "seemingly ancient" (i.e. huge feature sizes) processes in their sensors is you really don't need a more advanced process like in processors. When you manufacture something which computes, power consumption and internal noise improvement is more drastic with improved manufacturing processes. When you are measuring something, you don't need or want too small pixels or features to begin with. So having a small gigapixel sensor just because your process allows creates more disadvantage over having a sensor same size with a lower resolution, from light capturing angle. So, low-light sensitivity and resolution is a trade-off. Back-illuminated sensors used by all contemporary cameras created this leap rather than reducing feature size via improved processes. You already pack the sensor as dense as possible (you don't want gaps or "smaller" pixels w/o increasing resolution either), and moving data/power plane away from pixels is the biggest contributor to noise in the sensor. See the link [0]. Top left image is full frame, top right is APS-C, bottom left is M4/3, and bottom right is full frame / high-res (60+MP) sensors. When you look at the images, smaller the sensor, worse the noise performance. When you compare full-size images of top left to bottom right, top left image is better in terms of noise. I selected RAW to surface "what sensor sees" The selected spot is the darkest point in that scene. You can select JPEG to see what in camera image processing does to these images. Shutter speed is around 1/40s and ISO is fixed at 12800 since it's the de-facto standard for night photography. > Also you bypassed the possibility of timing multiple sensors separately to achieve 2000fps. Working on an image which doesn't reflect real world is a bit dangerous, isn't it? [0]: https://www.dpreview.com/reviews/image-comparison?attr18=lowlight&attr13_0=sony_a7iv&attr13_1=fujifilm_xt5&attr13_2=omsystem_om5&attr13_3=sony_a7rv&attr15_0=raw&attr15_1=raw&attr15_2=raw&attr15_3=raw&attr16_0=12800&attr16_1=12800&attr16_2=12800&attr16_3=12800&attr126_2=1&attr126_3=1&normalization=full&widget=1&x=-0.12030701173953562&y=-0.9532479323599214 https://www.dpreview.com/reviews/image-comparison?attr18=low...
rich_sasha 1y ago
Why 2k FPS? I'm not being facetious; human eye sees, apparently, at around 25fps, which is why this is what TVs and cinemas used to use. At that rate, and 144kph, say, the car moves 1.4m between frames. Fine, so maybe you think this is too much. But 10x this still gives you 14cm between frames, at what is already speeding in most jurisdictions I know of. 2000 FPS seems to my untrained eye like a problem, not a feature.
- storus 1y ago
  Because you are processing a sequence of a fixed length in deep learning models and the more frames you have, the more accurate your FSD output is. Driving 1.4m between the frames with single-frame accuracy of 80% is quite risky and input correction quite discrete; 14cm is still risky for a proper trajectory planning. Now make it in millimeters and suddenly your trajectory is nearly perfect with only little noise.
- dns_snek 1y ago
  > human eye sees, apparently, at around 25fps What do you mean by "sees"? I'll bet you that you can't walk around wearing a VR headset running at 25 FPS for more than 30 seconds without violently emptying your stomach. Trying to watch a movie on a display that doesn't exhibit motion blur also makes me motion sick. Human brain doesn't see in terms of frames at all. There's a limit where an increase in FPS likely becomes imperceptible to most people but that limit is at least 10 times higher (from personal experience), likely more.
- Vilian 1y ago
  Humans don't see around 25fps, it depends but the maximum for a trained person or competitive player is 300hz
diabllicseagull 1y ago
as expected too heavy of a data stream to be true. resolution is lower than 4K while frame rates aren’t even in the ballpark of 2000 fps. https://www.blogordie.com/2023/09/hw4-tesla-new-self-driving-hardware/ https://www.blogordie.com/2023/09/hw4-tesla-new-self-driving...
- storus 1y ago
  I have this first-hand from the FSD team at Tesla from 5 years ago. Who knows where they are nowadays. You can believe whatever you like.
  - estearum 1y ago
    There's no way the FSD team overstated its capabilities, right?
    - storus 1y ago
      Everything is possible. They also might have used some creative metrics giving 2000+ fps. I don't know. Or they might have found some neat trick nobody thought about before.
croon 1y ago
That would be something like 371Gbps (some assumptions) raw data to process, per camera. I would assume a lot of shortcuts to get that down, but still an unreasonably huge scope to process in "real time" in a car.
ModernMech 1y ago
> The long-tail issues they hit are present on LiDAR vehicles as well but LiDAR is much slower, more difficult to process and sensor fusion adds its own errors. The long tail is long no matter what. Which is why the most robust solutions deploy sensors with orthogonal sensing modalities that can compliment one another. By relying on only one sensor type, Tesla has made it hard for their system overly brittle, which has resulted in avoidable deaths and destruction. > LiDAR is much slower, more difficult to process LiDAR in my experience is much easier to process, as the sensor stream is just an array of distances. Camera in my experience is much harder to process, as the sensor stream is an array of RGB values from which you have to infer distances. So by what metric are you alleging LiDAR is more difficult to process? > sensor fusion adds its own errors. You'll have to do a degree of sensor fusion across all the camera sensors anyway, so going camera-only don't absolve you of having to fuse sensor streams and come up with a belief. Sensor fusion in general tends to decrease overall system error as more sensors are added.
tim333 1y ago
Googling suggests the Tesla cameras do 36 fps https://x.com/nextbigfuture/status/1895692497898390021 https://x.com/nextbigfuture/status/1895692497898390021
Zigurd 1y ago
5mp 36fps. 1mp on older Teslas.

by storus 1y ago

bayindirh 1y ago

2000 FPS is impressive. Not detecting overturned semis, road debris, and swerving to road dividers is even more impressive with that tech. Where a relatively simple radar can prevent without running a slow-motion camera rig and a wannabe supercomputing cluster on the car. To be frank, I'm not against 2000 FPS cameras, but I can't come into terms with not adding a simple radar to detect something unknown is dangerously close and the land missile needs to stop.

4gotunameagain 1y ago

Assuming of course that the relevant computer vision components can run at 2000 fps as well.. I highly doubt it.

bayindirh 1y ago
Running at 2000FPS in low light (and getting meaningful data at that sensor size) is also impossible to begin with. Even if you can do constant 60, you're in good shape. 2000 can be good for doing multiexposure and maybe detecting fine movement, but assuming that everything running 2000FPS (and processing 16000 frames/sec) is not a simple thing, esp, if you're running in an uncontrolled and chaotic environment.
- piva00 1y ago
  I was about to comment the same, 2k FPS means a maximum shutter speed of 1/2000, you need a lot of light to capture an image this quickly, in low light conditions it's simply impossible to capture enough light even if using very high end optics and sensors.
  - storus 1y ago
    I don't know the specifics, maybe they are timing individual cameras in a way they achieve 2000fps with a crisp image in each camera and merging them together. Or maybe they are using some MIT tech that was able to capture super low light conditions.
    - bayindirh 1y ago
      Being able to capture in super low light conditions is dependent on two things. 1. Your sensor's noise floor, 2. The number of photons you can get per unit time. First one is dependent on the manufacturing process, and the second one is dependent on your sensor size. Currently, the leading sensor manufacturers (namely Sony Semiconductor and Canon) are doing very low noise sensors. However, to get both these low noise levels and convincing images needs full frame sensors, at least. APS-C can somewhat close, but it can't be there (because physics). Even in that case, you can't do 2000FPS and get meaningful images from every one of them. There's no way that a Tesla car cam sports full frame or APS-C sensors. So, it's physics.
      - storus 1y ago
        AFAIK Sony and Canon are still using some ancient manufacturing process for sensors as chips have the priority and if Tesla has access to e.g. 5nm process for manufacturing sensors that would drastically expand possibilities. Also you bypassed the possibility of timing multiple sensors separately to achieve 2000fps.
        bayindirh 1y ago
        The reason sensor manufacturers use "seemingly ancient" (i.e. huge feature sizes) processes in their sensors is you really don't need a more advanced process like in processors. When you manufacture something which computes, power consumption and internal noise improvement is more drastic with improved manufacturing processes. When you are measuring something, you don't need or want too small pixels or features to begin with. So having a small gigapixel sensor just because your process allows creates more disadvantage over having a sensor same size with a lower resolution, from light capturing angle. So, low-light sensitivity and resolution is a trade-off. Back-illuminated sensors used by all contemporary cameras created this leap rather than reducing feature size via improved processes. You already pack the sensor as dense as possible (you don't want gaps or "smaller" pixels w/o increasing resolution either), and moving data/power plane away from pixels is the biggest contributor to noise in the sensor. See the link [0]. Top left image is full frame, top right is APS-C, bottom left is M4/3, and bottom right is full frame / high-res (60+MP) sensors. When you look at the images, smaller the sensor, worse the noise performance. When you compare full-size images of top left to bottom right, top left image is better in terms of noise. I selected RAW to surface "what sensor sees" The selected spot is the darkest point in that scene. You can select JPEG to see what in camera image processing does to these images. Shutter speed is around 1/40s and ISO is fixed at 12800 since it's the de-facto standard for night photography. > Also you bypassed the possibility of timing multiple sensors separately to achieve 2000fps. Working on an image which doesn't reflect real world is a bit dangerous, isn't it? [0]: https://www.dpreview.com/reviews/image-comparison?attr18=lowlight&attr13_0=sony_a7iv&attr13_1=fujifilm_xt5&attr13_2=omsystem_om5&attr13_3=sony_a7rv&attr15_0=raw&attr15_1=raw&attr15_2=raw&attr15_3=raw&attr16_0=12800&attr16_1=12800&attr16_2=12800&attr16_3=12800&attr126_2=1&attr126_3=1&normalization=full&widget=1&x=-0.12030701173953562&y=-0.9532479323599214 https://www.dpreview.com/reviews/image-comparison?attr18=low...

rich_sasha 1y ago

Why 2k FPS? I'm not being facetious; human eye sees, apparently, at around 25fps, which is why this is what TVs and cinemas used to use. At that rate, and 144kph, say, the car moves 1.4m between frames. Fine, so maybe you think this is too much. But 10x this still gives you 14cm between frames, at what is already speeding in most jurisdictions I know of. 2000 FPS seems to my untrained eye like a problem, not a feature.

storus 1y ago
Because you are processing a sequence of a fixed length in deep learning models and the more frames you have, the more accurate your FSD output is. Driving 1.4m between the frames with single-frame accuracy of 80% is quite risky and input correction quite discrete; 14cm is still risky for a proper trajectory planning. Now make it in millimeters and suddenly your trajectory is nearly perfect with only little noise.
dns_snek 1y ago
> human eye sees, apparently, at around 25fps What do you mean by "sees"? I'll bet you that you can't walk around wearing a VR headset running at 25 FPS for more than 30 seconds without violently emptying your stomach. Trying to watch a movie on a display that doesn't exhibit motion blur also makes me motion sick. Human brain doesn't see in terms of frames at all. There's a limit where an increase in FPS likely becomes imperceptible to most people but that limit is at least 10 times higher (from personal experience), likely more.
Vilian 1y ago
Humans don't see around 25fps, it depends but the maximum for a trained person or competitive player is 300hz

diabllicseagull 1y ago

as expected too heavy of a data stream to be true. resolution is lower than 4K while frame rates aren’t even in the ballpark of 2000 fps. https://www.blogordie.com/2023/09/hw4-tesla-new-self-driving-hardware/ https://www.blogordie.com/2023/09/hw4-tesla-new-self-driving...

storus 1y ago
I have this first-hand from the FSD team at Tesla from 5 years ago. Who knows where they are nowadays. You can believe whatever you like.
- estearum 1y ago
  There's no way the FSD team overstated its capabilities, right?
  - storus 1y ago
    Everything is possible. They also might have used some creative metrics giving 2000+ fps. I don't know. Or they might have found some neat trick nobody thought about before.

croon 1y ago

That would be something like 371Gbps (some assumptions) raw data to process, per camera. I would assume a lot of shortcuts to get that down, but still an unreasonably huge scope to process in "real time" in a car.

ModernMech 1y ago

> The long-tail issues they hit are present on LiDAR vehicles as well but LiDAR is much slower, more difficult to process and sensor fusion adds its own errors. The long tail is long no matter what. Which is why the most robust solutions deploy sensors with orthogonal sensing modalities that can compliment one another. By relying on only one sensor type, Tesla has made it hard for their system overly brittle, which has resulted in avoidable deaths and destruction. > LiDAR is much slower, more difficult to process LiDAR in my experience is much easier to process, as the sensor stream is just an array of distances. Camera in my experience is much harder to process, as the sensor stream is an array of RGB values from which you have to infer distances. So by what metric are you alleging LiDAR is more difficult to process? > sensor fusion adds its own errors. You'll have to do a degree of sensor fusion across all the camera sensors anyway, so going camera-only don't absolve you of having to fuse sensor streams and come up with a belief. Sensor fusion in general tends to decrease overall system error as more sensors are added.

tim333 1y ago

Googling suggests the Tesla cameras do 36 fps https://x.com/nextbigfuture/status/1895692497898390021 https://x.com/nextbigfuture/status/1895692497898390021

Zigurd 1y ago

5mp 36fps. 1mp on older Teslas.