4 ms·
> The Waymo World Model can convert those kinds of videos, or any taken with a regular camera, into a multimodal simulation—showing how the Waymo Driver would s
by xnx 9mo ago
> The Waymo World Model can convert those kinds of videos, or any taken with a regular camera, into a multimodal simulation—showing how the Waymo Driver would see that exact scene.
Subtle brag that Waymo could drive in camera-only mode if they chose to. They've stated as much previously, but that doesn't seem widely known.
- bonsai_spool 9mo agoI think I'm misunderstanding - they're converting video into their representation which was bootstrapped with LIDAR, video and other sensors. I feel you're alluding to Tesla, but Tesla could never have this outcome since they never had a LIDAR phase. (edit - I'm referring to deployed Tesla vehicles, I don't know what their research fleet comprises, but other commenters explain that this fleet does collect LIDAR)
- yakz 9mo agoTesla does collect LIDAR data (people have seen them doing it, it's just not on all of the cars) and they do generate depth maps from sensor data, but from the examples I've seen it is much lower resolution than these Waymo examples.
- justapassenger 9mo agoTesla does it to map the areas to come up with high def maps for areas where their cars try to operate.
- vardump 9mo agoTesla uses lidar to train their models to generate depth data out of camera input. I don’t think they have any high definition maps.
- smallmancontrov 9mo agoThey can and they do. https://youtu.be/LFh9GAzHg1c?t=872 https://youtu.be/LFh9GAzHg1c?t=872 They've also built it into a full neural simulator. https://youtu.be/LFh9GAzHg1c?t=1063 https://youtu.be/LFh9GAzHg1c?t=1063 I think what we are seeing is that they both converged on the correct approach, one of them decided to talk about it, and it triggered disclosure all around since nobody wants to be seen as lagging.
- tfehring 9mo agoI watched that video around both timestamps and didn't see or hear any mention of LIDAR, only of video.
- smallmancontrov 9mo agoExactly: they convert video into a world model representation suitable for 3D exploration and simulation without using LIDAR (except perhaps for scale calibration).
- tfehring 9mo agoMy mistake - I misinterpreted your comment, but after re-reading more carefully, it's clear that the video confirms exactly what you said.
- IhateAI_3 9mo agotesla is not impressive, I would never put my child in one
- uejfiweun 9mo agoI've always wondered... if Lidar + Cameras is always making the right decision, you should theoretically be able to take the output of the Lidar + Cameras model and use it as training data for a Camera only model.
- olex 9mo agoThat's exactly what Tesla is doing with their validation vehicles, the ones with Lidar towers on top. They establish the "ground truth" from Lidar and use that to train and/or test the vision model. Presumably more "test", since they've most often been seen in Robotaxi service expansion areas shortly before fleet deployment.
- bob_theslob646 9mo agoIs that exactly true though? Can you give a reference for that?
- olex 9mo agoI don't have a specific source, no. I think it was mentioned in one of their presentation a few years back, that they use various techniques for "ground truth" for vision training, among those was time series (depth change over time should be continuous etc) and iirc also "external" sources for depth data, like LiDAR. And their validation cars equipped with LiDAR towers are definitely being seen everywhere they are rolling out their Robotaxi services.
- senordevnyc 9mo agoare definitely being seen everywhere they are rolling out their Robotaxi services So...nowhere?
- deleted 9mo ago[deleted]
- __alexs 9mo ago> you should theoretically be able to take the output of the Lidar + Cameras model and use it as training data for a Camera only model. Why should you be able to do that exactly? Human vision is frequently tricked by it's lack of depth data.
- shihab 9mo agoI think there are two steps here: converting video to sensor data input, and using that sensor data to drive. Only the second step will be handled by cars on road, first one is purely for training.
- ActorNightly 9mo agoThe purpose of lidar is to prove error correction when you need it most in terms of camera accuracy loss. Humans do this, just in the sense of depth perception with both eyes.
- dbt00 9mo ago(Always worth noting, human depth perception is not just based on stereoscopic vision, but also with focal distance, which is why so many people get simulator sickness from stereoscopic 3d VR)
- mikepurvis 9mo agoMy understanding is that contextual clues are a big part of it too. We see a the pitcher wind up and throw a baseball as us more than we stereoscopically track its progress from the mound to the plate. More subtly, a lot of depth information comes from how big we expect things to be, since everyday life is full of things we intuitively know the sizes of, frames of reference in the form of people, vehicles, furniture, etc . This is why the forced perspective of theme park castles is so effective— our brains want to see those upper windows as full sized, so we see the thing as 2-3x bigger than it actually is. And in the other direction, a lot of buildings in Las Vegas are further away than they look because hotels like the Bellagio have large black boxes on them that group a 2x2 block of the actual room windows.
- kevindamm 9mo agoActually the reason people experience vection in VR is not focal depth but the dissonance between what their eyes are telling them and what their inner ear and tactile senses are telling them. It's possible they get headaches from the focal length issues but that's different.
- wolrah 9mo ago> Always worth noting, human depth perception is not just based on stereoscopic vision, but also with focal distance Also subtle head and eye movements, which is something a lot of people like to ignore when discussing camera-based autonomy. Your eyes are always moving around which changes the perspective and gives a much better view of depth as we observe parallax effects. If you need a better view in a given direction you can turn or move your head. Fixed cameras mounted to a car's windshield can't do either of those things, so you need many more of them at higher resolutions to even come close to the amount of data the human eye can gather.
- mycall 9mo agoThat is still important for safety reasons in case someone uses a LiDAR jamming system to try to force you into an accident.
- Jyaif 9mo agoIf somebody wants to hurt you while you are traveling in a car, there are simpler ways.
- etrautmann 9mo agoIt’s way easier to “jam” a camera with bright light than a lidar, which uses both narrow band optical filters and pulsed signals with filters to detect that temporal sequence. If I were an adversary, going after cameras is way way easier.
- sroussey 9mo agoOh yeah, point a q-beam at a Tesla at night, lol. Blindness!
- dooglius 9mo agoThey may be trying to suggest that, that claim does not follow from the quoted statement.
- sschueller 9mo agoAutonomous cars need to be significantly better than humans to be fully accepted especially when an accident does happen. Hence limiting yourself to only cameras is futile.
- mavhc 9mo agoSurely as soon as they're safer than humans they should be deployed as fast as possible to save some of the 3000 people who are killed by human drivers every day
- cman1444 9mo agoOf course they should be, but that's not what will happen. Humans are not rational, so self-driving cars must be significantly safer than human drivers to avoid as much political pushback as possible.