15 ms·
The Waymo World Model
- xnx 9mo ago> The Waymo World Model can convert those kinds of videos, or any taken with a regular camera, into a multimodal simulation—showing how the Waymo Driver would see that exact scene. Subtle brag that Waymo could drive in camera-only mode if they chose to. They've stated as much previously, but that doesn't seem widely known.
- bonsai_spool 9mo agoI think I'm misunderstanding - they're converting video into their representation which was bootstrapped with LIDAR, video and other sensors. I feel you're alluding to Tesla, but Tesla could never have this outcome since they never had a LIDAR phase. (edit - I'm referring to deployed Tesla vehicles, I don't know what their research fleet comprises, but other commenters explain that this fleet does collect LIDAR)
- yakz 9mo agoTesla does collect LIDAR data (people have seen them doing it, it's just not on all of the cars) and they do generate depth maps from sensor data, but from the examples I've seen it is much lower resolution than these Waymo examples.
- justapassenger 9mo agoTesla does it to map the areas to come up with high def maps for areas where their cars try to operate.
- vardump 9mo agoTesla uses lidar to train their models to generate depth data out of camera input. I don’t think they have any high definition maps.
- smallmancontrov 9mo agoThey can and they do. https://youtu.be/LFh9GAzHg1c?t=872 https://youtu.be/LFh9GAzHg1c?t=872 They've also built it into a full neural simulator. https://youtu.be/LFh9GAzHg1c?t=1063 https://youtu.be/LFh9GAzHg1c?t=1063 I think what we are seeing is that they both converged on the correct approach, one of them decided to talk about it, and it triggered disclosure all around since nobody wants to be seen as lagging.
- tfehring 9mo agoI watched that video around both timestamps and didn't see or hear any mention of LIDAR, only of video.
- smallmancontrov 9mo agoExactly: they convert video into a world model representation suitable for 3D exploration and simulation without using LIDAR (except perhaps for scale calibration).
- tfehring 9mo agoMy mistake - I misinterpreted your comment, but after re-reading more carefully, it's clear that the video confirms exactly what you said.
- IhateAI_3 9mo agotesla is not impressive, I would never put my child in one
- uejfiweun 9mo agoI've always wondered... if Lidar + Cameras is always making the right decision, you should theoretically be able to take the output of the Lidar + Cameras model and use it as training data for a Camera only model.
- olex 9mo agoThat's exactly what Tesla is doing with their validation vehicles, the ones with Lidar towers on top. They establish the "ground truth" from Lidar and use that to train and/or test the vision model. Presumably more "test", since they've most often been seen in Robotaxi service expansion areas shortly before fleet deployment.
- bob_theslob646 9mo agoIs that exactly true though? Can you give a reference for that?
- olex 9mo agoI don't have a specific source, no. I think it was mentioned in one of their presentation a few years back, that they use various techniques for "ground truth" for vision training, among those was time series (depth change over time should be continuous etc) and iirc also "external" sources for depth data, like LiDAR. And their validation cars equipped with LiDAR towers are definitely being seen everywhere they are rolling out their Robotaxi services.
- senordevnyc 9mo agoare definitely being seen everywhere they are rolling out their Robotaxi services So...nowhere?
- deleted 9mo ago[deleted]
- __alexs 9mo ago> you should theoretically be able to take the output of the Lidar + Cameras model and use it as training data for a Camera only model. Why should you be able to do that exactly? Human vision is frequently tricked by it's lack of depth data.
- shihab 9mo agoI think there are two steps here: converting video to sensor data input, and using that sensor data to drive. Only the second step will be handled by cars on road, first one is purely for training.
- ActorNightly 9mo agoThe purpose of lidar is to prove error correction when you need it most in terms of camera accuracy loss. Humans do this, just in the sense of depth perception with both eyes.
- dbt00 9mo ago(Always worth noting, human depth perception is not just based on stereoscopic vision, but also with focal distance, which is why so many people get simulator sickness from stereoscopic 3d VR)
- mikepurvis 9mo agoMy understanding is that contextual clues are a big part of it too. We see a the pitcher wind up and throw a baseball as us more than we stereoscopically track its progress from the mound to the plate. More subtly, a lot of depth information comes from how big we expect things to be, since everyday life is full of things we intuitively know the sizes of, frames of reference in the form of people, vehicles, furniture, etc . This is why the forced perspective of theme park castles is so effective— our brains want to see those upper windows as full sized, so we see the thing as 2-3x bigger than it actually is. And in the other direction, a lot of buildings in Las Vegas are further away than they look because hotels like the Bellagio have large black boxes on them that group a 2x2 block of the actual room windows.
- kevindamm 9mo agoActually the reason people experience vection in VR is not focal depth but the dissonance between what their eyes are telling them and what their inner ear and tactile senses are telling them. It's possible they get headaches from the focal length issues but that's different.
- wolrah 9mo ago> Always worth noting, human depth perception is not just based on stereoscopic vision, but also with focal distance Also subtle head and eye movements, which is something a lot of people like to ignore when discussing camera-based autonomy. Your eyes are always moving around which changes the perspective and gives a much better view of depth as we observe parallax effects. If you need a better view in a given direction you can turn or move your head. Fixed cameras mounted to a car's windshield can't do either of those things, so you need many more of them at higher resolutions to even come close to the amount of data the human eye can gather.
- mycall 9mo agoThat is still important for safety reasons in case someone uses a LiDAR jamming system to try to force you into an accident.
- Jyaif 9mo agoIf somebody wants to hurt you while you are traveling in a car, there are simpler ways.
- etrautmann 9mo agoIt’s way easier to “jam” a camera with bright light than a lidar, which uses both narrow band optical filters and pulsed signals with filters to detect that temporal sequence. If I were an adversary, going after cameras is way way easier.
- sroussey 9mo agoOh yeah, point a q-beam at a Tesla at night, lol. Blindness!
- dooglius 9mo agoThey may be trying to suggest that, that claim does not follow from the quoted statement.
- sschueller 9mo agoAutonomous cars need to be significantly better than humans to be fully accepted especially when an accident does happen. Hence limiting yourself to only cameras is futile.
- mavhc 9mo agoSurely as soon as they're safer than humans they should be deployed as fast as possible to save some of the 3000 people who are killed by human drivers every day
- cman1444 9mo agoOf course they should be, but that's not what will happen. Humans are not rational, so self-driving cars must be significantly safer than human drivers to avoid as much political pushback as possible.
- OGEnthusiast 9mo agoWhat's going to happen to all the millions of drivers who will lose their job overnight? In a country with 100 million guns, are we really sure we've thought this through?
- 0x457 9mo agoYes, let's stop all progress and roll-back all automation to keep hypothetical angry people with guns happy.
- Phenomenit 9mo agoSeems like a good description on current events.
- runarberg 9mo agoAutonomous private cars is not the technological progress you think it is. We’ve had autonomous trains for decades, and while it provides us with a more efficient and cost effective public transit system, it didn’t open the doors for the next revolutionary technology. Self driving cars is a dead end technology, that will introduce a whole host of new problems which are already solved with public transit, better urban planning, etc.
- pnut 9mo agoNope. Humans are statistically fallible and their attention is too valuable to be obliged to a mundane task like executing navigation commands. Redesigning and rebuilding city transportation infrastructure isn't happening, look around. Also personal agency limits public transportation as a solution.
- runarberg 9mo agoUnlike autonomous driving, public transit is a proven solution employed in thousands of cities around the world, on various scales, economies, etc. > Redesigning and rebuilding city transportation infrastructure isn't happening, look around. We have been redesigning and rebuilding city transportation infrastructure since we had cities. Where I live (Seattle) they are opening a new light rail bridge crossing just next month (first rail over a floting bridge; which is technologically very interesting), and two new rail lines are being planned. In the 1960s the Bay area completely revolutionized their transit sytem when they opened BART. I think you are simply wrong here.
- add-sub-mul-div 9mo ago"Autonomous" https://cybernews.com/news/waymo-overseas-human-agents-robotaxi-security-concerns/ https://cybernews.com/news/waymo-overseas-human-agents-robot...
- Rebelgecko 9mo agoMy understanding is that support is basically playing an RTS (point and click), not a 1P driving game. Which makes sense, if they were directly controlling the vehicles they'd put support in central America for better latency, like the food delivery bot drivers
- jonas21 9mo agoYeah. Waymo described how this works a couple of years ago: https://waymo.com/blog/2024/05/fleet-response/ https://waymo.com/blog/2024/05/fleet-response/
- turtlesdown11 9mo agoRight, I totally believe Waymo, just like I totally believed Amazon's checkout-less stores.
- Rebelgecko 9mo agoWhat's your theory for how it works?
- TulliusCicero 9mo agoThis isn't news, they've always acknowledged that they have remote navigators that tell the cars what to do when they get stuck or confused. It's just that they don't directly drive the car.
- tamimio 9mo agoYeah I have some videos of these drivers in action, I think the sensors are as assistance and but not the whole story, so yeah there’s models lidars etc etc but human factor is there, unfortunately this means we should see soon many cobitics are teleopetated remotely from India and Philippines and the likes to satisfy the greed of these companies to pay peanuts to operate them.
- mgaunard 9mo agoStill needs to be trained on the final boss: dense cities with narrow streets.
- xnx 9mo agoWhat would be an example city? Waymo just announced they're ramping up in Boston: https://waymo.com/blog/?modal=short-back-to-boston https://waymo.com/blog/?modal=short-back-to-boston "we’re excited to continue effectively adapting to Boston’s cobblestones, narrow alleyways, roundabouts and turnpikes."
- ginko 9mo agoNot grandparent but I was rather thinking of medieval city centers in Italy or Spain. edit: Case in point: https://maps.app.goo.gl/xxYQWHrzSMES8HPL8 https://maps.app.goo.gl/xxYQWHrzSMES8HPL8 This is an alley in Coimbra, Portugal. A couple years ago I stayed at a hotel in this very street and took a cab from the train station. The driver could have stopped in the praça below and told me to walk 15m up. Instead the guy went all the way up then curved through 5-10 alleys like that to drop me off right right in front of my place. At a significant speed as well. It was one of the craziest car rides I've ever experienced.
- stackedinserter 9mo agoDo we really need FSD cars (any cars, actually) in medieval city centers?
- pants2 9mo agoAny small city in Italy is going to be 10X more challenging than Boston
- threetonesun 9mo agoDepends, which is harder: a narrow street or a three lane one with no obvious lane markers with people double parking?
- NullHypothesist 9mo agoI wonder if they can simulate the Beatles crossing the street at Abbey Road in the late '60s
- seanhunter 9mo agoAs a Londoner who used to have to ride up Abbey Road at least once per week there are people on that crossing pretty much all day every day reproducing that picture. So now Waymo are in Beta in London[1] they have only to drive up there and they'll get plenty of footage they could use for taht. [1] I've seen a couple of them but they're not available to hire yet and are still very rare.
- permenant 9mo agoWill Google finally fund Christopher Wren's post great fire "wide streets" rebuild of the City?
- ddalex 9mo agoi think we might need aother great fire to widen the streets at this point
- seanhunter 9mo agoYeah it’s interesting to imagine a London that had such a rebuild, like Napoleon’s rebuild of Paris. I personally love the weird narrow streets and little alleyways of the City, but that’s because when I’m there I’m pretty much exclusively on foot having taken the tube in.
- devmor 9mo agoWow, interesting timing for this PR blast considering the admission in the Senate Commerce Committee hearing. Not transparent at all!
- WarmWash 9mo agoWhat was the admission? That they use cheap labor to provide the waymo clarity when it is confused? That has been known for a long time.
- devmor 9mo agoSoftware doesn’t get confused - it fails. Referring to your software as autonomous when you have to staff a 24/7 response center of humans to control it is not just misleading, it’s a lie.
- WarmWash 9mo agoIt's worse than that for you, they never take control of the waymo, the waymo provides solutions but is unsure which one is correct. The people then tell the waymo which solution is the best one. So maybe unsure is a better term than confused?
- devmor 9mo agoI don’t think anthropomorphizing an algorithm is the right approach to discussing this topic. It’s still failing software.
- ra7 9mo agoThe novel aspect here seems to be 3D LiDAR output from 2D video using post-training. As far as I'm aware, no other video world models can do this. IMO, access to DeepMind and Google infra is a hugely understated advantage Waymo has that no other competitor can replicate.
- codexb 9mo ago3d from moving 2d images has been a thing for decades.
- ra7 9mo agoThis is 3D LiDAR output (multimodal) from 2D images.
- promiseofbeans 9mo agoLiDAR is the technology used to do spatial capture. The output is just point clouds of surfaces. So they’re generating surface point clouds from video
- moffkalast 9mo agoIt's not unheard of, there are a handful [0] of metric monodepth methods that output data that's not unlike a really inaccurate 3D lidar, though theirs certainly looks SOTA. [0] https://github.com/YvanYin/Metric3D https://github.com/YvanYin/Metric3D
- anupamchugh 9mo agoThis is the real story buried under the simulation angle. If you can generate reliable 3D LiDAR from 2D video, every dashcam on earth becomes training data. Every YouTube driving video, every GoPro clip, every security camera feed. Waymo's fleet is ~700 cars. The internet has millions of hours of driving footage. This technique turns the entire internet into a sensor suite. That's a bigger deal than the simulation itself.
- PeterStuer 9mo agoImagine driving in a Waymo 'out of a raging fire'. Talk about edge cases. But, what would you do? Trust the Waymo, or get out (or never get in) at the first sign of trouble?
- breckinloggins 9mo agoInteresting question. If the Waymo was driving aggressively to remove us from the situation but relatively safely I might stay in it. This does bring up something, though: Waymo has a "pull over" feature, but it's hidden behind a couple of touch screen actions involving small virtual buttons and it does not pull over immediately. Instead, it "finds a spot to pull over". I would very much like a big red STOP IMMEDIATELY button in these vehicles.
- tensor 9mo agoCan you not just unlock and open the door? Wouldn't that cause it to immediately stop? Or can you not unlock the door manually? I'd be surprised if there was not an emergency door release.
- bragr 9mo ago>it's hidden behind a couple of touch screen actions involving small virtual buttons and it does not pull over immediately It was on the home screen when I've taken it, and when I tested it, it seemed to pull to the first safe place. I don't trust the general pubic with a stop button.
- phainopepla2 9mo agoImagine how many drunk/careless passengers might press it. Stopping in the middle of the street or highway could be a serious safety hazard.
- arctic-true 9mo agoI feel like this ends with drunk morons accidentally creating Waymo barricades and totally ruining Mardi Gras
- kylehotchkiss 9mo ago
- jimt1234 9mo agoThis might be relevant to the timing here: https://eletric-vehicles.com/waymo/waymo-exec-admits-remote-operators-in-philippines-help-guide-us-robotaxis/ https://eletric-vehicles.com/waymo/waymo-exec-admits-remote-...
- m0llusk 9mo agoSeems interesting, but why is it broken. Waymo repeatedly directed multiple automated vehicles into the private alley off of 5th near Brannan in SF even after being told none of them have any business there ever, period. If they can sense the weather and stuff then maybe they could put out a virtual sign or fence that notes what appears to be a road is neither a through way nor open to the public? I'm really bullish on automated driving long term, but now that vehicles are present for real we need to start to think about potentially getting serious about finding some way to get them to comply with the same laws that limit what people can do.
- tanseydavid 9mo ago>> get them to comply with the same laws that limit what people can do I think you meant, "Attempt" to limit what people can do. Driving in SF (for example) provides many opportunities to see "free will" exerted in the most extreme ways -- laws be damned.
- m0llusk 9mo agoHave you ever seen a cop trying to pull over a Waymo? It isn't going well.
- turtlesdown11 9mo agoHow many Filipinos, who do not have US drivers licenses, does it take to drive this new model?
- andrewmcwatters 9mo agoIt feels intellectually negligent to me that models like those used by Openpilot have no auditable concept of what a car, or a sign, or a person is. Just where lines are and when a car should accelerate or break. The rest of the latent state is "based on pixels."
- mellosouls 9mo agoDeepmind's Project Genie under the hood (pun intended). Deepmind & Waymo both Alphabet(Google) subsidiaries obv. https://deepmind.google/blog/genie-3-a-new-frontier-for-world-models/ https://deepmind.google/blog/genie-3-a-new-frontier-for-worl... Discussed here,eg. Genie 3: A new frontier for world models (1510 points, 497 comments) https://news.ycombinator.com/item?id=44798166 https://news.ycombinator.com/item?id=44798166 Project Genie: Experimenting with infinite, interactive worlds (673 points, 371 comments) https://news.ycombinator.com/item?id=46812933 https://news.ycombinator.com/item?id=46812933
- paxys 9mo agoRegardless of the corporate structure DeepMind is a lot more than just another Alphabet subsidiary at this point considering Demis Hassabis is leading all of Google AI.
- Kapura 9mo agoInteresting that this should come out right as lawmakers are beginning to understand that Waymos have overseas operators making major decisions. [*] https://futurism.com/advanced-transport/waymos-controlled-workers-philippines https://futurism.com/advanced-transport/waymos-controlled-wo...
- thethimble 9mo agoWhy is this relevant at all? Having humans in the loop at some level is necessary for handling rare edge cases safely.
- mrcwinn 9mo agoIf that’s true the system isn’t finished. That’s what reasoning is for.
- sroussey 9mo agoWho ever said they were finished? You think the laid off the team since everything is “done”?
- AlotOfReading 9mo agoThe word "loop" here has multiple meanings. Only one is what you mean and the other person responding to you has understood another. The first is the DDT control loop, what a human driver does. Waymo's remote assistants aren't involved in that. The computer always has responsibility for the safety of the vehicle and decisionmaking while operating, which is why Waymo's humans are remote assistants and not remote drivers. Their safety drivers do participate in the DDT loop, hence the name. But there's also another "loop" of human involvement. Sometimes the vehicle doesn't understand the scene and asks humans for advice about the appropriate action to take. It's vaguely similar to captchas. The human will usually confirm the computer's proposed actions, but they can also suggest different actions. The computer the advice as a prior to continue operating instead of giving up the DDT responsibility. There's very likely a closely monitored SLA between a few seconds to a few minutes on how long it takes humans to start looking at the scene. If something causes the computer to believe the advice isn't safe, it will ignore it. There have been cases where Waymos have erroneously detected collisions and remote assistants were unable to override that decisionmaking. When that happens, a vehicle recovery team is physically sent out to the location. The SLA here is likely between tens of minutes and a couple hours.
- mempko 9mo agoOne interesting thing from this paper is how big of a LiDaR shadow there is around the waymo car which suggests they rely on cameras for anything close (maybe they have radar too?). Seems LiDaR is only useful for distant objects.
- xnx 9mo agoAt least 6 radar units: https://support.google.com/waymo/answer/9190838?hl=en https://support.google.com/waymo/answer/9190838?hl=en
- 999900000999 9mo agoIt doesn't look like they're going to open sources or anything, but I could imagine this would be great for city planning. Or the most realistic game of SimCity you could imagine.
- mattlondon 9mo agoSuddenly all this focus on world models by Deep mind starts to make sense. I've never really thought of Waymo as a robot in the same way as e.g. a Boston Dynamics humanoid, but of course it is a robot of sorts. Google/Alphabet are so vertically integrated for AI when you think about it. Compare what they're doing - their own power generation , their own silicon, their own data centers, search Gmail YouTube Gemini workspace wallet, billions and billions of Android and Chromebook users, their ads everywhere, their browser everywhere, waymo, probably buy back Boston dynamics soon enough (they're recently partnered together), fusion research, drugs discovery.... and then look at ChatGPT's chatbot or grok's porn. Pales in comparison.
- smeeth 9mo agoI always understood this to be why Tesla started working on humanoid robots
- ACCount37 9mo agoPretty much. They banked on "if we can solve FSD, we can partially solve humanoid robot autonomy, because both are robots operating in poorly structured real world environments".
- smt88 9mo agoThey started working on humanoid robots because Musk always has to have the next moonshot, trillion-dollar idea to promise "in 3 years" to keep the stock price high. As soon as Waymo's massive robotaxi lead became undeniable, he pivoted to from robotaxis to humanoid robots.
- senordevnyc 9mo agoYeah, that and running Grok on a trillion GPUs in space lol
- jasondigitized 9mo agoI don't want a humanoid robot. I want a purpose built robot.
- Vosporos 9mo agoThe new frontier is manifestly the Phillipines.
- elliotec 9mo agoCan you explain? I lived in PH, and my guess is that you mean navigating and modeling the unending and constantly changing chaos of the street systems (and lack thereof) is going to be a monumental task which I completely agree with. It would be an impressive feat if possible. Edit: or are you talking about the allegations of workers in the Philippines controlling the Waymos: https://futurism.com/advanced-transport/waymos-controlled-workers-philippines https://futurism.com/advanced-transport/waymos-controlled-wo... I guess both are valid.
- hazrmard 9mo agocue the bell curve meme for learning autonomy: ____.----.____ ______/ \______ _____/ \_____ ________________________________________ (simulations) (real world data) (simulations) Seems like it, no? We started with physics-based simulators for training policies. Then put them in the real world using modular perception/prediction/planning systems. Once enough data was collected, we went back to making simulators. This time, they're physics "informed" deep learning models.
- crazygringo 9mo agoThat's a very interesting way of looking at it. Yes, you start with simulating something simpler than the real world. Then you use the real world. Then you need to go back to simulations for real-world things that are too rare in the real world to train with. Seems like there ought to be a name for this, like so-and-so's law.
- buddhistdude 9mo agohazrmard's law
- crazygringo 9mo agoIt is thus.
- tgrowazay 9mo agoThis page crashes my browser. Vivaldi 7.8.3931.63 on iOS 26.2.1 iPhone 16 pro
- AndrewKemendo 9mo agoFor whatever it’s worth World models is going to be the dominant computing structure of the future I started working heavily on realizing them in 2016 and it is unquestionably (finally) the future of AI
- 01100011 9mo agoNvidia has had this for years. What am I missing?
- smcl 9mo agoThe Waymo driving model: hire some guys in Philippines: https://futurism.com/advanced-transport/waymos-controlled-workers-philippines https://futurism.com/advanced-transport/waymos-controlled-wo...
- themafia 9mo agoDig deep enough into any "AI" idea and you'll find the bottom end of the scam looks exactly like this. We've simply relabeled the "Mechanical Turk" into "AI." The rest is built on stolen copyrighted data. The new corporate model: "just lie the government clearly doesn't give a shit anymore."
- ASalazarMX 9mo agoThis is not false, but gives the wrong idea that foreigners are driving them in real time. > After being pressed for a breakdown on where these overseas operators operate, Peña said he didn’t have those stats, explaining that some operators live in the US, but others live much further away, including in the Philippines. > “They provide guidance,” he argued. “They do not remotely drive the vehicles. Waymo asks for guidance in certain situations and gets an input, but the Waymo vehicle is always in charge of the dynamic driving tasks, so that is just one additional input.”
- ddalex 9mo agoHave you read the article ? The guys in the Philippines are providing high level executive indications, they don't drive remotely the car or have any low level control of the car.
- andreyk 9mo agoThis is quite misleading... From the article: “When the Waymo vehicle encounters a particular situation on the road, the autonomous driver can reach out to a human fleet response agent for additional information to contextualize its environment,” the post reads. “The Waymo Driver [software] does not rely solely on the inputs it receives from the fleet response agent and it is in control of the vehicle at all times.” [from Waymo's own blog https://waymo.com/blog/2024/05/fleet-response/ https://waymo.com/blog/2024/05/fleet-response/] What's the problem with this?
- ge96 9mo agoWhat is the 5/3 tiles? Cameras?
- spaceywilly 9mo agoThe model generates camera and Lidar data. As if it was a Waymo car that drove through the simulated scenario with its cameras running. This synthetic training data can then be used to train the driving models.
- ge96 9mo agoWonder how it'll do. The trees change shape (presumably the Lidar patterns do too). I get the premise/why but it seems odd to me (armchair) to use fake data. Real trees don't change shape (in real time) although it can be windy. It probably doesn't matter though, "this general blob over there"
- LowLevelKernel 9mo agoInstructions to load it on WAYMAX simulator?
- jrm4 9mo ago1. Still hard not to think that this is a huge waste of time as opposed to something that's a little more like a public transport train-ish thing, i.e. integrate with established infrastructure. 2. No seriously, is the filipino driver thing confirmed? It really feels like they're trying to bury that.
- smotched 9mo agoAmerica is not europe, how would public transport work for the last 1/2miles
- goatlover 9mo agoWalking, bikes and scooters.
- smotched 9mo agoagain, this is not europe. we don't ride bikes, and we certainly don't walk.
- kube-system 9mo agoYeah, that’s the root of a lot of problems in the US. But pedestrian infrastructure is a very simple problem to solve, even in the US. Just look at how the market responds to public transit projects in the US —- developers trip over themselves to build projects in response to the induced demand. Likewise they also responded to the reduced demand as a result of ripping out public transit infrastructure in the 20th century. Which is why doughnut-shaped cities are common in the US.
- airstrike 9mo ago"The Filipino driver thing" is simply that there's a manual override ability when this profoundly complex and marvelously novel technology gets trapped in edge cases. Once it gets unstuck, it runs autonomously.
- iknowstuff 9mo ago
- ActorNightly 9mo agoThis is cool, but they are still not going about it the right way. Its much easier to build everything into the compressed latent space of physical objects and how they move, and operate from there. Everyone jumped on the end-2-end bandwagon, which then locks you into the input to your driving model being vision, which means that you have to have things like genie to generate vision data, which is wasteful.
- sagarm 9mo agoThe article is about using the world model to generate simulations, not for controlling the vehicle.
- ActorNightly 9mo agoThey form control policy from vision data directly, which is why they need to have a massive model generate simulation vision data.
- sagarm 9mo agoSystem architecture aside, how else would you test end to end behavior other than starting from sensor inputs?
- ActorNightly 9mo agoMy other comment in the thread explains it. Basically, driving policy needs to be MCTS search on a space that represents physical objects. If I were to build a self driving system here is how I would do it: * Define a 3d representation of the physical space around the car and how it evolves. Basically a very compressed simulator that has an input of initial conditions, and then predicts the evolution of the scene. The big difference here is that you would be manually coding this sim (i.e not training it), because you would be defining rules for things like collisions. You can also conveniently integrate your car control in this sim, with motion based on tire behavior that happens when you turn the steering wheel. * Build probablistic behaviour of other objects (i.e cars/pedestrians) from real world driving data. I.e given a time span of driving, these essentially represent the probability of what the human pedestrian or the human driver would do. * On the sensor side, you would train models to take lidar/camera and create the initial conditions for the sim. I.e things like big trucks would map to big trucks with a lot of mass and inertia, things like obstruction on the road would represent essentially "walls" that you cannot hit, and traffic control objects that represent "soft" boundaries. * On the driver side, you would train something like MuZero to essentially play the driving game within the sim, building both the prediction model at training, and at inference time running MCTS to chose the best optimal policy. Scoring would be done based on things like following traffic control signals and not hitting things, minimizing traffic disturbance, following the GPS route, and so on e.t.c. And this is how you would get superhuman driving. Just like in the cases where a neural net learns to play a particular game and finds really unique strategies, you would see similar things with this. For example, It would be able to avoid collision situations where you would get rear ended, because it would predict a collision, see that emergency lane is open, and create a control plan to move the car out of the way. And from a product perspective, you can imagine how advantageous this would be in terms of development and improvement. And to answer your question, you wouldn't really even need to do end to end as a test for bugs, you would just need to make sure your sensor model is accurate, which can be done simply by driving the car and it observing the world. Its much simpler to do than comparable systems because you don't care about what the object is, you just care whether its part of the terrain or not, and if its not, you really just care about its size in terms of taking up space and its trajectory.
- phailhaus 9mo agoFinally I understand the use case for Genie 3. All the talk about "you can make any videogame or movie" seems to have been pure distraction from real uses like this: limited, time-boxed simulated footage.
- fabmilo 9mo agoVery impressive work from Waymo. The driving with a tornado in the horizon example kind of struck my imagination, many people actually panic in such scenarios. I wonder though the compute requirements to run these simulations and producing so many data points.
- xvxvx 9mo ago[flagged]
- AceJohnny2 9mo agoIIUC, there's a confusion of meaning for "World Model", between Waymo/Deepmind's which is something that can create a consistent world (for use to train Waymo's Driver), vs Yann LeCun/Advanced Machine Intelligence (AMI) which is something that can understand a world.
- xnx 9mo agoI don't think there's a conflict. If you can predict the world you understand it.
- joshuamerrill 9mo agoIt’s impressive to see simulation training for floods, tornadoes, and wildfires. But it’s also kind of baffling that a city full of Waymos all seemed to fail simultaneously in San Francisco when the power went out on Dec 22. A power outage feels like a baseline scenario—orders of magnitude more common than the disasters in this demo. If the system can’t degrade gracefully when traffic lights go dark, what exactly is all that simulation buying us?
- flutas 9mo agoThey've also been seen driving directly into flood waters, with one driving through the middle of a flooded parking lot. https://www.reddit.com/r/SelfDrivingCars/comments/1pem9ep/hmmm_so_this_lot_flooded_but_the_waymo_still/ https://www.reddit.com/r/SelfDrivingCars/comments/1pem9ep/hm...
- coolfox 9mo agocurious what your take away from that is given the announcement.
- GoatOfAplomb 9mo agoAll this simulation buys a single vehicle that drives better. That failure was a fleet-wide event (overloading the remote assistance humans). That is, both are true: this high-fidelity simulation is valuable and it won't catch all failure modes. Or in other words, it's still on Waymo for failing during the power outage, but it's not uniquely on Waymo's simulation team.
- xnxnxkx 9mo ago[flagged]
- deleted 9mo ago[deleted]
- t1234s 9mo agoCould these world models be used to build some sort of endless GranTurismo type street racing game?
- crazygringo 9mo agoIt seems inevitable that they'll soon be used as the starting points for developing almost all video game environments. Not for the rendering (that's still way too expensive), but for the initial world generation that gets iteratively refined and then still ultimately gets converted into textured triangles.
- creata 9mo ago> It seems inevitable that they'll soon be used as the starting points for developing almost all video game environments. Almost all video game environments? No way. That statement really needs to be qualified with the genres of games you're considering.
- nightpool 9mo agoInteresting, but it feels like it's going to cope very poorly with actually safety-critical situations. Having a world model that's trained on successful driving data feels like it's going to "launder" a lot of implicit assumptions that would cause a car to get into a crash in real life (e.g. there's probably no examples in the training data where the car is behind a stopped car, and the driver pulls over to another lane and another car comes from behind and crashes into the driver because it didn't check its blindspot). These types of subtle biases are going to make AI-simulated world models a poor fit for training safety systems where failure cannot be represented in the training data, since they basically give models "free reign" to do anything that couldn't be represented in world model training.
- deleted 9mo ago[deleted]
- 420official 9mo agoYou're forgetting that they are also training with real data from the 100+ million miles they've driven on real roads with riders, and using that data to train the world model AI. > there's probably no examples in the training data where the car is behind a stopped car, and the driver pulls over to another lane and another car comes from behind and crashes into the driver because it didn't check its blindspot This specific scenario is in the examples: https://videos.ctfassets.net/7ijaobx36mtm/3wK6IWWc8UmhFNUSyy4kCy/4595f848baa762961c945a8e080bbc16/fast_long_rollout_1.webm https://videos.ctfassets.net/7ijaobx36mtm/3wK6IWWc8UmhFNUSyy... It doesn't show the failure mode, it demonstrates the successful crash avoidance.
- MillionOClock 9mo agoWhile there most likely is going to be some bias in the training of those kinds of models, we can also hope that transfer learning from other non-driving videos will at least help generate something close enough to the very real but unusual situations you are mentioning. We could imagine an LLM serving as some kind of fuzzer to create a large variety of prompts for the world model, which as we can see in the article seems pretty capable at generating fictive scenarios when asked to. As always tho the devil lies in the details: is an LLM based generation pipeline good enough? What even is the definition of "good enough"? Even with good prompts will the world model output something sufficiently close to reality so that it can be used as a good virtual driving environment for further training / testing of autonomous cars? Or do the kind of limitations you mentioned still mean subtle but dangerous imprecisions will slip through and cause too poor data distribution to be a truly viable approach? My personal feeling is that this we will land somewhere in between: I think approaches like this one will be very useful, but I also don't think the current state of AI models mean we can have something 100% reliable with this. The question is: is 100% reliability a realistic goal? Human drivers are definitely not 100% reliable. If we come up with a solution 10x more reliable than the best human drivers, that maybe has some also some hard proof that it cannot have certain classes of catastrophic failure modes (probably with verified code based approaches that for instance guarantees that even if the NN output is invalid the car doesn't try to make moves out of a verifiably safe envelope) then I feel like the public and regulators would be much more inclined to authorize full autonomy.
- ok_dad 9mo agoI'd like to see Waymo have a few of their Drivers do some sim racing training and then compete in some live events. It wouldn't matter much to me if they were fast at all, I'd like to see them go into the rookie classes in various games and see how they avoid crashes from inexperienced players. I believe that it would be the ultimate "shitty drivers vs. AI" test.
- JBorrow 9mo agoRacing and street driving are completely different. Racing involves detailed knowledge of vehicle dynamics and grip. Street driving is mainly obstacle recognition and avoidance. No waymo ever operates anywhere close to the limit of grip, which is where you are all the time when racing.
- ok_dad 9mo agoSure but accident avoidance in sim racing is basically the ultimate test for any driver. I also said it wouldn’t matter if they’re fast, I don’t care about driving at the limit of grip here, just avoiding accidents.
- pcurve 9mo agoDumb question - Why would Waymo disclose this much information to public and competitors?
- xnx 9mo agoIt's easier to build trust for such a safety-critical service when you're more open about how it works an performs. For the complete opposite approach, see Tesla.
- wnevets 9mo agoMaybe to distract from the story that they use remote drivers after one of their cars hit a kid? [1] [1] https://people.com/waymo-exec-reveals-company-uses-operators-in-the-philippines-to-assist-autonomous-vehicles-11900507 https://people.com/waymo-exec-reveals-company-uses-operators... edit: fixed kill -> hit
- docere 9mo agoThe child did not die, and suffered only minor injuries: https://abc7.com/post/california-teamsters-call-suspension-waymos-operating-license-child-hit-santa-monica/18531172/ https://abc7.com/post/california-teamsters-call-suspension-w... Under the same circumstances (kid suddenly emerging between two parked cars and running out onto the street), it could be debated that the outcome could have been worse if a human were driving.
- dan-g 9mo agoIt’s awful a child was hit, but they only suffered minor injuries [1]. Nowhere in your linked article does it say they were killed. [1] https://people.com/waymo-car-hits-child-walking-to-school-during-drop-off-prompting-investigation-11895826 https://people.com/waymo-car-hits-child-walking-to-school-du...
- nurettin 9mo agoI don't know about the remote driver conspiracy, but waymo slowing down and that kid surviving a crash after jumping on the road from behind a tall vehicle was the best PR waymo could have asked for.
- caycep 9mo agoAll this work is impressive, but I'd rather have better trains
- chufucious 9mo agoMe too but given our extensive car brain culture, Waymo is an amazing step to getting less drivers & cars off the road, and to further cement future generations not ever needing to drive or own cars
- xnx 9mo agoIsn't a vehicle that goes from anywhere to anywhere on your own schedule, safely, privately, cleanly, and without billions in subsidies better?
- anigbrowl 9mo agoI don't think individual vehicles can ever achieve the same envirnmental economies of scale as trains. Certainly they're far more convenient (especially for short-haul journeys) but I also think they're somewhat alienating, in that they're engineering humans out of the loop completely which contributes to social atomization.
- xnx 9mo ago> I don't think individual vehicles can ever achieve the same envirnmental economies of scale as trains. I think you'd be surprised. Look at the difference in cost per passenger mile.
- Mawr 9mo agoI'm looking. Comes out unfavorably to cars. Obviously. I guess you're comparing the total cost of trains vs a subset of costs of cars, as is usual. Road use and pollution are free externalities after all.
- kidk 9mo agoBillions of subsidies? Im confused you talking about cars or trains.
- threethirtytwo 9mo agoWhat if we put this mechanism of recording the world on people. We have mics listening to people talking to us and noises we hear. Also we record body position actuation and self speech. As output then we put this on thousands of people to get as much data as Waymo gets. I mean that’s what we need to imitate agi right? I guess the only thing missing is the memory mechanism. We train everything as if it’s an input and output function without accounting for memory.
- anigbrowl 9mo agoSeems relevant: Waymo exec admits remote operators in Philippines help guide US Robotaxis https://news.ycombinator.com/item?id=46918043 https://news.ycombinator.com/item?id=46918043
- KeyBoardG 9mo agoand I literally just saw the other headline "Waymo says its robotaxis get help from remote workers in the Philippines"
- cratermoon 9mo agoMeanwhile. https://eletric-vehicles.com/waymo/waymo-exec-admits-remote-operators-in-philippines-help-guide-us-robotaxis/ https://eletric-vehicles.com/waymo/waymo-exec-admits-remote-...
- selenajennifer 9mo ago[dead]
- yummypaint 9mo agoBy leveraging Genie’s immense world knowledge, it can simulate exceedingly rare events—from a tornado to a casual encounter with an elephant—that are almost impossible to capture at scale in reality. The model’s architecture offers high controllability, allowing our engineers to modify simulations with simple language prompts, driving inputs, and scene layouts. Notably, the Waymo World Model generates high-fidelity, multi-sensor outputs that include both camera and lidar data. How do you know the generated outputs are correct? Especially for unusual circumstances? Say the scenario is a patch of road is densely covered with 5 mm ball bearings. I'm sure the model will happily spit out numbers, but are they reasonable? How do we know they are reasonable? Even if the prediction is ok, how do we fundamentally know that the prediction for 4 mm ball bearings won't be completely wrong? There seems to be a lot of critical information missing.
- joshfee 9mo agoIt is great being able to generate a much larger universe of possibilities than what they can gather from real world data collection, but I'd be curious to learn how they check that the generated data is a superset of the possibility-space seen in the real world (e.g. confirm that their models closely match what is seen in the real world too)
- LightBug1 9mo agoHave been seeing Waymo test vehicles regularly around central London recently, operating at speed. For shits and giggles, I did stop randomly while crossing the road and acted like a jerk. The Waymo did, in fact, stop. Kudos, Waymo
- 0xTJ 9mo agoInteresting, but I am very sceptical. I'd be interested in seeing actual verified results of how it handles a road with heavy snow, where the only lane references are the wheel tracks of other vehicles, and you can't tell where the road ends and the snow-filled ditch begins.
- londons_explore 9mo agoDo wayno models really use side cameras at only like 4 FPS?
- SebastianSosa 9mo agoVery concerned with this direction of training “counterfactual events such as whether the Waymo Driver could have safely driven more confidently instead of yielding in a particular situation.” Seems dicey. This could lead in the direction to a less safe Waymo. Since the counterfactual will be generated, I suspect that that the generations will be biased towards survivor situations where most video footage in its training data will be from environments where people reacted well not those that ended in tragedy. Emboldening Waymo on generated best case data. THIS IS DANGEROUS!!!
- kevincox 9mo agoDriving is always a balance between speed and safety. If you want ultimate safety you just sit in the driveway. But obviously that isn't useful. So functionally one of the most important things a self-driving system will decide is "how fast is it safe to drive right now". Slower is not always better and it has to balance safety with productivity.
- Mawr 9mo agoNot entering a roundabout when it's clearly safe to do so is a mark against you at a driving exam. So would be always driving at 5mph. It's just not that simple.
- ajcp 9mo agoNot at all. It's not the counter-factual they're generating, it's the "too rare to capture often enough to train a response to" they're generating. They're implying that without the model having knowledge, even approximate, of a scene to react to, it simply doesn't react at all; it simply "yields" to the situation until it passes. In my experience taking Waymo's almost daily this holds. I would rather not have the Waymo yield to a tornado, rising flood-waters, or charging elephant...
- b_brief 9mo agoI would love to see more visibility into how this model’s simulation fidelity maps onto measurable safety improvements on public roads, especially in unusual edge conditions like partial sensor occlusion or atypical weather.
- ex-aws-dude 9mo agoI don't get how this solves the problem of edge cases with self driving Even if you can generate simulated training data, don't you still have the problem where you don't even know what the edge cases you need to simulate are in the first place?
- sixo 9mo agoWell it certainly helps,doesn't it? This system is going to encounter more edge cases than a single human ever would. Hopefully the lessons from known unknowns generqlise to unknowns. And once they've been seen once they took can become part of the corpus.
- ex-aws-dude 9mo agoRight but does this just rely on some human "brainstorming" a bunch of edge cases? It just strikes me as neverending edge case wack-a-mole A human doesn't need to see tons of examples of tornados and elephants to know to stop the car Doesn't that indicate some fundamental difference between the model and a human driver?
- Mawr 9mo agoNeither does the car — it won't drive into what LIDAR sees as a wall. But stopping is not good enough, it needs to be able to navigate the obstacle as well. Also, even if the car behaved perfectly anyway, these scenarios are useful for testing — validating that the expected behavior happens.
- sixo 9mo agoIt might be "never-ending", but you're going to encounter edge cases in approximate proportion to the rate at which they actually occur. Anyway, the hope would be to learn behaviors which generalize, not to respond to each edge case ad-hoc; the edge cases provide out-of-sample tests of generalizability.
- dyauspitr 9mo agoI don’t think coming up with novel situations is all that hard. LLMs already do it in text form all the time.
- tbmtbmtbmtbmtbm 9mo agoso insane that this is the direction things are going, instead of just reducing our reliance on cars
- RivieraKid 9mo agoThe term "world model" seems almost meaningless. This is a world model in the same sense as ChatGPT is a world model. Both have some ability to model aspects of the real world.
- unit149 9mo ago[dead]
- andrewdb 9mo agoSo when will multiple Waymo cars communicate input data to one another to avoid the blind spots? This would give the ability to see things other cars cannot see as well.
- heohk 9mo agoI'm curious how they simulate equipment failures, like a flat tire or something
- nurettin 9mo ago> Simulation of the Waymo Driver evading a vehicle going in the wrong direction. It really looks like waymo is the one going in the wrong direction and driving dangerously to evade traffic in this simulation.
- nelsondev 9mo agoNeat! What happens when the simulated data is hallucinated/incorrect? In the example videos, the Golden Gate bridge with snow shows the bridge as 1 road, with total of 3 lanes. But in reality, it’s a split highway with divider, so 2 sides both have 3 lanes, 6 total lanes. What happens when the car “learns” to drive on the simulated incorrect 3 lane example? For example will next time it goes on the real GG bridge hug to the rightmost lane?
- sixo 9mo agoIdeally it would learn a relationship between the sensor input and the correct actions, even if the sensor input is not realistic for the GG in reality.
- seydor 9mo agoWhy don't they call it 'the Matrix' and should i prepare for the plugs?
- LowLevelBasket 9mo agoEvery time I'm in town I use a waymo. It's still a little weird to be a passenger with no driver
- uvesten 9mo agoSo the waymo driver is dreaming, testing out different scenarios, basically…
- crooked-v 9mo agoI'm a little sad that they talk about counterfactuals in the simulations, but then don't show any examples of even a single sharknado or giant loop-de-loop.
- deleted 9mo ago[deleted]
- ubixar 9mo agobeen playing around with world models for sim-to-real transfer lately. the waymo approach looks solid, but curious how you're handling the distribution shift between generated scenes and real sensor data. any tricks for that besides the usual domain randomization?
- patrick451 9mo agoNo human needs to have seen an elephant standing in the road before to know that you should not drive through an elephant standing in the road. These are not "long tail" events as the waymo says. It's a big object in the road. You have seen that hundreds of thousands of times. Calling that a long tail event is an admission that your model has zero ability to generalize.
- simontheowl 9mo agoDoes this model include things like how slippery the road is, or head wind/ cross wind. Or is it only visual / depth model?
- DiscourseFan 9mo agoSo in order to create a self driving car, you must first create the universe
- Zigurd 9mo agoThe amount to which supposedly engineering based rationales are applied to Elon's decisions is cult like behavior. Here's what's really going on: That thing I called full self driving just to troll government regulators? Just make it work and don't complain about what I called it. That rocket that's way too big for orbital payloads but can't go beyond orbit without sending 20 more rockets full of fuel? Just make it work. Occupy Mars! The two car models that work and have a halo effect on the rest of the product line? I'mma cancel that! Just make the stupid truck we're keeping work! We've all had a boss like that. I'm sure they salute each other.
- VivaTechnics 9mo agoSimilar to Kiva Systems which was Amazon's best acquisition, Waymo is simply Google's best acquisition. (We live in San Francisco and it feels much safer around these Waymo cars than average "drivers".)
- gorfian_robot 9mo agothose zoox cars though. watchout!
- qeternity 9mo agoHave there been any reported instances of Waymo cars being stolen? Disabled and then loaded into a lead-lined trailer or something. I imagine the IP running locally on the cars is worth billions.
- gpm 9mo agoI doubt Waymo would publicly talk about this if it did happen. I also doubt the IP is worth that much. Most of the secret sauce to starting a competitor probably isn't an end model tuned for a specific configuration of a car but the ability to produce end models, which wouldn't be stealable from the car.
- _trampeltier 9mo agoHow large is the controlled area around the car? And how high do they look for objects? Like something is falling from a bridge, a falling pole or more extreme an falling plane.
- diamondfist25 9mo agoAh the typical MDS npc comes out in the swarm, saying how FSD is no better than GM solutions. Time to eat pop corn, while FSD drives me for the next hour while I scroll hackernews
- pstuart 9mo agoIt's clear that lidar is superior to the Tesla cameras only approach, but I wonder if being able to take all that data and be able to better correlate what is being seen visually with what the lidar is showing. That said, for autonomous driving I'd like to see as many sensor options as possible: lidar, radar, cameras, sonar. Belt and suspenders. I imagine as pricing continues to drop all will be embraced.
- engelo_b 9mo ago[dead]
- rfv6723 9mo agoThe "world model" is a convenient fiction. Whether we’re talking about a carbon-based brain or a silicon-based transformer, there is no miniature, objective map of reality tucked away inside. What we mistake for a "model" is actually just the layered residue of experience. From the perspective of enactivism and radical empiricism, intelligence doesn't "represent" the world; it simply navigates it. A biological organism doesn't need a 3D CAD file of a tree to survive; it only needs a history of sensory-motor contingencies—the "if I move this way, I see that" patterns. It’s a synthesis of interactions, not a library of blueprints. AI operates on the same logic, albeit through a different medium. It isn't simulating the physical laws of the universe or "understanding" gravity. Instead, it navigates the high-dimensional geometry of human data. It’s a sophisticated engine of association, performing a high-speed synthesis of the patterns we've left behind. In this view, "knowing" isn't about matching an internal image to an external truth. It is the seamless flow of past inputs into future predictions. There is no world model—only the habit of being.
- amai 9mo agoSo the solution to autonomous driving will be taking all the data from Google Street View, build a world model for training, and in the end you have an AI model basically remembering every street on planet earth?