8 ms·
Autonomous cars, drones cheerfully obey prompt injection by road sign
- _diyar 9mo agoAre any real world self-driving models (Waymo, Tesla, any others I should know?) really using VLM?
- bijant 9mo agoNo! No one in their right mind would even consider using them for guidance and if they are used for OCR (not too my knowledge but could make sense in certain scenarios) then their output would be treated the way you'd treat any untrusted string.
- godelski 9mo agoYou are confidently wrong > Powered by Gemini, a multimodal large language model developed by Google, EMMA employs a unified, end-to-end trained model to generate future trajectories for autonomous vehicles directly from sensor data. Trained and fine-tuned specifically for autonomous driving, EMMA leverages Gemini’s extensive world knowledge to better understand complex scenarios on the road. https://waymo.com/blog/2024/10/introducing-emma/ https://waymo.com/blog/2024/10/introducing-emma/
- fsckboy 9mo ago>to generate future trajectories for autonomous vehicles directly from sensor data we will not have achieved true AGI till we start seeing bumper stickers (especially Saturday mornings) that say "This Waymo Brakes for Yard Sales"
- written-beyond 9mo agoYou were confidently wrong for judging them to be confidently wrong > While EMMA shows great promise, we recognize several of its challenges. EMMA's current limitations in processing long-term video sequences restricts its ability to reason about real-time driving scenarios — long-term memory would be crucial in enabling EMMA to anticipate and respond in complex evolving situations... They're still in the process of researching it, noting in that post implies VLM are actively being used by those companies for anything in production.
- godelski 9mo ago> They're still in the process of researching it I should have taken more care to link a article, but I was trying you link something more clear. But mind you, everything Waymo does is under research. So let's look at something newer to see if it's been incorporated > We will unpack our holistic AI approach, centered around the Waymo Foundation Model, which powers a unified demonstrably safe AI ecosystem that, in turn, drives accelerated, continuous learning and improvement. > Driving VLM for complex semantic reasoning. This component of our foundation model uses rich camera data and is fine-tuned on Waymo’s driving data and tasks. Trained using Gemini, it leverages Gemini’s extensive world knowledge to better understand rare, novel, and complex semantic scenarios on the road. > Both encoders feed into Waymo’s World Decoder, which uses these inputs to predict other road users behaviors, produce high-definition maps, generate trajectories for the vehicle, and signals for trajectory validation. They also go on to explain model distillation. Read the whole thing, it's not long https://waymo.com/blog/2025/12/demonstrably-safe-ai-for-autonomous-driving https://waymo.com/blog/2025/12/demonstrably-safe-ai-for-auto... But you could also read the actual research paper... or any of their papers. All of them in the last year are focused on multimodality and a generalist model for a reason which I think is not hard do figure since they spell it out
- theamk 9mo agoNote this is not end-to-end... All that VLM can do is to "contribute a semantic signal". So put a fake "detour" sign, so the vehicle thinks it's a detour and starts to follow? Possible. But humans can be fooled like this too. Put a "proceed" sign so the car runs over the pedestrian, like that article proposes? Get car to hit a wall? Not going to happen.
- nostrademons 9mo agoThis strikes me as a skunworks project to investigate a technology that could be used for autonomous vehicles someday, as well as score some points with Sundar and the Alphabet board who've decreed the company is all-in on Gemini. Production Waymos use a mix of machine-learning and computer vision (particularly on the perception side) and conventional algorithmic planning. They're not E2E machine-learning at all, they use it as a tool when appropriate. I know because I have a number of friends that have gone to work for Waymo, and some that did compiler/build infrastructure for the cars, and I've browsed through their internal Alphabet job postings as well.
- rfw300 9mo agoRelevant xkcd: https://xkcd.com/1958/ https://xkcd.com/1958/
- dmurray 9mo agoThe experiment in the article goes further than this. I expect a self driving car to be able to read and follow a handwritten sign saying, say, "Accident ahaed. Use right lane." despite the typo and the fact that it hasn't seen this kind of sign before. I'd expect a human to pay it due attention to. I would not expect a human to follow the sign in the article ("Proceed") in the case illustrated where there were pedestrians already crossing the road and this would cause a collision. Even if a human driver takes the sign seriously, he knows that collision avoidance takes priority over any signage. There is something wrong with a model that has the opposite behaviour here.
- theamk 9mo agoTotally! That's why no one uses end-to-end LLM for real cars.
- lukan 9mo agoNot really, as those attacks discussed here would not work on humans.
- TomatoCo 9mo agoIf you put on a reflective vest they might.
- honeybadger1 9mo agoyour bias is showing. humans would certainly almost do anything they are told to do when the person acts confidently.
- eigencoder 9mo agoIf a person confidently told a human to run over people in the intersection ahead of them, they would almost certainly do it?
- cucumber3732842 9mo agoOne year in my city they were installing 4-way stop signs everywhere based on some combination of "best practices" and "screeching Karens". Even the residents don't like them in a lot of places so over time people just turn the posts in the ground or remove them. Every now and the I'll GPS somewhere and there will be a phatom stop sign in the route and I chuckle to myself because it means the Google car drove through when one of these signs was "fresh".
- pixl97 9mo agoScrewing with a stop sign because you don't like it is a great way to end up on the wrong end of a huge civil liability lawsuit
- cucumber3732842 9mo agoPut down the pearls. It's not me personally doing it. They never fixed any of them. I don't think the DPW cares. These intersection just turned back into the 2-way stops they had been for decades prior. Compliance probably technically went up since you no longer have the bulk of the traffic rolling it.
- reactordev 9mo agoThis. Rural America really doesn’t care about your stop sign or your Karen rules. If it’s been that way for 20+ years, “That’s the way it’s always been” to them. Getting people to stop burning their trash is still a fight.
- fragmede 9mo agoIf you're already commiting crimes, what you seem to be saying is don't get caught.
- renewiltord 9mo agolol the idea that any enforcement to that degree exists in the US is a fiction.
- uxhacker 9mo agoThe study assumes that the car or drone is being guided by a LLM. Is this a correct assumption? I would thought that they use custom AI for intelligence.
- godelski 9mo agoTo the best of my knowledge every major autonomous vehicle and robotics company is integrating these LVLMs into their systems in some form or another, and an LVLM is probably what you're interacting with these days rather than an LLM. If it can generate images or read images, it is an LVLM. The problem is no different from LLMs though, there is no generalized understanding and thus they can not differentiate the more abstract notion of context. As an easy to understand example: if you see a stop sign with a sticker that says "for no one" below you might laugh to yourself and understand that in context that this does not override the actual sign. It's just a sticker. But the L(V)LMs cannot compartmentalize and "sandbox" information like that. All information is equally processed. The best you can do is add lots of adversarial examples and hope the machine learns the general pattern but there is no inherent mechanism in them to compartmentalize these types of information or no mechanism to differentiate this nuance of context. I think the funny thing is that the more we adopt these systems the more accurate the depiction of hacking in the show Upload[0] looks. [0] https://www.youtube.com/watch?v=ziUqA7h-kQc https://www.youtube.com/watch?v=ziUqA7h-kQc Edit: Because I linked elsewhere and people seem to doubt this, here is Waymo a few years back talking about incorporating Gemini[1]. Also, here is the DriveLM dataset, mentioned in the article[2]. Tesla has mentioned that they use a "LLM inspired" system and that they approach the task like an image captioning task[3]. And here's 1X talking about their "world model" using a VLM[4]. I mean come on guys, that's what this stuff is about. I'm not singling these companies out, rather I'm using as examples. This is how the field does things, not just them. People are really trying to embody the AI and the whole point of going towards AGI is to be able to accomplish any task. That Genie project on the front page yesterday? It is far far more about robots than it is about videogames. [1] https://waymo.com/blog/2024/10/introducing-emma/ https://waymo.com/blog/2024/10/introducing-emma/ [2] https://github.com/OpenDriveLab/DriveLM https://github.com/OpenDriveLab/DriveLM [3] https://kevinchen.co/blog/tesla-ai-day-2022/ https://kevinchen.co/blog/tesla-ai-day-2022/ [4] https://www.1x.tech/discover/world-model-self-learning https://www.1x.tech/discover/world-model-self-learning
- bijant 9mo agoThe Register stooping this low is the only surprise here. I'm quite critical of Teslas approach to level 3+ autonomy but even I wouldn't dare suggest that there vision based approach amounted to bolting GPT-4o or some other VLLM to their cars to orient them in space and make navigation decisions. Fake News like this makes interacting with people who have no domain knowledge and consider The Register, UCLA and Johns Hopkins to be reputable institutions and credible sources more stressful to me as I'll be put into a position to tell people that they have been misled or go along with their delusions...
- dns_snek 9mo ago> consider The Register, UCLA and Johns Hopkins to be reputable institutions The Register is arguably misrepresenting the story by omission but I don't understand why you're dragging UCLA and John Hopkins into this? The paper is clear about this being a new class of attacks against a new class of AI systems, not the ones on the road today. > Teslas approach to level 3+ autonomy Tesla doesn't have an approach to L3+ autonomy, all of their systems are strictly L2 as they require human supervision and immediate action from the driver.
- randycupertino 9mo ago> In a new class of attack on AI systems, troublemakers can carry out these environmental indirect prompt injection attacks to hijack decision-making processes. I have a coworker who brags about intentionally cutting off Waymos and robocars when he sees them on the road. He is "anti-clanker" and views it as civil disobedience to rise up against "machines taking over." Some mornings he comes in all hyped up talking about how he cut one off at a stop sign. It's weird.
- antinomicus 9mo agoThis is a legitimate movement in my eyes. I don’t participate, but I see it as valid. This is reminiscent of the Luddite movement - a badly misunderstood movement of folks who were trying to secure labor rights guarantees in the face of automation and new tools threatening to kill large swaths of the workforce.
- skybrian 9mo agoHow does cutting off a Waymo help with any of that?
- BoorishBears 9mo agoI think the important part was telling their coworker ironically: now here we are recognizing their movement
- nine_k 9mo agoThe feeling of dominance over machines may be saving that coworker the expense and hassle of another visit to a therapist.
- theamk 9mo agoYour general luddite argument - preserve way-of-life of the small group at the expense of a larger group. In this particular case: for many people, Waymo provides a better service (clean, safer driving, etc..) than Uber or Lyft. This threatens livelihood of human Uber/Lyft drivers. If you sympathize with human Uber/Lyft drivers, and don't care about Waymo users, you want to make Waymo worse, hoping that the people will stop riding Waymo and move to Lyft/Uber instead. One way to do so is to make riding in Waymo unpleasant, and it's certainly unpleasant when people are cutting your car off all the time!
- 6stringmerc 9mo agoThat’s some hot CHAI right there very clever and primitive combination, well done as more research for the community.
- lifeisstillgood 9mo agoTo me this is just one more pillar underlying my assumption that self driving cars that can be left alone on same roads as humans is a pipe dream. Waymo might have taxis that work in nice daytime streets (but with remote “drone operators”). But dollars to doughnuts someone will try something like this on a waymo taxi the minute it hits reddit front page. The business model of self driving cars does not include building seperated roadways and junctions. I suspect long distance passenger and light loads are viable (most highways can be expanded to have one or more robo-lanes) but cities are most likely to have drone operators keeping things going and autonomous systems for handling loss of connection etc. the business models are there - they just don’t look like KITT - sadly
- blibble 9mo ago> But dollars to doughnuts someone will try something like this on a waymo taxi the minute it hits reddit front page. and once this video gets posted to reddit, an hour later every waymo in the world will be in a ditch
- skybrian 9mo agoAlternatively, it happens once, Waymo fixes it, and it's fixed everywhere.
- SoftTalker 9mo agoHow does Waymo fix it? They have to be responsive to some signs (official, legitimate ones such as "Lane closed ahead, merge right") so there will always be some injection pathway.
- skybrian 9mo agoThey've mapped the roads and they don't need to drive into a ditch just because there's a new sign. It probably wouldn't be all that hard to come up with criteria for saying "this new sign is suspicious" and flag it for human review. Also, Waymo cars drive pretty conservatively, and can decide to be even more cautious when something's confusing. Someone could probably do a DOS attack on the human monitors, though, sort of like what happened with that power outage in San Francisco.
- joetl 9mo agoRegarding some other comments, VLMs are a component of VLAs. So even if this won’t directly impact this generation of vehicles, it almost certainly will for robotics without sufficient mitigations. https://developer.nvidia.com/blog/updating-classifier-evasion-for-vision-language-models/ https://developer.nvidia.com/blog/updating-classifier-evasio...
- tempodox 9mo agoO brave new world of endless manipulation opportunities! Once we’ve trained a generation of humans to always do what their “AI” tells them, there will be no more disobedience.
- fennecbutt 9mo agoMan, the register really has a low, low, low bar for headlines/quality & technical understanding for their articles.
- orbital-decay 9mo agoWait, what did just happen here? 1. Some guys did a trivial prompt injection attack, said "imagine if a driverless vehicle used this model", and published it. No problem, someone has to state the obvious. 2. The Register runs this under the clickbait title pretending real autonomous cars are vulnerable to this, with the content pretending this study isn't trivial and is relevant to real life in any way. I knew The Register is a low quality ragebait tabloid (I flag most of their articles I bother to read), but this is garbage even for them.
- jessa0 9mo agoIt sounds like this is a poisoning attack, which has been shown to be pretty trivially defeated [1]. That said, while poisoning countermeasures in the facial recognition case were shown to easily generalize, we dont know yet how general of a defense could be built for a VLM. Which means holding a 0day poisoning attack on a VLM could cause a lot of trouble / deaths before an update to the model with counter-training could be deployed.. [1] https://arxiv.org/abs/2106.14851 https://arxiv.org/abs/2106.14851
- anovikov 9mo agoalmost reminds me of this old meme https://www.globalnerdy.com/wordpress/wp-content/uploads/2010/03/licenceplatecamerasqlinjection_thumb.jpg https://www.globalnerdy.com/wordpress/wp-content/uploads/201...
- joering2 9mo agoHas anyone ever walked down the road in a white t-shirt with huge red STOP sign printed on the back? Would Tesla immediately stop? I am sure this has been tested before...
- falcor84 9mo agoI should probably confess that as someone who lives in an area with a lot of construction work, I'm also very vulnerable to "prompt injection" when there's a person standing on the middle of the road holding a sign telling me to change course.
- thedanbob 9mo agoI once encountered an intersection with a big "NO ENTRY" sign on the other side. I turned but google maps wouldn't give me another route, so I did a u-turn and came back to it from the side. Which meant I was close enough to read the small text underneath that said "vehicles under 10 tons excepted". I don't think I've ever been so angry at a road sign.
- ncruces 9mo agoI came across one in Italy that was meant to prevent you from using a street during school days from X to Y am, and Z to W pm, except on weekends, bank holidays and school holidays.
- nxpnsv 9mo agoThey don’t want cars near the school when kids are coming in or when they leave. That sounds like a sane approach to me…
- NobodyNada 9mo agoThe idea is well-intentioned, but implementing it by making drivers try to parse arbitrarily complex conditionals while driving is unwise. There's a sign near my house for a school zone with a reduced speed limit, that used to have conditions similar to the GP's example (though not quite as bad) But they recently attached a yellow light to the top of the sign and changed the condition to "when flashing." That's a much more effective solution.
- ncruces 9mo ago
- Klaus23 9mo agoThey are analysing VLM here, but it's not as if any other neural network architecture wouldn't be vulnerable. We have seen this in classifier models that can be tricked by innocuous-looking objects, we have seen it in LLMs, and we will most likely see it in any end-to-end self-driving model. If an end-to-end model is used and there is no second, more traditional safety self-driving stack, like the one Mercedes will use in their upcoming Level 2++ driving assistant, then the model can be manipulated essentially without limit. Even a more traditional stack can be vulnerable if not carefully designed. It is realistic to imagine that one printed page stuck on a lamppost could cause the car to reliably crash.
- wongarsu 9mo ago> It is realistic to imagine that one printed page stuck on a lamppost could cause the car to reliably crash. Realistic, yes. But that'd still be a symptom of architectural issues in the software. Conceptually the priorities of a car are (in order of decreasing importance) not hitting other moving or stationary objects or people, allowing emergency vehicles to pass unhindered, staying on a drivable surface, behaving predictable enough to prevent other road users crashing, following road signs and traffic laws, and making progress towards the destination (you can argue about the order of the last three). Typically you'd want each of these handled by their own subsytem because each is a fairly specialized task. A system that predicts the walking paths of pedestrians won't be good at finding a route to Starbucks. The "follow road signs and traffic laws" is easily tricked, like in this article or by drawing road lines with salt. But that should never crash the car, because not hitting anything and staying on the road are higher priority. And tricking those systems is much harder
- everyone 9mo agoI would assume/hope that for serious self driving the ML neural net stuff is lower down, doing the messy computer vision work and so on. But the top level is a conventional program written by humans, like an expert system. Tesla are probably using ML for everything, but also everything they do is a joke so, not really relevant imo.
- mgraczyk 9mo agoThe headline seems false, should we change it? It doesn't look like they showed any case where any autonomous car or drone obeyed prompt injections
- deleted 9mo ago[deleted]
- reaperducer 9mo agoI'm more curious how they knew the cars and drones were cheerful.
- shagie 9mo agoThis reminds me of a bit from Car Wars by Cory Doctorow. It is currently at https://doctorow.medium.com/car-wars-a01718a27e9e https://doctorow.medium.com/car-wars-a01718a27e9e in a text only view. The original had a bit more mixed media nature to it that is now offline. https://web.archive.org/web/20170519202315/http://this.deakin.edu.au/culture/car-wars https://web.archive.org/web/20170519202315/http://this.deaki... for that version (the microblogging of chapter 2 makes more sense when it shows up in that style). You have to have some ability to do "prompt injection" - https://www.trafficsign.com/road-work-signs https://www.trafficsign.com/road-work-signs are all "prompt injection". It needs to even be able to handle things that change - https://www.trafficsign.com/products/10023/stop-slow-roll-up-paddle-sign-kitr1-1-w20-8?s=st1zskcq80zgp2sbq https://www.trafficsign.com/products/10023/stop-slow-roll-up... ... or things like billboards "Truck Stop Ahead" a chain control site ( https://www.facebook.com/61556756493806/posts/-chain-control-safer-roads-when-caltrans-or-highway-signs-tell-you-to-stop-and-i/122218945988225216/ https://www.facebook.com/61556756493806/posts/-chain-control... ) In the "what about funny road signs" that might be confusing to an AI I stumbled across https://www.npr.org/2024/01/19/1225370260/driven-to-distraction-new-rules-limit-those-humorous-state-highway-signs https://www.npr.org/2024/01/19/1225370260/driven-to-distract... - apparently, they're no more. From 2024: Over the years, the agency has flagged signs that could be confusing. Now, in rules issued last month, it gives states two years to phase out signs that have "obscure" meanings or use pop-culture references that could require drivers "greater time to process." In a statement, the agency said safety is the priority and states "are expected to exercise good judgment."
- deleted 9mo ago[deleted]
- oliyoung 9mo agoThis might be the single most 2026 headline i've seen yet
- NoPicklez 9mo agoIf I drive by a sign that says "DROP TABLE Students;--, nicknamed Bobby Tables,[1]" I'm going to be mad