18 ms·

Tesla Autopilot tricked into accelerating from 35 to 85 mph with modified sign

71 points by harambae 7y ago

loser777 7y ago
This is as good of a demonstration of the big "common sense" gap that end-to-end computer vision approaches have as it is of adversarial examples. Even with an occluded speed limit sign, humans have strong priors on what a sensible speed limit would be based on a more general understanding of the environment (width of lanes, curvature of roads, visibility, road quality, frequency of law enforcement vehicles ;), etc.).
- castratikron 7y ago
  Wonder if the neural net could be trained to know the speed limit? "This scene doesn't look like the usual 85 mph zone..."
  - marrone12 7y ago
    You likely could train it to learn about trends and certain contexts. But then you still have to think and enumerate all of these separate possibilities.
  - fyfy18 7y ago
    Many cars now come with a database of speed limits and display it on the dashboard. Although this shouldn't be relied on for self driving, it could be used to double check. OSM data for types of roads and areas is pretty accurate too, so you could use that ('an 85mph speed limit for a single lane within a residential area doesn't sound correct'). This doesn't need AI, it just needs someone with common sense to come up with a few basic rules about speed limits. If there is some doubt you could prompt the driver to confirm the speed limit change, and use that to build up a database of real world data. It sounds like the Tesla is just using what the camera sees - which is bad if true. Admittedly this was tested on a race track, so maybe there is no data saying otherwise (or even the opposite).
- dr_hooo 7y ago
  I don't think that this in particular is really a good example of this gap. It feels more like a sloppy implementation. Just like humans, an autopilot has multiple sources available for guessing what an appropriate speed would be. Simply including information from a map application should be enough to prevent these kind of mistakes/hacks. Now things like children playing on the side of the road certainly seem more difficult to handle..
dmitrygr 7y ago
Further proving that "self driving" is approximately as hard as AGI, and is nowhere near as "close" as everyone thinks to think it is. A human would have context and common sense and thus know that a residential street will simply not ever have an 85 mph speed limit. And yes, you could special case this case in the code, but there are hundreds of cases where "common sense" is used in driving, and you will never teach your NNs all of it. Prediction: no actual "self driving" (to a point where driver can legally be asleep) on public streets till > 10 years from now
- joshvm 7y ago
  I don't see why you need special cases in the code. Sat nav units have had speed limits built in for years. Why not just look it up in a database? (the car knows where it is after all, and requiring a GPS fix could be mandatory - most drones won't arm without it, for example)
  - dmitrygr 7y ago
    Because those databases are out of date approximately 100% of the time... Road construction, for example is a reason why
    - joshvm 7y ago
      However if the sat nav gives you a prior that says 35 mph, and the road sign says 85 mph, do you trust it? Especially if the adjacent roads are also low speed, and it's clearly not a highway etc. This is perhaps a bad example as there are plenty of roads in the UK that transition from national to 30 in rural areas (and plenty of hamlets that you can bomb through). Roadworks are an obvious situation where you drive at a _lower_ speed. This would be a case where you could use computer vision to identity an obstruction, identify roadworks and then adjust speed. Those databases shouldn't be out of date. Most highway agencies (UK for sure) publish planned works and improvements weeks in advance. Google and Apple both show road closures for example. I belive speed changes are similar.
      - 7y ago
posix_compliant 7y ago
I'm not what you'd call a self-driving car "skeptic" since I do think that we will eventually be able to iron out these types of rules, but I think it will be another 30-50 years before I feel comfortable truly trusting one.
- emiliobumachar 7y ago
  How much do you "truly trust" human drivers?
vardump 7y ago
Another discussion on same topic: https://news.ycombinator.com/item?id=22370346 https://news.ycombinator.com/item?id=22370346
NicoJuicy 7y ago
I'm a self-driving skeptic, but this test is not fair. It's also illegal to do this, so it's not a valid edge-case.
- dmitrygr 7y ago
  Who cares? Would you feel better if you died due to someone breaking the law?
  - NicoJuicy 7y ago
    It's nice as a research part. But more people die because of human error and drunk drivers every day. I have yet to see someone die because of the illegal and "labor intensive" situation they created in this "research". > Would you feel better if you died due to someone breaking the law? I can respond that back, but then with problems that are factual ( drinking and driving). Not fictional or harder to execute do it in practice ( this research)
    - kelnos 7y ago
      I think the issue isn't always entirely rational. Even if self-driving cars could have a better accident record than humans by several orders of magnitude, incidents would still cause humans to react poorly -- "if I were in control that wouldn't have happened" -- even if it would have. Altering a speed limit sign may not be legal, but if it's done in a way that wouldn't fool a human, but would fool a self-driving system, that's just a way that a self-driving system could be unsafe that a human driver wouldn't, and people will react negatively toward the self-driving system. That's just human nature, even if it's not logical. Regardless, we need to be aware of these possible "exploits" on self-driving systems. Sure, no one would bother altering a speed limit sign to cause crashes, because human drivers have context and common sense that would make that mostly ineffective. But if it were known that some popular self-driving systems could be fooled by that, I'd expect to see this happen more often, perpetrated by people who would probably otherwise try to hurt people in other ways. It's not a "new idea" in the context of driving in general, but it's a new -- and potentially very effective -- idea to dangerously mess with self-driving systems.
asdff 7y ago
Some sick people get their kicks tossing rocks off of overpasses. Imagine the games people are going to play with cardboard cutouts of whatever or false signage. I predict an arms race.
- Robotbeat 7y ago
  People could already cause crashes by making fake signs or repainting lines. I think this XKCD summarizes the issue nicely: https://m.xkcd.com/1958/ https://m.xkcd.com/1958/
oyebenny 7y ago
I live in Atlanta and there's a digital speed limit sign on 75. Sometimes the light is out and when it means to say the speed limit is "55" it just says "5". Come to think of it I've always noticed traffic slowing down nearing that sign. I wonder if any Tesla's have any contribution to that.
- emiliobumachar 7y ago
  I would slow down to a speed limit of 5, even if I was sure it was a defective display. It's effectively an unknown speed limit. Maybe not if I'm very familiar with the road, but not everyone is. Yes, I'm sure the courts should throw out the ticket, but sometimes they mess up, and often you lose time and money to pursue your case.
Robotbeat 7y ago
This already happens occasionally with regular drivers: https://www.telegraph.co.uk/news/uknews/road-and-rail-transport/11138390/Drivers-tricked-by-fake-40mph-speed-limit-signs.html https://www.telegraph.co.uk/news/uknews/road-and-rail-transp... "Pranksters blamed for speeding offences after drivers caught out by 40mph signs in 30mph zone" Ironically, current Teslas are largely immune to this as they use a database of speed limits.
Sohcahtoa82 7y ago
While scary, it's not exactly relevant to any Tesla made after they broke off from MobilEye (Which I think was 2016?). Current Teslas don't read speed limit signs, they access an online database. Vandalize the speed limit signs all you want, as modern Teslas aren't even reading them.
luxuryballs 7y ago
Shouldn’t it know that 85 is beyond the speed limit? You’d think it would know the laws based on where the GPS is, like in some states above 80 is reckless driving. Seems trivial to make this data available.
kwhitefoot 7y ago
Very misleading. The car will only accellerate to the speed already chosen by the driver. It doesn't seem odd for itself to accellerate up to speed limits
consp 7y ago
Nice for them use Tesla but I'm pretty sure you can do it with every sign reading auto throttle system. There are plenty of cruise control systems which employ similar technologies.
- shiftpgdn 7y ago
  Or a person who misreads the sign. Though the whole point is that a Tesla doesn't have the context to know that a residential zone with an 85mph speed limit sign is obviously some sort of error.
  - toast0 7y ago
    You don't need residential context. You just need US context. I have never seen an 85 MPH speed limit sign in the US, ever. Up to 65 is common, with 70 or 75 sometimes on well maintained roads between urban areas. I can't recall seeing an 80, but I feel like maybe once or twice.
    - drunken-serval 7y ago
      There’s an 85mph in one place in Texas. https://en.m.wikipedia.org/wiki/Speed_limits_in_the_United_States https://en.m.wikipedia.org/wiki/Speed_limits_in_the_United_S...
    - pwinnski 7y ago
      Hello from Texas! There's a pretty long stretch of road between Austin and San Antonio that's 85 the whole way. It's not as big as the I-35, but the higher speed limit means it draws quite a bit of traffic for people traveling between the two cities. Point stands: 85 is ridiculous in anything like a residential area.
  - bazeblackwood 7y ago
    Accelerating to 85mph on a street where that speed is unsafe is absolutely not what a person who misreads the sign would do. 35mph tend to be that way for a reason (residential, stop signs, curved streets)
deleted 7y ago
[deleted]
wpietri 7y ago
Ah, that's especially good. They just changed the 3 in "35 MPH" speed limit sign to look more like an 8, and the Teslas dutifully sped up. A mistake no human driver would make. Rodney Brooks points out that if we ever get AGI [1], we'll have solved the autonomous vehicle problem. But it's far from clear to me that we'll truly solve the problem much before then, as cars and roadways are built with GI expectations in mind. [1] https://en.wikipedia.org/wiki/Artificial_general_intelligence https://en.wikipedia.org/wiki/Artificial_general_intelligenc...
- koboll 7y ago
  >A mistake no human driver would make. And probably prosecutable as felony manslaughter, the same way removing a stop sign would be, because you're acting in a way you know will get people killed.
  - draw_down 7y ago
    Some people like to attempt to prosecute security researchers for reporting vulnerabilities. Same energy here.
- seanmcdirmid 7y ago
  According to the article, only when they engage TACC for the first time while the sign is visible, not if they were driving for awhile and it was already turned on. Also note that TACC isn’t autopilot, one just controls your speed automatically, the other actually drives your car. It wasn’t clear if the flaw affected auto pilot as well, since it can’t be used for non freeway driving.
elcomet 7y ago
So basically they modified a 35MHP street sign to make it look like a 85MPH. I looked at it from a distance, and it was hard to tell between 35 and 85. So if it is hard for a human, then it seems obvious that a tesla would also fail to recognize a 35. Maybe the difference is that a human might not be certain and slow down just in case, whereas a deep neural network might be certain in its errors.
- tmpz22 7y ago
  A human has the basic intuition to assume the sign itself might be wrong or misleading, the programmers of the car automation failed to realize that in a way that might've gotten someone killed.
  - russfink 7y ago
    Good thing, too, that nobody stepped out in front of their car, like a kid chasing a ball, as they did their ad hoc experiment.
    - frosted-flakes 7y ago
      They didn't allow the car to reach 85 m/h, for obvious reasons.
- pathartl 7y ago
  There's several factors they could use to bring a level of uncertainty to the algorithm. Compare against the previously clocked speed limit, compare against map data, compare against the speed of other vehicles in a proximity, judge based on density of crosswalks, parked cars, traffic, etc. This isn't a failure of autonomy imo. Speed limits are arbitrary placeholders that are more for supplementing the slow reaction speed of the human brain. If we started focusing more on making the algorithm implement actual analysis of the environment instead of "the rules of the road", we might see better results.
- decebalus1 7y ago
  > So if it is hard for a human, then it seems obvious that a tesla would also fail to recognize a 35. You need to be a special type of stupid to think that a road with a speed limit of 35 MPH is 85 because of a road sign. The 'drive into the lake because the GPS nav tells me to' level of stupid.
rootusrootus 7y ago
They had to use older Teslas to make this work, because newer models don't use sign-reading to determine speed limit. Abstractly, it's interesting, if only because it probably wouldn't trick a human -- everyone would see the 85mph as BS regardless of the defaced sign. Has anyone hacked into openstreetmaps yet and fiddled with speed limits?
- ssheth 7y ago
  Yes ... the older Teslas have the MobilEye chipsets which use this technology. Many other cars use the same MobilEye chipsets today .. so potentially could be an issue in other cars as well. Its likely now that this error was pointed out that Tesla may add some speed limit validation code to ensure a car doesn't speed up much when a speed limit jumps by more than 25% or something like that..
  - seanmcdirmid 7y ago
    It only works when this is the first sign TACC sees. If TACC was previously engaged and saw other signs previously, the exploit doesn’t work. I’m sure Tesla fixed this, but given the nature of the edge case, it wasn’t simply a matter of sanity checking.
    - tjoff 7y ago
      Why would it only work if it is the first sign?
      - seanmcdirmid 7y ago
        Maybe because they were probably already doing sanity checking, but the kind that relied on some previous context? That seems reasonable, I’m sure they’ve thought about this kind of attack before, missing an obscure edge case.
- chris11 7y ago
  It still is a little bit unnerving that older models can accelerate based off of speed limit signs alone. That seems like a dangerous approach.
valine 7y ago
Putting aside the fact that this was found on an old Tesla with autopilot 1.0, this is not even a technically hard problem to solve. The current version of autopilot uses a database and gps to determine the speed limit. In the future if Tesla wants to return to a vision based speed limit system they can sanity check the vision readings against their database and throw out results like 85mph.
aero142 7y ago
If the point is that autonomous cars are succeptable to deliberate malicious actors, I've got bad news about the rest of the world. Human drivers are incredibly succeptable to me throwing a $2 brick throught the front windshield as well. It sounds like the issue is already resolved in this case.
- capableweb 7y ago
  I guess the difference here is the scope of how many you can impact. In order to get all Teslas who go past a sign on the highway, you just have to modify the sign slightly and everyone is affected (in theory). In order to affect the same amount of people with your "brick through the windshield" strategy, you'll need a lot more manpower than just a sticker on a sign.
  - justaguyhere 7y ago
    do you really need the manpower though? it shouldn't be that hard to build a mechanical device that can do the brick throwing?
    - aabeshou 7y ago
      you cant seriously be arguing its as easy or invisible to set up a brick throwing machine as it is to put a piece of tape on a speed limit sign?
  - throwanem 7y ago
    That depends on the size of wreck you manage to cause, don't you think? In any case, it takes no more people to drop a brick than it does to sticker a sign. To be clear, I find this entire concept of unsupervised robot vehicles both dangerous and absurd - even more so than the already dangerous and absurd baseline of a society so intimately bound up with automotive travel as ours. Exceeding such a high baseline as that is in its way impressive, and certainly demonstrates the astonishing overconfidence rampant in some segments of our very young and rather careless industry, but let's not get distracted from that essential point and waste our efforts on inconsequential arguments over whether a sticker is more dangerous than a brick.
skymt 7y ago
XKCD on tricking self-driving cars: https://xkcd.com/1958/ https://xkcd.com/1958/ I wanted to link that not because of the "there's an XKCD for everything" meme but because it makes an interesting point: sabotaging roads isn't difficult now, with human drivers. There's no reason to assume it would suddenly become a substantial threat when autonomous cars are commonly adopted. An issue worth considering and accounting for, but not worth public worry.
sebringj 7y ago
What I like about Tesla is they have over the air updates using a fleet of network data to quickly address issues or exploits that may arise... but this article is quite funny exposing the gap between AI and basic human intuition. I do believe human intuition may just be another model to train from in some sense in the end.
RcouF1uZ4gsC 7y ago
>The safety of Tesla's autopilot features has come under close scrutiny, but CEO Elon Musk has predicted the company will have "feature-complete full self-driving" this year. I think that is a real possibility if we are referring to years on Jupiter (which has an orbital period of 12 Earth years).
m0zg 7y ago
Sounds like an easy fix in most cases. Residential road or road work? Can't be 85, period. There are relatively few roads in the US where 85 is the actual speed limit.
whyaduck 7y ago
So cars are using OCR on speed limit signs? Next step: sql injection. https://hackaday.com/2014/04/04/sql-injection-fools-speed-traps-and-clears-your-record/ https://hackaday.com/2014/04/04/sql-injection-fools-speed-tr...
- masklinn 7y ago
  That's pretty common IME though the ones I've driven didn't make decisions based on that. Last few cars I've rented had a speed limit sign generally matching the speed limit on the dash, and on the one I rented this christmas passing a speed limit sign (or an other sign triggering a speed limit change) while on limiter or cruise control would flash a message suggesting pressing a button twice to change the configured speed to that (rather than manually adjust using the +/- buttons). They were mediums (C-segment) and compact MPV not large or luxury, which I expect is why the sign reading was mostly to purely indicative.
- bdcravens 7y ago
  Yes, many cars do now. Some of the "less smart" cars don't necessarily know which signs are which. My 2018 Accord thought the speed limit was 100mph when I was on Highway 99.
JMTQp8lwXL 7y ago
If a human drove at 85 mph because a bad actor maliciously modified the speed limit sign, would they be liable?
- binarymax 7y ago
  Co-reference resolution error: who is 'they' in your sentence? The human or the bad actor? No matter what, the answer is probably both. You can't modify street signs now anyway. You also can't drive 85 in residential/city areas.
  - JMTQp8lwXL 7y ago
    The human operating the vehicle who reads a maliciously edited sign. It seems unquestionable to me anybody modifying street signs would be held liable for the modifications.
tjoff 7y ago
Even if it was a legitimate 85 mph sign a human would realize that the road isn't safe at 85 mph speeds. You must react to the environment as well...
ebg13 7y ago
I keep seeing throughout this thread statements similar to "a human would never do this", and the only thing running through my head is awe at how little otherwise smart people understand their fellow humans. Like...y'all have lost your goddamn minds. OF COURSE many humans would do it! Humans fucking drive straight into lakes because the map told them to. Humans do not have some kind of magical ward against fucking up or against being tricked. Quite the opposite. But you know what the difference is? As soon as the computer system has programmed in a record of local contextual defaults, this problem won't happen again. Say the same about humans, I dare you.
- MeltySmelty 7y ago
  this post is so HN bubble that it hurts
- sharkmerry 7y ago
  >> Humans fucking drive straight into lakes because the map told them to. I had to check if this was real or just a bit from the office that imprinted. Googling "humans drive into lake" only finds one incident. Which occured at midnight.[1] and if you look at the research, they didnt convert it into an 8, they simply extended the middle part of the 3 a little bit [2] [1]https://fox8.com/news/a-little-embarrassed-woman-follows-cars-gps-straight-into-lake-huron/ https://fox8.com/news/a-little-embarrassed-woman-follows-car... [2]https://cdn0.tnwcdn.com/wp-content/blogs.dir/1/files/2020/02/Screenshot-2020-02-19-at-11.59.56.png https://cdn0.tnwcdn.com/wp-content/blogs.dir/1/files/2020/02...
  - ebg13 7y ago
    > Googling "humans drive into lake" only finds one incident. You'd find more examples if you tried slightly harder ("lake" isn't the important part!): https://kfgo.com/2020/02/10/man-drives-into-mississippi-river-while-following-google-maps/983425/ https://kfgo.com/2020/02/10/man-drives-into-mississippi-rive... https://www.cnet.com/news/man-drives-into-river-gps-china/ https://www.cnet.com/news/man-drives-into-river-gps-china/ https://www.mirror.co.uk/news/world-news/man-watches-wife-burn-alive-5435575 https://www.mirror.co.uk/news/world-news/man-watches-wife-bu... https://abc13.com/587100/ https://abc13.com/587100/ https://abcnews.go.com/blogs/headlines/2012/03/gps-tracking-disaster-japanese-tourists-drive-straight-into-the-pacific/ https://abcnews.go.com/blogs/headlines/2012/03/gps-tracking-... http://news.bbc.co.uk/2/hi/uk_news/england/bradford/7962212.stm http://news.bbc.co.uk/2/hi/uk_news/england/bradford/7962212.... https://www.boston25news.com/news/man-drives-into-pond-while-following-gps-directions/538872511/ https://www.boston25news.com/news/man-drives-into-pond-while... https://www.nbcnewyork.com/news/local/gps-leads-nj-motorist-into-house/2123304/ https://www.nbcnewyork.com/news/local/gps-leads-nj-motorist-... https://www.dailymail.co.uk/news/article-1164705/BMW-left-teetering-100ft-cliff-edge-sat-nav-directs-driver-steep-footpath.html https://www.dailymail.co.uk/news/article-1164705/BMW-left-te... https://www.westsiderag.com/2013/05/01/gps-brain-fail-driver-car-ends-up-stuck-on-riverside-park-stairs https://www.westsiderag.com/2013/05/01/gps-brain-fail-driver... https://www.news.com.au/lifestyle/real-life/driver-follows-gps-into-sand/news-story/081ea557f486757a0cdd2722892727bb https://www.news.com.au/lifestyle/real-life/driver-follows-g... https://nymag.com/intelligencer/2018/01/waze-app-directs-driver-to-drive-car-into-lake-champlain.html https://nymag.com/intelligencer/2018/01/waze-app-directs-dri... https://www.fox5atlanta.com/news/2-drivers-stuck-on-train-tracks-in-duluth-after-gps-error-1-car-hit https://www.fox5atlanta.com/news/2-drivers-stuck-on-train-tr... https://www.bostonmagazine.com/news/2013/06/19/mbta-train-accident-car-tracks/ https://www.bostonmagazine.com/news/2013/06/19/mbta-train-ac... https://www.abc.net.au/news/2015-10-01/car-being-hit-by-train-after-gps-sends-onto-tracks/6818564 https://www.abc.net.au/news/2015-10-01/car-being-hit-by-trai... https://abc7chicago.com/395218/ https://abc7chicago.com/395218/ People act like humans are immune from making idiotic decisions or from ignoring their surroundings or something. That's...super naive.
threatofrain 7y ago
Don't cities have easily accessible and authoritative records on locations of important traffic controls? If so, that would make this problem a little bit easier.

71 points by harambae 7y ago

This is as good of a demonstration of the big "common sense" gap that end-to-end computer vision approaches have as it is of adversarial examples. Even with an occluded speed limit sign, humans have strong priors on what a sensible speed limit would be based on a more general understanding of the environment (width of lanes, curvature of roads, visibility, road quality, frequency of law enforcement vehicles ;), etc.).

castratikron 7y ago
Wonder if the neural net could be trained to know the speed limit? "This scene doesn't look like the usual 85 mph zone..."
- marrone12 7y ago
  You likely could train it to learn about trends and certain contexts. But then you still have to think and enumerate all of these separate possibilities.
- fyfy18 7y ago
  Many cars now come with a database of speed limits and display it on the dashboard. Although this shouldn't be relied on for self driving, it could be used to double check. OSM data for types of roads and areas is pretty accurate too, so you could use that ('an 85mph speed limit for a single lane within a residential area doesn't sound correct'). This doesn't need AI, it just needs someone with common sense to come up with a few basic rules about speed limits. If there is some doubt you could prompt the driver to confirm the speed limit change, and use that to build up a database of real world data. It sounds like the Tesla is just using what the camera sees - which is bad if true. Admittedly this was tested on a race track, so maybe there is no data saying otherwise (or even the opposite).
dr_hooo 7y ago
I don't think that this in particular is really a good example of this gap. It feels more like a sloppy implementation. Just like humans, an autopilot has multiple sources available for guessing what an appropriate speed would be. Simply including information from a map application should be enough to prevent these kind of mistakes/hacks. Now things like children playing on the side of the road certainly seem more difficult to handle..

dmitrygr 7y ago

Further proving that "self driving" is approximately as hard as AGI, and is nowhere near as "close" as everyone thinks to think it is. A human would have context and common sense and thus know that a residential street will simply not ever have an 85 mph speed limit. And yes, you could special case this case in the code, but there are hundreds of cases where "common sense" is used in driving, and you will never teach your NNs all of it. Prediction: no actual "self driving" (to a point where driver can legally be asleep) on public streets till > 10 years from now

joshvm 7y ago
I don't see why you need special cases in the code. Sat nav units have had speed limits built in for years. Why not just look it up in a database? (the car knows where it is after all, and requiring a GPS fix could be mandatory - most drones won't arm without it, for example)
- dmitrygr 7y ago
  Because those databases are out of date approximately 100% of the time... Road construction, for example is a reason why
  - joshvm 7y ago
    However if the sat nav gives you a prior that says 35 mph, and the road sign says 85 mph, do you trust it? Especially if the adjacent roads are also low speed, and it's clearly not a highway etc. This is perhaps a bad example as there are plenty of roads in the UK that transition from national to 30 in rural areas (and plenty of hamlets that you can bomb through). Roadworks are an obvious situation where you drive at a _lower_ speed. This would be a case where you could use computer vision to identity an obstruction, identify roadworks and then adjust speed. Those databases shouldn't be out of date. Most highway agencies (UK for sure) publish planned works and improvements weeks in advance. Google and Apple both show road closures for example. I belive speed changes are similar.
    - 7y ago

posix_compliant 7y ago

I'm not what you'd call a self-driving car "skeptic" since I do think that we will eventually be able to iron out these types of rules, but I think it will be another 30-50 years before I feel comfortable truly trusting one.

emiliobumachar 7y ago
How much do you "truly trust" human drivers?

vardump 7y ago

Another discussion on same topic: https://news.ycombinator.com/item?id=22370346 https://news.ycombinator.com/item?id=22370346

NicoJuicy 7y ago

I'm a self-driving skeptic, but this test is not fair. It's also illegal to do this, so it's not a valid edge-case.

dmitrygr 7y ago
Who cares? Would you feel better if you died due to someone breaking the law?
- NicoJuicy 7y ago
  It's nice as a research part. But more people die because of human error and drunk drivers every day. I have yet to see someone die because of the illegal and "labor intensive" situation they created in this "research". > Would you feel better if you died due to someone breaking the law? I can respond that back, but then with problems that are factual ( drinking and driving). Not fictional or harder to execute do it in practice ( this research)
  - kelnos 7y ago
    I think the issue isn't always entirely rational. Even if self-driving cars could have a better accident record than humans by several orders of magnitude, incidents would still cause humans to react poorly -- "if I were in control that wouldn't have happened" -- even if it would have. Altering a speed limit sign may not be legal, but if it's done in a way that wouldn't fool a human, but would fool a self-driving system, that's just a way that a self-driving system could be unsafe that a human driver wouldn't, and people will react negatively toward the self-driving system. That's just human nature, even if it's not logical. Regardless, we need to be aware of these possible "exploits" on self-driving systems. Sure, no one would bother altering a speed limit sign to cause crashes, because human drivers have context and common sense that would make that mostly ineffective. But if it were known that some popular self-driving systems could be fooled by that, I'd expect to see this happen more often, perpetrated by people who would probably otherwise try to hurt people in other ways. It's not a "new idea" in the context of driving in general, but it's a new -- and potentially very effective -- idea to dangerously mess with self-driving systems.

asdff 7y ago

Some sick people get their kicks tossing rocks off of overpasses. Imagine the games people are going to play with cardboard cutouts of whatever or false signage. I predict an arms race.

Robotbeat 7y ago
People could already cause crashes by making fake signs or repainting lines. I think this XKCD summarizes the issue nicely: https://m.xkcd.com/1958/ https://m.xkcd.com/1958/

oyebenny 7y ago

I live in Atlanta and there's a digital speed limit sign on 75. Sometimes the light is out and when it means to say the speed limit is "55" it just says "5". Come to think of it I've always noticed traffic slowing down nearing that sign. I wonder if any Tesla's have any contribution to that.

emiliobumachar 7y ago
I would slow down to a speed limit of 5, even if I was sure it was a defective display. It's effectively an unknown speed limit. Maybe not if I'm very familiar with the road, but not everyone is. Yes, I'm sure the courts should throw out the ticket, but sometimes they mess up, and often you lose time and money to pursue your case.

Robotbeat 7y ago

This already happens occasionally with regular drivers: https://www.telegraph.co.uk/news/uknews/road-and-rail-transport/11138390/Drivers-tricked-by-fake-40mph-speed-limit-signs.html https://www.telegraph.co.uk/news/uknews/road-and-rail-transp... "Pranksters blamed for speeding offences after drivers caught out by 40mph signs in 30mph zone" Ironically, current Teslas are largely immune to this as they use a database of speed limits.

Sohcahtoa82 7y ago

While scary, it's not exactly relevant to any Tesla made after they broke off from MobilEye (Which I think was 2016?). Current Teslas don't read speed limit signs, they access an online database. Vandalize the speed limit signs all you want, as modern Teslas aren't even reading them.

luxuryballs 7y ago

Shouldn’t it know that 85 is beyond the speed limit? You’d think it would know the laws based on where the GPS is, like in some states above 80 is reckless driving. Seems trivial to make this data available.

kwhitefoot 7y ago

Very misleading. The car will only accellerate to the speed already chosen by the driver. It doesn't seem odd for itself to accellerate up to speed limits

consp 7y ago

Nice for them use Tesla but I'm pretty sure you can do it with every sign reading auto throttle system. There are plenty of cruise control systems which employ similar technologies.

shiftpgdn 7y ago
Or a person who misreads the sign. Though the whole point is that a Tesla doesn't have the context to know that a residential zone with an 85mph speed limit sign is obviously some sort of error.
- toast0 7y ago
  You don't need residential context. You just need US context. I have never seen an 85 MPH speed limit sign in the US, ever. Up to 65 is common, with 70 or 75 sometimes on well maintained roads between urban areas. I can't recall seeing an 80, but I feel like maybe once or twice.
  - drunken-serval 7y ago
    There’s an 85mph in one place in Texas. https://en.m.wikipedia.org/wiki/Speed_limits_in_the_United_States https://en.m.wikipedia.org/wiki/Speed_limits_in_the_United_S...
  - pwinnski 7y ago
    Hello from Texas! There's a pretty long stretch of road between Austin and San Antonio that's 85 the whole way. It's not as big as the I-35, but the higher speed limit means it draws quite a bit of traffic for people traveling between the two cities. Point stands: 85 is ridiculous in anything like a residential area.
- bazeblackwood 7y ago
  Accelerating to 85mph on a street where that speed is unsafe is absolutely not what a person who misreads the sign would do. 35mph tend to be that way for a reason (residential, stop signs, curved streets)

deleted 7y ago

[deleted]

wpietri 7y ago

Ah, that's especially good. They just changed the 3 in "35 MPH" speed limit sign to look more like an 8, and the Teslas dutifully sped up. A mistake no human driver would make. Rodney Brooks points out that if we ever get AGI [1], we'll have solved the autonomous vehicle problem. But it's far from clear to me that we'll truly solve the problem much before then, as cars and roadways are built with GI expectations in mind. [1] https://en.wikipedia.org/wiki/Artificial_general_intelligence https://en.wikipedia.org/wiki/Artificial_general_intelligenc...

koboll 7y ago
>A mistake no human driver would make. And probably prosecutable as felony manslaughter, the same way removing a stop sign would be, because you're acting in a way you know will get people killed.
- draw_down 7y ago
  Some people like to attempt to prosecute security researchers for reporting vulnerabilities. Same energy here.
seanmcdirmid 7y ago
According to the article, only when they engage TACC for the first time while the sign is visible, not if they were driving for awhile and it was already turned on. Also note that TACC isn’t autopilot, one just controls your speed automatically, the other actually drives your car. It wasn’t clear if the flaw affected auto pilot as well, since it can’t be used for non freeway driving.

elcomet 7y ago

So basically they modified a 35MHP street sign to make it look like a 85MPH. I looked at it from a distance, and it was hard to tell between 35 and 85. So if it is hard for a human, then it seems obvious that a tesla would also fail to recognize a 35. Maybe the difference is that a human might not be certain and slow down just in case, whereas a deep neural network might be certain in its errors.

tmpz22 7y ago
A human has the basic intuition to assume the sign itself might be wrong or misleading, the programmers of the car automation failed to realize that in a way that might've gotten someone killed.
- russfink 7y ago
  Good thing, too, that nobody stepped out in front of their car, like a kid chasing a ball, as they did their ad hoc experiment.
  - frosted-flakes 7y ago
    They didn't allow the car to reach 85 m/h, for obvious reasons.
pathartl 7y ago
There's several factors they could use to bring a level of uncertainty to the algorithm. Compare against the previously clocked speed limit, compare against map data, compare against the speed of other vehicles in a proximity, judge based on density of crosswalks, parked cars, traffic, etc. This isn't a failure of autonomy imo. Speed limits are arbitrary placeholders that are more for supplementing the slow reaction speed of the human brain. If we started focusing more on making the algorithm implement actual analysis of the environment instead of "the rules of the road", we might see better results.
decebalus1 7y ago
> So if it is hard for a human, then it seems obvious that a tesla would also fail to recognize a 35. You need to be a special type of stupid to think that a road with a speed limit of 35 MPH is 85 because of a road sign. The 'drive into the lake because the GPS nav tells me to' level of stupid.

rootusrootus 7y ago

They had to use older Teslas to make this work, because newer models don't use sign-reading to determine speed limit. Abstractly, it's interesting, if only because it probably wouldn't trick a human -- everyone would see the 85mph as BS regardless of the defaced sign. Has anyone hacked into openstreetmaps yet and fiddled with speed limits?

ssheth 7y ago
Yes ... the older Teslas have the MobilEye chipsets which use this technology. Many other cars use the same MobilEye chipsets today .. so potentially could be an issue in other cars as well. Its likely now that this error was pointed out that Tesla may add some speed limit validation code to ensure a car doesn't speed up much when a speed limit jumps by more than 25% or something like that..
- seanmcdirmid 7y ago
  It only works when this is the first sign TACC sees. If TACC was previously engaged and saw other signs previously, the exploit doesn’t work. I’m sure Tesla fixed this, but given the nature of the edge case, it wasn’t simply a matter of sanity checking.
  - tjoff 7y ago
    Why would it only work if it is the first sign?
    - seanmcdirmid 7y ago
      Maybe because they were probably already doing sanity checking, but the kind that relied on some previous context? That seems reasonable, I’m sure they’ve thought about this kind of attack before, missing an obscure edge case.
chris11 7y ago
It still is a little bit unnerving that older models can accelerate based off of speed limit signs alone. That seems like a dangerous approach.

valine 7y ago

Putting aside the fact that this was found on an old Tesla with autopilot 1.0, this is not even a technically hard problem to solve. The current version of autopilot uses a database and gps to determine the speed limit. In the future if Tesla wants to return to a vision based speed limit system they can sanity check the vision readings against their database and throw out results like 85mph.

aero142 7y ago

If the point is that autonomous cars are succeptable to deliberate malicious actors, I've got bad news about the rest of the world. Human drivers are incredibly succeptable to me throwing a $2 brick throught the front windshield as well. It sounds like the issue is already resolved in this case.

capableweb 7y ago
I guess the difference here is the scope of how many you can impact. In order to get all Teslas who go past a sign on the highway, you just have to modify the sign slightly and everyone is affected (in theory). In order to affect the same amount of people with your "brick through the windshield" strategy, you'll need a lot more manpower than just a sticker on a sign.
- justaguyhere 7y ago
  do you really need the manpower though? it shouldn't be that hard to build a mechanical device that can do the brick throwing?
  - aabeshou 7y ago
    you cant seriously be arguing its as easy or invisible to set up a brick throwing machine as it is to put a piece of tape on a speed limit sign?
- throwanem 7y ago
  That depends on the size of wreck you manage to cause, don't you think? In any case, it takes no more people to drop a brick than it does to sticker a sign. To be clear, I find this entire concept of unsupervised robot vehicles both dangerous and absurd - even more so than the already dangerous and absurd baseline of a society so intimately bound up with automotive travel as ours. Exceeding such a high baseline as that is in its way impressive, and certainly demonstrates the astonishing overconfidence rampant in some segments of our very young and rather careless industry, but let's not get distracted from that essential point and waste our efforts on inconsequential arguments over whether a sticker is more dangerous than a brick.

skymt 7y ago

XKCD on tricking self-driving cars: https://xkcd.com/1958/ https://xkcd.com/1958/ I wanted to link that not because of the "there's an XKCD for everything" meme but because it makes an interesting point: sabotaging roads isn't difficult now, with human drivers. There's no reason to assume it would suddenly become a substantial threat when autonomous cars are commonly adopted. An issue worth considering and accounting for, but not worth public worry.

sebringj 7y ago

What I like about Tesla is they have over the air updates using a fleet of network data to quickly address issues or exploits that may arise... but this article is quite funny exposing the gap between AI and basic human intuition. I do believe human intuition may just be another model to train from in some sense in the end.

RcouF1uZ4gsC 7y ago

>The safety of Tesla's autopilot features has come under close scrutiny, but CEO Elon Musk has predicted the company will have "feature-complete full self-driving" this year. I think that is a real possibility if we are referring to years on Jupiter (which has an orbital period of 12 Earth years).

m0zg 7y ago

Sounds like an easy fix in most cases. Residential road or road work? Can't be 85, period. There are relatively few roads in the US where 85 is the actual speed limit.

whyaduck 7y ago

So cars are using OCR on speed limit signs? Next step: sql injection. https://hackaday.com/2014/04/04/sql-injection-fools-speed-traps-and-clears-your-record/ https://hackaday.com/2014/04/04/sql-injection-fools-speed-tr...

masklinn 7y ago
That's pretty common IME though the ones I've driven didn't make decisions based on that. Last few cars I've rented had a speed limit sign generally matching the speed limit on the dash, and on the one I rented this christmas passing a speed limit sign (or an other sign triggering a speed limit change) while on limiter or cruise control would flash a message suggesting pressing a button twice to change the configured speed to that (rather than manually adjust using the +/- buttons). They were mediums (C-segment) and compact MPV not large or luxury, which I expect is why the sign reading was mostly to purely indicative.
bdcravens 7y ago
Yes, many cars do now. Some of the "less smart" cars don't necessarily know which signs are which. My 2018 Accord thought the speed limit was 100mph when I was on Highway 99.

JMTQp8lwXL 7y ago

If a human drove at 85 mph because a bad actor maliciously modified the speed limit sign, would they be liable?

binarymax 7y ago
Co-reference resolution error: who is 'they' in your sentence? The human or the bad actor? No matter what, the answer is probably both. You can't modify street signs now anyway. You also can't drive 85 in residential/city areas.
- JMTQp8lwXL 7y ago
  The human operating the vehicle who reads a maliciously edited sign. It seems unquestionable to me anybody modifying street signs would be held liable for the modifications.

tjoff 7y ago

Even if it was a legitimate 85 mph sign a human would realize that the road isn't safe at 85 mph speeds. You must react to the environment as well...

ebg13 7y ago

I keep seeing throughout this thread statements similar to "a human would never do this", and the only thing running through my head is awe at how little otherwise smart people understand their fellow humans. Like...y'all have lost your goddamn minds. OF COURSE many humans would do it! Humans fucking drive straight into lakes because the map told them to. Humans do not have some kind of magical ward against fucking up or against being tricked. Quite the opposite. But you know what the difference is? As soon as the computer system has programmed in a record of local contextual defaults, this problem won't happen again. Say the same about humans, I dare you.

MeltySmelty 7y ago
this post is so HN bubble that it hurts
sharkmerry 7y ago
>> Humans fucking drive straight into lakes because the map told them to. I had to check if this was real or just a bit from the office that imprinted. Googling "humans drive into lake" only finds one incident. Which occured at midnight.[1] and if you look at the research, they didnt convert it into an 8, they simply extended the middle part of the 3 a little bit [2] [1]https://fox8.com/news/a-little-embarrassed-woman-follows-cars-gps-straight-into-lake-huron/ https://fox8.com/news/a-little-embarrassed-woman-follows-car... [2]https://cdn0.tnwcdn.com/wp-content/blogs.dir/1/files/2020/02/Screenshot-2020-02-19-at-11.59.56.png https://cdn0.tnwcdn.com/wp-content/blogs.dir/1/files/2020/02...
- ebg13 7y ago
  > Googling "humans drive into lake" only finds one incident. You'd find more examples if you tried slightly harder ("lake" isn't the important part!): https://kfgo.com/2020/02/10/man-drives-into-mississippi-river-while-following-google-maps/983425/ https://kfgo.com/2020/02/10/man-drives-into-mississippi-rive... https://www.cnet.com/news/man-drives-into-river-gps-china/ https://www.cnet.com/news/man-drives-into-river-gps-china/ https://www.mirror.co.uk/news/world-news/man-watches-wife-burn-alive-5435575 https://www.mirror.co.uk/news/world-news/man-watches-wife-bu... https://abc13.com/587100/ https://abc13.com/587100/ https://abcnews.go.com/blogs/headlines/2012/03/gps-tracking-disaster-japanese-tourists-drive-straight-into-the-pacific/ https://abcnews.go.com/blogs/headlines/2012/03/gps-tracking-... http://news.bbc.co.uk/2/hi/uk_news/england/bradford/7962212.stm http://news.bbc.co.uk/2/hi/uk_news/england/bradford/7962212.... https://www.boston25news.com/news/man-drives-into-pond-while-following-gps-directions/538872511/ https://www.boston25news.com/news/man-drives-into-pond-while... https://www.nbcnewyork.com/news/local/gps-leads-nj-motorist-into-house/2123304/ https://www.nbcnewyork.com/news/local/gps-leads-nj-motorist-... https://www.dailymail.co.uk/news/article-1164705/BMW-left-teetering-100ft-cliff-edge-sat-nav-directs-driver-steep-footpath.html https://www.dailymail.co.uk/news/article-1164705/BMW-left-te... https://www.westsiderag.com/2013/05/01/gps-brain-fail-driver-car-ends-up-stuck-on-riverside-park-stairs https://www.westsiderag.com/2013/05/01/gps-brain-fail-driver... https://www.news.com.au/lifestyle/real-life/driver-follows-gps-into-sand/news-story/081ea557f486757a0cdd2722892727bb https://www.news.com.au/lifestyle/real-life/driver-follows-g... https://nymag.com/intelligencer/2018/01/waze-app-directs-driver-to-drive-car-into-lake-champlain.html https://nymag.com/intelligencer/2018/01/waze-app-directs-dri... https://www.fox5atlanta.com/news/2-drivers-stuck-on-train-tracks-in-duluth-after-gps-error-1-car-hit https://www.fox5atlanta.com/news/2-drivers-stuck-on-train-tr... https://www.bostonmagazine.com/news/2013/06/19/mbta-train-accident-car-tracks/ https://www.bostonmagazine.com/news/2013/06/19/mbta-train-ac... https://www.abc.net.au/news/2015-10-01/car-being-hit-by-train-after-gps-sends-onto-tracks/6818564 https://www.abc.net.au/news/2015-10-01/car-being-hit-by-trai... https://abc7chicago.com/395218/ https://abc7chicago.com/395218/ People act like humans are immune from making idiotic decisions or from ignoring their surroundings or something. That's...super naive.

threatofrain 7y ago

Don't cities have easily accessible and authoritative records on locations of important traffic controls? If so, that would make this problem a little bit easier.