6 ms·

Further proving that "self driving" is approximately as hard as AGI, and is nowhere near as "close" as everyone thinks to think it is. A human would have contex

Further proving that "self driving" is approximately as hard as AGI, and is nowhere near as "close" as everyone thinks to think it is. A human would have context and common sense and thus know that a residential street will simply not ever have an 85 mph speed limit. And yes, you could special case this case in the code, but there are hundreds of cases where "common sense" is used in driving, and you will never teach your NNs all of it. Prediction: no actual "self driving" (to a point where driver can legally be asleep) on public streets till > 10 years from now

joshvm 7y ago
I don't see why you need special cases in the code. Sat nav units have had speed limits built in for years. Why not just look it up in a database? (the car knows where it is after all, and requiring a GPS fix could be mandatory - most drones won't arm without it, for example)
- dmitrygr 7y ago
  Because those databases are out of date approximately 100% of the time... Road construction, for example is a reason why
  - joshvm 7y ago
    However if the sat nav gives you a prior that says 35 mph, and the road sign says 85 mph, do you trust it? Especially if the adjacent roads are also low speed, and it's clearly not a highway etc. This is perhaps a bad example as there are plenty of roads in the UK that transition from national to 30 in rural areas (and plenty of hamlets that you can bomb through). Roadworks are an obvious situation where you drive at a _lower_ speed. This would be a case where you could use computer vision to identity an obstruction, identify roadworks and then adjust speed. Those databases shouldn't be out of date. Most highway agencies (UK for sure) publish planned works and improvements weeks in advance. Google and Apple both show road closures for example. I belive speed changes are similar.
    - dmitrygr 7y ago
      Google maps keep saying 101N in San carlos is 65mph. It's been 55mph due to construction for 5 months... This is literally Google's back yard
      - t-writescode 7y ago
        65 in a 55 is much less dangerous than 85 in a 35. It wouldn’t be common that road conditions would permit very high speeds while at the same time having an artificially lowered speed limit (e.g construction). This shouldn’t be a problem, in general, with a couple extra sanity checks. I would expect the correct solution to more commonly have the opposite problem: too slow in a faster zone.
      - asterisk_ 7y ago
        Even so, what's the worst-case scenario in biasing towards lower speed and assisting your vision system with a knowledge base? You perform the task of vision 100% of the time driving. From a purely probabilistic standpoint, even assuming highly accurate ML, the relatively infrequent database updates still make for a sensible prior. This is somewhat adjacent to what Gary Marcus has been arguing, and I think it makes a lot of sense; there seems to be no compelling reason to rely exclusively on primitives (i.e. vision) when good priors are easily accessible.
  - kelnos 7y ago
    Out of date is probably not that big a deal. If a speed limit is lowered from 40 to 35, or even 65 to 55, and the database doesn't get updated, it's unlikely to create a dangerous situation if a car doesn't know about it. But a car thinking a 35mph zone is an 85mph zone... that could be a very big deal indeed. Regardless, if self-driving cars (or at least "computer assisted driving" cars) were to be widely deployed enough and shown to depend on these sorts of databases, I'd expect it to become a legal requirement to make sure these databases are kept up to date by city/county/state agencies, and that car systems get those updates. Otherwise, expensive liability ensues.
    - lostlogin 7y ago
      It gets complicated fast. Currently if there is some sort of emergency or requirement to notify road users of a change to driving conditions, a person can just put a sign up. It mostly just works. If road workers are also required to update a database that needs to be read by a car, that’s a lot of infrastructure that needs to be working. You also need to have the humans and machines reading the signs as the same at every time point, as “road closed” or “one way” needs to be correctly understood by everyone at the same time. In an emergency that may involve bad weather or some other sort of disaster, a remote database is not going to work very well. The car needs to read the actual sign properly.
- melling 7y ago
  Waze knows the speed limit and warns you when you exceed it.
  - Piskvorrr 7y ago
    Except when it "knows" wrong, or the limit is conditional (days, hours, vehicle classes), etc.
- rypskar 7y ago
  How much can you trust that data? My car tell me what the speed limit is by using a database and reading signs. On my commute to work there is a part where it tell me the speed limit is 5km/h when it actually is 80km/h and another place where it for a short bit shows the limit as 80km/h where the correct is 30km/h
  - leokennis 7y ago
    In The Netherlansds (and probably other places), many roads have a different maximum speed between 07:00-19:00 and 19:00-07:00.
    - t0mas88 7y ago
      Both Netherlands and Germany have these time based speeds on a lot of highways where the special case is in text underneath the sign. BMW handles it if the navigation database is up to date. At least on the A2 in the Netherlands, which has the 06:00 to 19:00 changes between 100 and 130. It also understands temporary changes for roadworks, but gets it wrong when a roadwork/temporary sign has an exception like "when road is wet" That tells me the camera only understands the main sign and then the navigation database tells it about exceptions / special cases. Nice as a small reminder on your screen, but I wouldn't trust it to self drive based on that data.
  - joshvm 7y ago
    Neither is ideal. But you should use what information you have. There is still no excuse for making a decision like 85 in a 35. Worst case this should be a disengage or the car could defer to you - eg visual warning on the dash "speed limit unclear, override?" In the UK smart motorways can vary from 30mph to 70mph according to traffic. This can change within minutes and obviously cannot be in databases. So this is a good case for sign recognition. In theory though, there is some network that's updating the speed limits and there's no technical reason why that couldn't be a public information service (maybe it already is). Then you have the sibling comments - speed limits which are time based. I would imagine this is country specific and is, mostly, a matter of public record. But would you trust CV to recognise text on a sign with specific hours? I don't know how complex the signage is, but trying to use ML for this seems absurdly wasteful of resources. Finally you have other information - how fast are the cars around you travelling? Have you detected an obstruction on the road (eg cones)? Is the weather inclement and therefore you should drive more slowly, etc. I would imaging that if self driving cars become ubiquitous there will eventually be a system for real time speed limit determination, whether that's some kind of wireless beacon at regular intervals or an online database that's kept up to date I don't know.
- Robotbeat 7y ago
  Tesla already looks it up in a database. Current Teslas CANNOT have this error simply because MobilEye patented reading speed limit signs and Tesla no longer uses the MobilEye system and haven't since mid-2016.
  - bsaul 7y ago
    side remark : how can you patent a "function" ? Wouldn't mobileeye have to patent a certain way of reading speed limit ?
    - brian_herman__ 7y ago
      It is called a process patent.
      - iqtidar 7y ago
        Thanks for the enlightenment!
kelnos 7y ago
I think "> 10" is even still pretty optimistic. I'd be surprised and impressed to see driver-asleep self-driving in the next 25 years, unless we see some serious unexpected breakthroughs in computing.
abhorrence 7y ago
Do the NNs actually need to learn this? I’ve always assumed that NNs primarily being used to do computer vision, but that all the driving decisions would essentially be a fancy rule system. Is this not how most self-driving works?
Rapzid 7y ago
When the implications of Adversarial AI clicked, I knew the autonomous vehicle thrust as we were lead to believe it was over. No way, no how.
yarg 7y ago
The ability to accurately map surroundings is a fundamental prerequisite to AGI but it's pretty much the whole meal for self driving. From there the car needs to segment its interpreted reality and run path finding algorithms. AGI needs to take that reality and seemingly consciously understand the many different ways that it can interact with it. I'm oversimplifying the post mapping implementation of self driving - but not so as to remove an insurmountable problem. Tesla's biggest fuck ups (such as driving under a truck) seem to me to be the consequences of initial mistakes in building a map of reality, rather than in what's being done with that map. For self driving, as far as I can tell, the map of reality is the hard part. For AGI, it gets you to the hard part.
hurricanetc 7y ago
It only failed on older gen1 MobilEye hardware. So this particular vision problem has been solved for a long time.
- Robotbeat 7y ago
  Not only that, but current Teslas don't (and can't, due to MobilEye's patents) rely on vision for speed limits.
Robotbeat 7y ago
Is no one pointing out that this is the older Tesla Model S that doesn't even use Tesla's autopilot? It's MobilEye. MobilEye patented reading speed limit signs, so Tesla's current autopilot doesn't even do this. People saying "see, THIS PROVES self-driving is as hard as AGI" without even applying any critical thinking is just confirmation bias. You can tell the Model S is old because it has the pre-April-2016 bubble nose. I believe all vehicles with the bubble nose have the old MobilEye system at best (which was never intended for self-driving).
- legolas2412 7y ago
  Him: self driving is more than a decade away. You: No, because this attack is for an old vehicle (implying that newer vehicles are able to evade it). You see, the new cars don't have this feature at all! Cool, so the new cars have even less features than before. And that justifies that self driving is near?
  - abc_lisper 7y ago
    Not really, he is saying that argument is not strong enough to say anything about immediacy of self driving.
- clouddrover 7y ago
  > I believe all vehicles with the bubble nose have the old MobilEye system at best (which was never intended for self-driving). If the MobilEye system is not trustworthy then why is the vehicle accelerating based on the data it receives from it? Why doesn't Tesla update the software to stop doing that?
Accujack 7y ago
It's probably closer than you think. The reason we don't already have self driving cars is that there hasn't been enough computing power in a small package to make the sort of decision logic needed practical until recently. As long as the prerequisite for self driving has been to emulate human's driving skills, there just hasn't been the computing power available. Now that there is, progress is being made, but really it's just a transitional period. What's eventually going to happen is that roads everywhere will get machine assists - basically, small beacons running on low power that provide easy machine readable data on everything from precise road location to speed limits to local road conditions. They'll be small, low powered radio transmitters or even NFC devices near the road that the car can just download data from. There's always been a chicken and egg problem with self driving vehicles, where no one wanted to invest in machine assists for all roads because no one wanted to put that much money and effort into something that no one needed (no one had self driving cars) but no one wanted to make self driving cars a product that wouldn't be successful without machine assists as well as a way to drive without them as a backup. Now that self driving cars are out there and everyone is getting comfortable with the concept, machine assists will be coming, and they're going to have benefits to ordinary drivers, too. If you want an example of why this will work, look at aircraft. Most jet liners today are perfectly capable of landing themselves in almost any weather because of the electronic navigation system they have built in to most airports. The planes aren't using machine vision and AI code, they're landing on a piece of ground defined by numbers in the navigation system which is precisely located by GPS coordinates. The same will work for cars, it just hasn't been built out yet.
- blacklion 7y ago
  Did you ever try to play GeoGuessr game? I'm not living in USA, I'm living in country which is infamous for its notoriously bad roads (you could say that it is 3rd world country and I will not blame you), so I will not use my homeland as example. But when I'm playing GeoGuessr in USA I often found "myself" on small road with not-so-good tarmac, with wore-out road markings, without any light posts… Google car was here (or there will be no street view), but still. Are you sure "roads everywhere will get machine assists" soon? Who will pay for that, if now nobody pays to fix tarmac and road markings, not to say about lights installation, which will help EVERYBODY RIGHT NOW? Airports are completely irrelevant example. There are ~2500 IATA recognized airports out of ~50K "airfields" total, and only ~1000 of these 2500 are equipped with ILS. It is not comparable with millions of kilometers of roads around the planet, even if we will count only "1st world countries".
MarkMc 7y ago
> Prediction: no actual "self driving" (to a point where driver can legally be asleep) on public streets till > 10 years from now We already have "driver-asleep self driving" on public streets: https://www.theverge.com/2019/12/9/21000085/waymo-fully-driverless-car-self-driving-ride-hail-service-phoenix-arizona https://www.theverge.com/2019/12/9/21000085/waymo-fully-driv...
emiliobumachar 7y ago
Your predictions might well be true, but this hack is not any evidence for it. So you can attack an AI driver with targeted sign changes which wouldn't work with human drivers. I don't think you'll need experimental evidence to agree that you could also attack human drivers with targeted sign changes which wouldn't work with a particular AI driver. Relevant xkcd: https://m.xkcd.com/1958/ https://m.xkcd.com/1958/. An AI driver will remain dumber than a human until we're very close to AGI, if not forever. But it never gets distracted, and always tries its best to follow the rules. This may well be enough to compensate for its low intelligence - and beyond. The worst human drivers ain't always the lowest IQ ones.

joshvm 7y ago

I don't see why you need special cases in the code. Sat nav units have had speed limits built in for years. Why not just look it up in a database? (the car knows where it is after all, and requiring a GPS fix could be mandatory - most drones won't arm without it, for example)

dmitrygr 7y ago
Because those databases are out of date approximately 100% of the time... Road construction, for example is a reason why
- joshvm 7y ago
  However if the sat nav gives you a prior that says 35 mph, and the road sign says 85 mph, do you trust it? Especially if the adjacent roads are also low speed, and it's clearly not a highway etc. This is perhaps a bad example as there are plenty of roads in the UK that transition from national to 30 in rural areas (and plenty of hamlets that you can bomb through). Roadworks are an obvious situation where you drive at a _lower_ speed. This would be a case where you could use computer vision to identity an obstruction, identify roadworks and then adjust speed. Those databases shouldn't be out of date. Most highway agencies (UK for sure) publish planned works and improvements weeks in advance. Google and Apple both show road closures for example. I belive speed changes are similar.
  - dmitrygr 7y ago
    Google maps keep saying 101N in San carlos is 65mph. It's been 55mph due to construction for 5 months... This is literally Google's back yard
    - t-writescode 7y ago
      65 in a 55 is much less dangerous than 85 in a 35. It wouldn’t be common that road conditions would permit very high speeds while at the same time having an artificially lowered speed limit (e.g construction). This shouldn’t be a problem, in general, with a couple extra sanity checks. I would expect the correct solution to more commonly have the opposite problem: too slow in a faster zone.
    - asterisk_ 7y ago
      Even so, what's the worst-case scenario in biasing towards lower speed and assisting your vision system with a knowledge base? You perform the task of vision 100% of the time driving. From a purely probabilistic standpoint, even assuming highly accurate ML, the relatively infrequent database updates still make for a sensible prior. This is somewhat adjacent to what Gary Marcus has been arguing, and I think it makes a lot of sense; there seems to be no compelling reason to rely exclusively on primitives (i.e. vision) when good priors are easily accessible.
- kelnos 7y ago
  Out of date is probably not that big a deal. If a speed limit is lowered from 40 to 35, or even 65 to 55, and the database doesn't get updated, it's unlikely to create a dangerous situation if a car doesn't know about it. But a car thinking a 35mph zone is an 85mph zone... that could be a very big deal indeed. Regardless, if self-driving cars (or at least "computer assisted driving" cars) were to be widely deployed enough and shown to depend on these sorts of databases, I'd expect it to become a legal requirement to make sure these databases are kept up to date by city/county/state agencies, and that car systems get those updates. Otherwise, expensive liability ensues.
  - lostlogin 7y ago
    It gets complicated fast. Currently if there is some sort of emergency or requirement to notify road users of a change to driving conditions, a person can just put a sign up. It mostly just works. If road workers are also required to update a database that needs to be read by a car, that’s a lot of infrastructure that needs to be working. You also need to have the humans and machines reading the signs as the same at every time point, as “road closed” or “one way” needs to be correctly understood by everyone at the same time. In an emergency that may involve bad weather or some other sort of disaster, a remote database is not going to work very well. The car needs to read the actual sign properly.
melling 7y ago
Waze knows the speed limit and warns you when you exceed it.
- Piskvorrr 7y ago
  Except when it "knows" wrong, or the limit is conditional (days, hours, vehicle classes), etc.
rypskar 7y ago
How much can you trust that data? My car tell me what the speed limit is by using a database and reading signs. On my commute to work there is a part where it tell me the speed limit is 5km/h when it actually is 80km/h and another place where it for a short bit shows the limit as 80km/h where the correct is 30km/h
- leokennis 7y ago
  In The Netherlansds (and probably other places), many roads have a different maximum speed between 07:00-19:00 and 19:00-07:00.
  - t0mas88 7y ago
    Both Netherlands and Germany have these time based speeds on a lot of highways where the special case is in text underneath the sign. BMW handles it if the navigation database is up to date. At least on the A2 in the Netherlands, which has the 06:00 to 19:00 changes between 100 and 130. It also understands temporary changes for roadworks, but gets it wrong when a roadwork/temporary sign has an exception like "when road is wet" That tells me the camera only understands the main sign and then the navigation database tells it about exceptions / special cases. Nice as a small reminder on your screen, but I wouldn't trust it to self drive based on that data.
- joshvm 7y ago
  Neither is ideal. But you should use what information you have. There is still no excuse for making a decision like 85 in a 35. Worst case this should be a disengage or the car could defer to you - eg visual warning on the dash "speed limit unclear, override?" In the UK smart motorways can vary from 30mph to 70mph according to traffic. This can change within minutes and obviously cannot be in databases. So this is a good case for sign recognition. In theory though, there is some network that's updating the speed limits and there's no technical reason why that couldn't be a public information service (maybe it already is). Then you have the sibling comments - speed limits which are time based. I would imagine this is country specific and is, mostly, a matter of public record. But would you trust CV to recognise text on a sign with specific hours? I don't know how complex the signage is, but trying to use ML for this seems absurdly wasteful of resources. Finally you have other information - how fast are the cars around you travelling? Have you detected an obstruction on the road (eg cones)? Is the weather inclement and therefore you should drive more slowly, etc. I would imaging that if self driving cars become ubiquitous there will eventually be a system for real time speed limit determination, whether that's some kind of wireless beacon at regular intervals or an online database that's kept up to date I don't know.
Robotbeat 7y ago
Tesla already looks it up in a database. Current Teslas CANNOT have this error simply because MobilEye patented reading speed limit signs and Tesla no longer uses the MobilEye system and haven't since mid-2016.
- bsaul 7y ago
  side remark : how can you patent a "function" ? Wouldn't mobileeye have to patent a certain way of reading speed limit ?
  - brian_herman__ 7y ago
    It is called a process patent.
    - iqtidar 7y ago
      Thanks for the enlightenment!

kelnos 7y ago

I think "> 10" is even still pretty optimistic. I'd be surprised and impressed to see driver-asleep self-driving in the next 25 years, unless we see some serious unexpected breakthroughs in computing.

abhorrence 7y ago

Do the NNs actually need to learn this? I’ve always assumed that NNs primarily being used to do computer vision, but that all the driving decisions would essentially be a fancy rule system. Is this not how most self-driving works?

Rapzid 7y ago

When the implications of Adversarial AI clicked, I knew the autonomous vehicle thrust as we were lead to believe it was over. No way, no how.

yarg 7y ago

The ability to accurately map surroundings is a fundamental prerequisite to AGI but it's pretty much the whole meal for self driving. From there the car needs to segment its interpreted reality and run path finding algorithms. AGI needs to take that reality and seemingly consciously understand the many different ways that it can interact with it. I'm oversimplifying the post mapping implementation of self driving - but not so as to remove an insurmountable problem. Tesla's biggest fuck ups (such as driving under a truck) seem to me to be the consequences of initial mistakes in building a map of reality, rather than in what's being done with that map. For self driving, as far as I can tell, the map of reality is the hard part. For AGI, it gets you to the hard part.

hurricanetc 7y ago

It only failed on older gen1 MobilEye hardware. So this particular vision problem has been solved for a long time.

Robotbeat 7y ago
Not only that, but current Teslas don't (and can't, due to MobilEye's patents) rely on vision for speed limits.

Robotbeat 7y ago

Is no one pointing out that this is the older Tesla Model S that doesn't even use Tesla's autopilot? It's MobilEye. MobilEye patented reading speed limit signs, so Tesla's current autopilot doesn't even do this. People saying "see, THIS PROVES self-driving is as hard as AGI" without even applying any critical thinking is just confirmation bias. You can tell the Model S is old because it has the pre-April-2016 bubble nose. I believe all vehicles with the bubble nose have the old MobilEye system at best (which was never intended for self-driving).

legolas2412 7y ago
Him: self driving is more than a decade away. You: No, because this attack is for an old vehicle (implying that newer vehicles are able to evade it). You see, the new cars don't have this feature at all! Cool, so the new cars have even less features than before. And that justifies that self driving is near?
- abc_lisper 7y ago
  Not really, he is saying that argument is not strong enough to say anything about immediacy of self driving.
clouddrover 7y ago
> I believe all vehicles with the bubble nose have the old MobilEye system at best (which was never intended for self-driving). If the MobilEye system is not trustworthy then why is the vehicle accelerating based on the data it receives from it? Why doesn't Tesla update the software to stop doing that?

Accujack 7y ago

It's probably closer than you think. The reason we don't already have self driving cars is that there hasn't been enough computing power in a small package to make the sort of decision logic needed practical until recently. As long as the prerequisite for self driving has been to emulate human's driving skills, there just hasn't been the computing power available. Now that there is, progress is being made, but really it's just a transitional period. What's eventually going to happen is that roads everywhere will get machine assists - basically, small beacons running on low power that provide easy machine readable data on everything from precise road location to speed limits to local road conditions. They'll be small, low powered radio transmitters or even NFC devices near the road that the car can just download data from. There's always been a chicken and egg problem with self driving vehicles, where no one wanted to invest in machine assists for all roads because no one wanted to put that much money and effort into something that no one needed (no one had self driving cars) but no one wanted to make self driving cars a product that wouldn't be successful without machine assists as well as a way to drive without them as a backup. Now that self driving cars are out there and everyone is getting comfortable with the concept, machine assists will be coming, and they're going to have benefits to ordinary drivers, too. If you want an example of why this will work, look at aircraft. Most jet liners today are perfectly capable of landing themselves in almost any weather because of the electronic navigation system they have built in to most airports. The planes aren't using machine vision and AI code, they're landing on a piece of ground defined by numbers in the navigation system which is precisely located by GPS coordinates. The same will work for cars, it just hasn't been built out yet.

blacklion 7y ago
Did you ever try to play GeoGuessr game? I'm not living in USA, I'm living in country which is infamous for its notoriously bad roads (you could say that it is 3rd world country and I will not blame you), so I will not use my homeland as example. But when I'm playing GeoGuessr in USA I often found "myself" on small road with not-so-good tarmac, with wore-out road markings, without any light posts… Google car was here (or there will be no street view), but still. Are you sure "roads everywhere will get machine assists" soon? Who will pay for that, if now nobody pays to fix tarmac and road markings, not to say about lights installation, which will help EVERYBODY RIGHT NOW? Airports are completely irrelevant example. There are ~2500 IATA recognized airports out of ~50K "airfields" total, and only ~1000 of these 2500 are equipped with ILS. It is not comparable with millions of kilometers of roads around the planet, even if we will count only "1st world countries".

MarkMc 7y ago

> Prediction: no actual "self driving" (to a point where driver can legally be asleep) on public streets till > 10 years from now We already have "driver-asleep self driving" on public streets: https://www.theverge.com/2019/12/9/21000085/waymo-fully-driverless-car-self-driving-ride-hail-service-phoenix-arizona https://www.theverge.com/2019/12/9/21000085/waymo-fully-driv...

emiliobumachar 7y ago

Your predictions might well be true, but this hack is not any evidence for it. So you can attack an AI driver with targeted sign changes which wouldn't work with human drivers. I don't think you'll need experimental evidence to agree that you could also attack human drivers with targeted sign changes which wouldn't work with a particular AI driver. Relevant xkcd: https://m.xkcd.com/1958/ https://m.xkcd.com/1958/. An AI driver will remain dumber than a human until we're very close to AGI, if not forever. But it never gets distracted, and always tries its best to follow the rules. This may well be enough to compensate for its low intelligence - and beyond. The worst human drivers ain't always the lowest IQ ones.