14 ms·
“It's a truck full of traffic lights”
- tppiotrowski 5y agoWho could've ever predicted this scenario? A great example of something that's trivial for human beings to understand but where ML training sets will come up short.
- matt-attack 5y agoThat’s what’s frightening about this. I feel not enough people internalize just how primitive the car’s understanding/perception of the world is. A two year old human brain would have no confusion in this scenario. We discount how much the concept of “understanding” is required in visual perception. We don’t just see shapes we have a complete understanding of what we’re seeing. I might see a big rectangle flopping in the lane in front of me. I can immediately ascertain whether it’s a piece of tumbling plywood, or foam based on movement characteristics, color, apparently size, etc. I can then use that understanding to decide what evasive actions are required. A Tesla it seems has absolutely zero of this capability.
- hobofan 5y agoI'm very curious how a LIDAR based system would have faired here. It very much looks like problem that would only appear with a limited vision-only system like Tesla has.
- deleted 5y ago[deleted]
- TheAdamAndChe 5y agoOne time, I fell asleep as a passenger in a car. I hit a bump, looked up, and screamed as I saw what looked like a car coming right for me. It was a tow truck towing a car backwards. It was just enough in my half-asleep state to scare the shit out of me. Humans have millions of years of evolution behind our visual processing systems. We have developed hacks that prevent our brain from getting tricked by unusual situations like traffic lights on trucks or backwards cars being towed. We only developed the first computers a hundred years ago, and only in the last 40 years have a small subset of people started learning about visual processing systems. It's easy to look at this video and scoff because of how trivial it seems. But it's instead a marvel of our minds that we can pick up on context clues so quickly and accurately that such oddities basically never puzzle us. Given the pace of our innovation, it wouldn't surprise me if our computer systems match ours within a few human generations at latest.
- Syonyk 5y ago> It was a tow truck towing a car backwards. It was just enough in my half-asleep state to scare the shit out of me. And you're sure that it was a bump, and not the driver pulling up rather close behind it and then giving a sharp tap of the brakes to wake you up? ;)
- belltaco 5y ago>One time, I fell asleep in a car. I hit a bump, looked up, and screamed as I saw what looked like a car coming right for me. >It was a tow truck towing a car backwards. It was just enough in my half-asleep state to scare the shit out of me. There are some funny(?) youtube videos of people in the passenger seat waking up to this with tractor trailers being towed in reverse.
- formerly_proven 5y ago> Given the pace of our innovation, it wouldn't surprise me if our computer systems match ours within a few human generations at latest. This is not a FLOPS problem. Moore's law can't save you here.
- a3n 5y agoThat is effing hilarious. Except we're trusting ML to perform surgery, choose conviction sentencing, evaluate job CVs, determine acceptable marriage partners (why not?), determine who can have kids (why not?), determine who gets into college (why not?), determine who gets a loan (why not?), determine who gets to work on ML (why not?). And drive cars.
- david_allison 5y agoNot if the GDPR has anything to say about it > The data subject shall have the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her. https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:32016R0679&from=EN#d1e2838-1-1 https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CEL...
- a3n 5y agoAh. Sounds great. But unfortunately I live in the US, the consumer's paradise. Question on your snippet though. It says "subject to a decision based solely on automated processing ..." Since I do live in the consumer's paradise I'm naturally suspicious. I wonder if people will sometimes be subject to decisions based solely on automated processing, and if they don't bother to ask for a human in the loop after the automated decision, then that's that. And how would you even know, one way or the other?
- visarga 5y agoWe still use electricity even though it's deadly and it accidentally kills people every year. Why not use ML even if it has it's own issues? As humans who know better we're responsible to take all the precautions.
- a3n 5y agoIt's a fair question. One difference is that electricity is not making decisions or suggestions on the fly. Another example is roads, where the design of the road allows one to turn into an opposing vehicle. The road itself is not deciding to turn your car into a semi trailer because it looks like sky, but an ML driven car has, multiple times. I don't even own a Tesla, and I resent having to participate in its beta program. ML in a car is quite different from a brake recall, or even a stuck accelerator software recall.
- mikewarot 5y agoTesla clearly needs to hire a "Red Team" to find weaknesses of their autopilot (and other systems). The odd thing is that even a randomly stupid AI for self driving is statistically safer than most drivers. Clearly there is a ton of room for improvement in both AI and Humans.
- Closi 5y ago> The odd thing is that even a randomly stupid AI for self driving is statistically safer than most drivers. This hasn’t been shown yet at all. Statistics showing autopilot have less crashes per mile always ignore that Autopilot is doing the type of driving that has the least accidents per mile (motorway driving).
- bigdict 5y ago> Red Team I'm sorry, did you mean customers?
- falcolas 5y agoHe didn't. Telsa did.
- ModernMech 5y agoWhy hire a team to beta test your AI self driving car when you have customers willing to shell out $10k for the privilege?
- Syonyk 5y ago> The odd thing is that even a randomly stupid AI for self driving is statistically safer than most drivers. A strong claim, lacking actual evidence for it. All we have to go on are some Elon tweets (rather the definition of a biased source) and the actual crash rate. Without a lot more data (which Tesla steadfastly refuses to release) about environments, corrections, etc, it's quite impossible to make that sort of statement with any confidence. The Tesla hardware is a weird combination of capable and insanely dumb, and it's far from obvious which it will be in any given situation until it's gone through it. If an honest statistical analysis of the data indicated that Tesla's automation was better than human drivers (or better than other driver assist systems), I would fully expect them to have released the values. Since they haven't, and only hint at it and make statements that sound statistical but really aren't, I assume they've done the numbers internally and know it's not nearly as good as they like to imply. If I drove in a city like their "self driving" beta was a few months back, I would be hauled from the car on suspicions of driving while hammered.
- akomtu 5y agoUsing Tesla's "autopilot" isnt very different than working as an unpaid driving instructor to keep an eye on the "autopilot" trainee. To make things worse, the trainee is tripping on mushrooms and sees nonexistent cars appearing from nowhere, trees morphing into pedestrians and pedestrians morphing into traffic lights. The trainee also has a problem with epilepsy and youre expected to take control on a second notice. Edit: also, the trainee has cognitive ability of a mentally challenged frog.
- ed25519FUUU 5y agoI actually don’t want autopilot for the fast, freeway speed driving. That’s easy. I want it for the mundane, exhausting stop-and-go, bumper to bumper traffic as we all slowly creep through the traffic.
- asdff 5y agoThere are already cruise control systems that do this for you
- deleted 5y ago[deleted]
- ed25519FUUU 5y agoThey don’t typically work below 25 MPH, and definitely don’t stop and go and keep you within the lane.
- etaioinshrdlu 5y agoAs an ML engineer I'm a little bit baffled that Tesla has not solved this by now. It's not like they lack data or ML knowledge. It seems like they should have a million hard test cases that must pass in simulation before releasing a new model. The simulations should be harder and more extreme than anything encountered in real life. I think the real problem is obvious. They're trying to rush the work because Elon said so.
- AceJohnny2 5y agoYears ago, a friend worked in the Autopilot group. It took them a year to procure servers to store the telemetry of the existing cars, and then weeks to have them setup. They don't work there anymore. From their experience, I know one thing: I will never work for Elon Musk. He may be a great visionary and salesman, but he's a horrible manager.
- etaioinshrdlu 5y agoThis is fascinating, but what caused the delays specifically?
- AceJohnny2 5y agoServer procurement was a CFO thing, and their specific requirements didn't fit under the existing buckets, so it took them a long time to get it approved. It was stunning to see the complete disconnect between Musk's grand declarations and what the organization was actually setup to deliver. Frankly, it just gives me more respect for Tim Cook, who as COO at Apple made his company able to turnaround and deliver HW in record time. Edit: in retrospect I wonder if Musk's grand public declarations were actually a way to control and pressure his own organization. Remember, Musk didn't actually found Tesla, he rescued it from bankruptcy after the Roadster didn't return as much as needed, so he inherited an existing structure.
- systemvoltage 5y agoJust like your anecdote, I have one to share as well: I know a close friend that works on space lasers at SpaceX and has a blast, best job ever according to him and he thinks Elon is an excellent manager mostly because there is zero bureaucracy and people are not afraid of "Do nothing" option as well as removing complexity. In fact, he wouldn't work anywhere else after seeing the company culture. I know it's cool to hate Elon.
- blhack 5y agoIs the car trying to stop? The visualization here is for autopilot (which ignores traffic lights), so even if the truck driver was trying to do something malicious, the car would ignore it.
- detaro 5y agoAt least in the clip it's accelerating, so it doesn't seem to take them into account - which would make sense for non-lit traffic lights anyways?
- nucleardog 5y agoI'd wager the average case of a non-lit traffic light is more likely to be a traffic light that's... out (at least around here newly installed ones are covered until they're activated) so no, I wouldn't say ignoring them would make sense. It would generally be a clue that there's an intersection busy enough to require signals that now lacks signals or signage which would warrant extra caution. I'd expect the car to at least slow down significantly, if not come to a complete stop before proceeding.
- Sohcahtoa82 5y agoAutopilot is not engaged during this video. You can tell because the steering wheel icon is grey, meaning AP is available. If it was engaged, it would be blue. The driver is driving manually.
- codeulike 5y agoThe car is not under autopilot, the driver is driving. The car is just displaying what it thinks it can see. (the grey steering wheel icon means autopilot is not engaged, it would be blue if it was on)
- schmorptron 5y agoI don't think autopilot is actually enabled in this clip, the steering wheel icon is greyed out.
- nullc 5y agoI saw a good recent youtube video showing numerous concerning FSD failures in urban driving: https://www.youtube.com/watch?v=antLneVlxcs https://www.youtube.com/watch?v=antLneVlxcs
- dawkins 5y agoI saw the video and it is crazy that they think it is even close to production.
- tibbydudeza 5y agoClearly the probability of it happening is rather low unless you live close to the local municipality road maintenance depot. But it clearly means another edge case like detecting deer (wonder if they can handle our local Kudu) that they need to deal with. They might as well develop AGI at this rate.
- mortenjorck 5y agoThat the Autopilot world model is extrapolating these lights to follow the passing geography in ~500ms cycles, similar to how game netcode sends players running in their last known direction when encountering packet loss, is an interesting insight into how the system works.
- deleted 5y ago[deleted]
- lifeisstillgood 5y agoI am fairly confident that the autonomous car business model is pretty dead. Electric cars - that's great ! But self driving is waaaay harder. And self driving has a inbuilt expectation of nearly perfect. It's not that if we swapped to self driving cars today we would halve car deaths (which would be something like 500k people pa - which would be a huge very important thing) but it would not be perceived as robots save 500k people a year, it would be instead "robots kill the other 500k per year" Liability, insurance, legal minefields and plain old marketing would never allow cars ,that perform as well as the best performing cars do today, to be on the roads in "every day" conditions. My conjecture is we end up building AV only roads. initially one lane of a highway, then ring roads round cities and major warehousing hubs, then across urban areas. Walled off in some way they simply become railways with benefits. At that point every business model ever written with Self Driving in the title goes in the bin. I am not saying the tech is useless - frankly it's fucking awesome that this is happening in my lifetime. But fucking awesome tech and workable business model aren't always the same thing. Not sure where I am going with the rant but I sure hope we get more out of the billions spent here than Teflon.
- hakfoo 5y agoI still have hope, because we have two different mentalities at play. The individual still thinks in terms of "I'm an above-average driver, I'm better than a silly bleep-bloop box." They'll resist self-driving en masse if it's not 1000% perfect. Insurers, however, will see self-drivers as predictable, never drunk, always following the speed limit, and in aggregate preferrable to humans. They will be able to see "3% better" as meaningful even if it's not perfect, and price accordingly. Eventually, you'll pay a rapidly increasing penalty rate for having a steering wheel. This is where the avalanche comes from. I also figure self-driving could unlock new features-- in particular I'm imagining the only way we get road speeds much over 120kph is self-driving, as the human response time becomes a limiting factor. And of course the in-dashboard wet bar. This will feed back into the consumer market even if it's only closed markets at first: why can't I go 250kph on the highways considering the self-driving airport shuttle does?
- failwhaleshark 5y agoNope. You're just not patient enough. ACs will come, but very gradually. They're already (mostly) here but it will take another decade or two to be completely A.
- sfblah 5y agoI recently test drove a Tesla just to see how the autopilot system works. The way they handle traffic lights is pretty entertaining. It seems them, particularly yellow lights, in a lot of situations that are truly perplexing. It also had a tendency to turn trees into traffic cones and to be truly impressive at detecting garbage cans. On some level it's hard to understand exactly what they're trying to do there. I also remember being at an intersection where I was turning left and was waiting behind another car. The display repeatedly showed cars the cars crossing in front of us crashing into the car in front of me. Not sure why.
- Sohcahtoa82 5y agoWhen I got my Tesla back in November 2019, when stopped at traffic lights, cars next to me would be shown to rotate randomly and jiggle and dance. It works much better now, but it was always kind of amusing. It only did it when we were all stopped. Moving cars (whether it was them moving or me) were rendered just fine. My guess is that it was extrapolating movement based on more than one "frame" of sensor data, but at a stop there's no changes beyond noise, and the noise was being extrapolated into extreme movements. I have seen it show cars "crashing" into me, but it's only semis when in the neighboring lane.
- codeulike 5y agoIn this video, I dont think the car is actually in autopilot. Its just displaying what it thinks it can see, while the driver drives. (the steering wheel icon at the top of the screen is grey, not blue)
- failwhaleshark 5y agoDid they get to level 17? I hear you get an extra car.
- dylan604 5y agoHopefully, an image of this truck never shows up in a captcha challenge.
- cs702 5y agoWell, of course. This is probably the first time the car has come across a truck carrying multiple street lights, without cover, stacked on the back of the truck, well above eye level. It's a rather unusual edge case. It's only a matter of time before the software in these cars can handle the vast majority of edge cases as well as or better than human beings. Human vision isn't exactly reliable.[a] In the meantime, someone should make a playable game in which trucks throw street lights at cars. Maybe someone at Tesla is willing to make this game in good jest? [a] See, for example http://www.ritsumei.ac.jp/~akitaoka/index-e.html http://www.ritsumei.ac.jp/~akitaoka/index-e.html
- visarga 5y agoI bet carrying streetlights in a truck exposed like that is going to become illegal. Also painting bikes on trucks.
- tsjq 5y agogood one. :)
- des1nderlase 5y agoKnowing what i know about ML, it's the problem if we are rellying on cars to see unusual cases on the streets for the first time and handle them well.
- ajross 5y agoThis genre of posts is so tiresome. As the second video in the thread demonstrates, the truck is literally hauling traffic lights. The AI recognition is correct, the only thing worth complaining about is that they're displayed as static objects for the user after recognition, just to be re-recognized a few seconds later in a different place. Note that the car is correctly not detecting they are lit, so not inferring direction (though AP isn't engaged, so I guess we'll never know what it would have done). No doubt you could play the same game by putting a traffic cone on your bike. The car wants to see important traffic objects, it's literally what it's trained for.
- theamk 5y agoAnd that illustrates the important problem with self-driving car: if you want L5 autonomy, you need to be handle all the weird cases.
- ajross 5y agoThe car handled this just fine, though. (Though obviously AP wasn't engaged so we don't know exactly what it would have done). The bug report here is that the correctly-detected signs, in the correctly-detected state (i.e. off) were presented to the user in a confusing way in the UI visualization. UI bugs aren't blockers for mythical SAE Level 5 Autonomy. You just... fix them.
- theamk 5y agoI am not sure why you think it was UI blocker and not actual problem affecting the driving? In many places, the road rules say that if you see a broken traffic light, you should treat it as a stop sign. So a theoretical L5 car should have stopped there, in the middle of the road -- and again, and again. Seems pretty bad to me. And yes, I agree that you "just.. fix them" -- but of the difficulties of the real road driving is that the number of unique situations like that is very large, and many of them would not happen during test drives / development.
- 5y ago
- tigerBL00D 5y agoVery interesting. There must be some kind of assumption in models that traffic lights are stationary objects which results in them falling off the back of the truck.
- jschwartzi 5y agoI mean it makes sense. If you've never seen traffic lights be delivered to a new intersection you'd probably blithely assume you'll never encounter a stack of them that are unlit and moving. Somewhere at Tesla there's a junior engineer who's telling a senior engineer "I told you so!"
- deleted 5y ago[deleted]
- darepublic 5y agoMy God, it's full of stars
- samatman 5y agoThis is a great example of how Tesla's strategy to build out using real-life training data can pay off. Tesla now has data about what it looks like to drive behind a truck transporting traffic lights! No team in the world would solve this problem, in advance, in simulation. Not saying their strategy is going to work, not saying it's an unbeatable advantage, but: just look at it! This is a compelling demonstration.
- gameswithgo 5y agowho isn’t using that strategy? openpilot uses it.
- adflux 5y agoA demonstration of why it will be a long time before it'll be rendered safe enough to drive on its own
- deleted 5y ago[deleted]
- offsky 5y agoIf one of the lights in the back of that truck had illuminated red, would the car have put on the breaks in the middle of the road?
- visarga 5y agoLong tail ... you can never solve all the absurd cases.
- brokenmachine 5y agoImagine all the hilarity that can ensue when they release this beta stuff in production... oh wait. But seriously, these are just slightly funny obscure weird cases, imagine when the hackers start coming up with malicious cases to mess with these cars.