31 ms·

Tesla's self driving algorithm's overlay [video]

• dylan604 7y ago
  I have often wondered about this, so I do find this interesting. Of all of the info presented, one question I have is how does the AI decide when to go at a 4-way stop? In real life, I'm am constantly amazed at how confusing a 4-way stop is to humans. Not that I ever had any doubt into how complicated real-time video analysis could be, this just makes my appreciation of the complexity of the problem that much more qualified.
  • wenc 7y ago
    Yes. In theory, 4 way stops follow some simple rules -- whoever gets there first and makes a full stop goes first, and if more than one party makes a full stop at the same time, the party on the right goes first. In practice we live in a world of (very illegal) rolling stops, people beckoning at each other to go first, people inching forward, etc. That's not even considering some very stupidly designed four-way stops here in Illinois with two lanes on each side -- how the heck is one supposed to even figure these things out? It ends up being a game of chicken.
    • yeahboats 7y ago
      Near where I live we've got two lanes + turn lanes on all 4 sides. It is a nightmare both as a driver and as a pedestrian. I'm not sure how that gets put in place or gets addressed as it inevitably gets busier. Traffic circles perhaps?
      • wenc 7y ago
        Traffic circles work better, but most people in North America have trouble with 2-lane traffic circles. (they're common in most places around the world)
        • toast0 7y ago
          Most people in north america have trouble with 1-lame traffic circles, because they're almost uniformly too small. Traffic engineers like to wedge them in as retrofits, when there's already a lot of stuff built up around an intersection, and it's not feasible to enlarge the intersection to make it work well. Traffic signals with good sensors are more space efficient, and usually more time efficient when there's congestion, and more pedestrian friendly. Proper sensor design means no or little waiting during low traffic times too. Traffic circles can help with akward intersections though, where there's more than 4 entrances or exits, and some three way intersections work well too.
    • aeternum 7y ago
      Even the theory is not that clean cut. What if two cars arrive simultaneously at opposite sides of the 4-way and are both turning left? This could easily deadlock two self-driving cars if not considered.
      • rimliu 7y ago
        Your situation is a bit ambiguous. Do you mean they turn to their respective left (in that case their trajectories should not intersect), or to the same left? Not sure about USA, but where I am from the right of the way belongs to the driver who turns to the same side as the side of the road they are on, i.e. to his right (we drive on the right side).
  • jrockway 7y ago
    Google had to program their car to stop treating 4 way stops exactly as the law required, and instead pretend to be a little aggressive: https://www.nytimes.com/2015/09/02/technology/personaltech/google-says-its-not-the-driverless-cars-fault-its-other-drivers.html https://www.nytimes.com/2015/09/02/technology/personaltech/g...
    • winternett 7y ago
      Unless you're driving in a place known for courtesy like Canada or Hawaii ಠ_ಠ... People often stop late and won't move until you go first...
      • rrauenza 7y ago
        My father taught me this trick -- if you're going to arrive simultaneously, just make sure you stop clearly second (especially if the other car is on your right). I find it is usually pretty efficient and have yet to notice someone doing the same to me at the same time.
        • abstractbarista 7y ago
          Hey I really like that! I often feel awkward at these where it is ambiguous who among 3 people actually stopped first. Not to mention people who roll right through without stopping (when other cars are present.)
        • Slartie 7y ago
          I do exactly the same - not with 4-way stops as we don't have those in Germany, but with areas where a two-lane road is intentionally narrowed to one lane for a very short piece of road, which we sometimes have in order to slow down traffic around pedestrian crossings. If I see someone approaching from the other side and we are about to arrive roughly at the same time, I slow down just enough so the other person hopefully realizes she can go first. If I see the other person doing the same I add light flashing, which is typically understood as "You go first". But it is rarely necessary, because if I start going slow early enough, the other person realizes he/she is a tiny bit closer to the narrow part, so they slightly speed up in order to ensure they are first and thus get the right to come through before me, which is exactly the reaction that I want, because my interest usually isn't to get through first, but to be able to roll through the narrow part without coming to a full stop. When I was road tripping in the US, I also attempted to prevent unclear situations at 4-way stops by intentionally slowing down just a bit so it is clear that I'm not first to drive. Since I couldn't prevent the stop anyway I was usually much more comfortable with letting other people drive first, or at least with having a clear order and knowing about my position in that order.
  • emilecantin 7y ago
    The algorithm I apply is actually pretty simple: First at the intersection goes first. So you, individually, should let everyone who was there when you arrived go before you, but you should go before everyone who arrived after you. It gets messy when people try to be polite or don't understand the rules correctly, but otherwise it's pretty easy. It might depend on your jurisdiction, though.
    • shaftway 7y ago
      My algorithm is so much more complicated than yours. In my area you get rear-ended if you were going straight and you didn't go when someone coming from the opposite direction but also going straight went (your algorithm doesn't allow that). Assuming cars are already at the intersection, I wait until each of them goes and then take my turn, unless it's someone's turn who wouldn't cause a conflict with me and nobody else could go at the same time and would cause a conflict with me. When I arrive at the intersection at nearly the exact same time as someone else then some more complicated heuristics come into play (I'm not sure how you handle it when the timing is close enough to be in question). The person on a significantly larger road has priority over smaller road. People going straight have priority over right turners, which have priority over left turners, and if all else is equal, the right-most person should go first. I guess I could sum it up as "if you can go safely, and nobody who was there before you can go safely, then go. Otherwise wait your turn", but with a lot more nuance around tie-breaking.
      • emilecantin 7y ago
        Yeah, I forgot about the going straight part, but I actually do pretty much the same thing as you. I guess one additional trick I use is to intentionally avoid conflicts: if it looks like I'm going to arrive at the same time as another car, I slow down a bit before arriving at the stop, then clearly stop AFTER they did, so it's clear that it's their turn to go. It works pretty well.
      • erikpukinskis 7y ago
        People in your area get rear ended while stopped at a 4-way stop?
  • winternett 7y ago
    The system isn't AI based... It's currently based on static programming. This is why a development team would have to think of every possible scenario that could happen on roads to make self driving cars really work. You'd have to ask all kinds of wild questions too like: "What if kids played a prank and re-painted road lanes on a highway at 50MPH?" "What if a sand storm happened?" "What would the car do if a tire blows out at 55MPH?" "Could the car detect a Tsunami is coming and abandon a trip assignment?" And that's just a tiny tiny sample of scenarios necessary. Some AI elements may exist within the network, but I'm pretty sure they are only running on the back end in terms of maps. The computing power to really permit the car to run autonomously, and to "adapt and learn" about driving only comes with experience and it also takes a lot more computing power and storage than your home computer can handle, much less the components in Tesla cars. Launching Self Driving Cars so close to their infancy is pretty reckless if you ask me. People are being paid off and buried online by paid staff online to keep quiet about all the accidents and issues with them. To make self riding cars reliable at this point in time, they pretty much need perfect circumstances, and they need to operate on fixed, predictable, and well maintained paths... Kind of like a train. They should really work on this technology in mass transit, shipping, big industry and on planes before pushing out thousands of smaller vehicles with it. There is plenty of proof that development teams aren't and may never be flawlessness enough to make error free updates, if you don't believe that, then just ask Boeing.
    • Erlich_Bachman 7y ago
      > than your home computer can handle, much less the components in Tesla cars. Ehm a recent Tesla has very powerful specialized hardware, developed in-house, which is much more powerful than a commodity home PC with expensive GPUs. You might want to do some basic research before you make such claims... > The system isn't AI based... It's currently based on static programming A large neural network is used to make many of the decisions, as well as to run the perception of the sensors/cameras. Unclear what you mean by "AI", but neural networks are usually covered in the "AI" term by researchers and engineers in this field. > may never be flawlessness enough to make error free updates It seems that you are new in this argument, because this point has been rebutted so many times in the past, but here goes: it does not have to be flawless, it just has to statistically cause fewer accidents per kilometer/year than what human drivers cause in a similar situation (which is already non-zero, and in some countries actually quite high).
  • calebegg 7y ago
    The proper rules[1] of a 4 way stop are probably pretty straightforward to encode, and probably easier for a computer to apply than a human, but the real issue is that you can't trust the other drivers (or AIs) to understand/follow them consistently. So the interesting part is how these kinds of systems can almost instantly react when another driver starts to go, and let them go. In NYC, where I live now, the de facto rule seems to be if you hesitate at all then the other person just goes, rules be damned. In the rural south, where I'm from, there's a lot of "no no, you go first" waving/gesturing/light flashing, which I'm curious if/how a self driving car would handle. (Do we need to give the self driving cars hands to gesture with?!) 1. https://en.wikipedia.org/wiki/All-way_stop#Operation https://en.wikipedia.org/wiki/All-way_stop#Operation
    • tamersalama 7y ago
      Wouldn't the data decide what the rule is? I'm guessing if in NYC, a more aggressive driving style is the norm - the models and the pipeline should be capable of adjusting to that.
      • runj__ 7y ago
        Adjusted for New York driving the AI evens flings up the doors randomly to smack down cyclists!
        • deleted 7y ago
          [deleted]
      • spookthesunset 7y ago
        > Wouldn't the data decide what the rule is? Lawyers will define what the rules are for a manufacturers self driving cars. After all, any accident at a 4-way stop will immediately result in blaming and then suing the manufacturer. And what are they gonna have for a defense? "Sorry, your honor, we use machine learning to decide how our car drives--as a result there is no real way to truly understand how our car will react to every situation it encounters".
    • 7y ago
  • notmyfuture 7y ago
    As a casual observer of self-driving, I feel like this is a really interesting part of the overall problem space. Until such time as a vehicle fleet is 100% autonomous, human piloted & autonomous vehicles need to co-exist. Humans bend & break driving rules by social interactions which autonomous vehicles don't currently participate in.
  • veeralpatel979 7y ago
    I've wondered why cars don't have a built-in communication mechanism. I should be able to broadcast somehow (using a voice message?) to nearby cars that "I'm going now, don't go".
    • Symmetry 7y ago
      That works if every car has the same communication mechanism and is honest and nobody interferes and everybody's semantic understanding of the messages is the same.
    • w_t_payne 7y ago
      Vehicle-to-X (A generalisation of vehicle-to-vehicle communication) https://en.wikipedia.org/wiki/Vehicle-to-everything https://en.wikipedia.org/wiki/Vehicle-to-everything
    • outworlder 7y ago
      As long as it is there just to improve safety. But it should not be relied on. In other words, if you 'hear' "don't go", you don't go. But if you don't hear anything, it still doesn't mean it's safe to go, so you'll have to rely on something else. And if you are doing that anyway, is the added complexity worth it?
      • V-2 7y ago
        It certainly will be, once the ocean is boiled = as soon as self-driving cars begin to comprise a significant percentage of cars on the road. This, in the long run, is a selling point that seems to be missing from most discussions in this comment section. I can't communicate "telepathically" with fellow human drivers.
    • winternett 7y ago
      We tried it, but people kept on using it only to say the most innovative curse words at each other... Hah. :P
    • GoToRO 7y ago
      The system exists but nobody is buying it. Probably because if your car has it first, there will be no other cars to talk to. https://www.google.com/search?client=firefox-b-d&q=car+v2x https://www.google.com/search?client=firefox-b-d&q=car+v2x
  • agumonkey 7y ago
    There's a research dedicated to ambiguous decision in swarms.
  • triska 7y ago
    There is an interesting and important theorem from control theory that is relevant for this situation. In a paper from 1984, Leslie Lamport has called it Buridan's Principle and phrased it as follows: “A discrete decision based upon an input having a continuous range of values cannot be made within a bounded length of time.” The paper is available from: http://lamport.azurewebsites.net/pubs/buridan.pdf http://lamport.azurewebsites.net/pubs/buridan.pdf It shows that under very general assumptions, a decision cannot be made in bounded time, leading to starvation (metaphorically and also literally). An example is a 4-way stop with other cars arriving at various times. The paper's history is also very interesting: https://lamport.azurewebsites.net/pubs/pubs.html#buridan https://lamport.azurewebsites.net/pubs/pubs.html#buridan
    • marnett 7y ago
      thanks for sharing this - this is fascinating. Admittedly I have never been exposed to Control Theory before, so I see a rabbit hole in my future.
      • triska 7y ago
        Essentials of Control Techniques and Theory by John Billingsley is a book with many superb examples.
    • gowld 7y ago
      Note that Lamport admitted that his result was never accepted by the scientific community. > continuous (The paper implicitly assumes non-discrete continuity) Differential-equation based hysical models are continuous. Computational systems are discrete, as long as the clock is slower than the . Lamport surely knows this, the transition duration. He waves away the empirical disproof of his claim by arguing that reality is merely a finite approximation where low probabilities runs down to zero. He even mentions this stuff in the paper, making the paper quite weird. My only guess is that he fell into the trap of forgetting that infinitesimal objects ate idealized mathematical models not physical realities.
  • 0xff00ffee 7y ago
    I've been driving for 40 years. I still hate 4-way stops. In the late 70's in Connecticut, I was taught the first person to arrive goes first, followed by the person to their right and so on. But there are so many holes with that heuristic itself! And putting aside guilt and anxieties of humans that prevent them from being assertive or committing--it's just a cluster fuck.* However, I have noticed that people are becoming more comfortable with zipper merging in major cities on both coasts, so if we could arrive at some consistent 4-way stop rule that's 1) succinct, and 2) easy to remember, I have hope. :) [* I'm referring to San Fran and the South Bay... but still better than my experiences driving through Italy and in Bengaluru!]
    • dheera 7y ago
      In California 4-way stops are often never marked "4-way" like in some other places. Whenever you come to a stop sign you have to inspect the crossing road to see whether or not it also has a stop sign. Often you stop and wait several seconds for a crossing car, only to realize that they also have a stop sign.
    • 72deluxe 7y ago
      Sorry for the dumb question but what is a 4-way stop? In the UK we have roundabouts (but many drivers still have no clue which lane to get into, and don't migrate to the lane on the left after passing an exit, like they should).
• willvarfar 7y ago
  Which sensor has primacy? radar, lidar or camera? I recall it mentioned somewhere that, after a well publicized tragedy hitting the side of a white truck on a bright day, Tesla was moving to radar? And yet a lot of the cues this awesome video shows seems to be camera-based line detection etc?
  • jrockway 7y ago
    If you look at the overlay, the algorithm seems extra paranoid about the following aspects: HIGH_BEAM, BLINDED, RAINING, TIRE_SPRAY, which all sound like camera weaknesses, not lidar/radar weaknesses. My guess is that this is camera-based still. Also interesting is the vision fps of ~13. That's 75ms latency.
    • yathern 7y ago
      75ms sounds like a lot, but it isn't so bad - average human reaction time is about 200ms. Source: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4456887/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4456887/
  • pixelHD 7y ago
    IIRC tesla still doesn't use LIDAR. Just wanted to point that out.
    • gandhi00 7y ago
      Correct. I believe they use vision along with front facing radar only
  • deleted 7y ago
    [deleted]
  • modeless 7y ago
    Tesla doesn't use lidar. They have always used radar, but radar can't always tell the difference between a stationary metal sign near/over the road and a stalled car in the middle of the lane, so for stationary objects they need the cameras. This is why every adaptive cruise control system (not just Tesla's) has a warning that it may not detect stationary objects. https://arstechnica.com/cars/2018/06/why-emergency-braking-systems-sometimes-hit-parked-cars-and-lane-dividers/ https://arstechnica.com/cars/2018/06/why-emergency-braking-s...
• agumonkey 7y ago
  Brings back so many memories of 80s shows.
• ArtWomb 7y ago
  I believe this was released as part of Tesla's Autopilot hiring announcement. More details here: https://www.tesla.com/autopilotAI https://www.tesla.com/autopilotAI Definitely someone's dream job ;) I particularly like the applicant query: "Tell us, what extraordinary work you have done?"
  • Traster 7y ago
    Two issues, first: I thought fully autonomous driving was meant to be done by now? Second, don't Tesla & SpaceX have a reputation for being a terrible place to work, with Musk expecting everyone to be working as hard (or harder) than him, and firing people in weird and capricious ways.
    • anchpop 7y ago
      > I thought fully autonomous driving was meant to be done by now? It was, but software often goes over-schedule, especially in a brand new field.
      • FireBeyond 7y ago
        The prudent thing to do there, would be to exercise prudence. Not announce to investors and the public ever more grandiose targets after each miss. As it is we're on about Autonomous Driving Promise Mk IV from Musk, "fully autonomous, coast to coast, by the end of 2020". (That's after 2018 was missed. Which was after 2016 was missed.)
        • optimiz3 7y ago
          > Not announce to investors and the public ever more grandiose targets after each miss. Those poor investors are really taking it on the chin today because of those overly optimistic software projections!
          • mrich 7y ago
            A massive short-squeeze/bitcoin-like retail bubble is certainly not indicative of the strength of the company. Look at the last earnings report, it wasn't impressive for a growth company.
• akira2501 7y ago
  I like these videos, but I'd really like to see one in snowy conditions, or other less-than-ideal road scenarios. I think there's a lot more to observe and learn from in those conditions than in bright, sunny days, in well maintained streets.
  • latchkey 7y ago
    I'd love to see a Tesla on auto pilot on a Vietnamese 'freeway'... buffalo crossing street while 3 people are driving directly at you on your side of the road, in the wrong direction, while 8 people are crossing the street laterally from both sides (2 of them are drunk), while one person is merging without looking and also dodging a wheel eating pothole and a truck driven by a meth addict, is about to rear end you.
    • kn0where 7y ago
      Actually, sounds like it wouldn’t be tough for a Tesla to drive safer than the human drivers on that road ;)
      • lolc 7y ago
        Trivially true since car driving must always include the option of refusing to drive out of safety concerns.
• dillonmckay 7y ago
  Is RESTRICTED the fourth prime directive?
• throwaway55554 7y ago
  Awesome! Now do it again and when it passes one set of parked cars on the side, have someone roll a soccer ball in front of the Tesla. Will the car assume a child will follow from in between the cars or not?
  • jedberg 7y ago
    > Will the car assume a child will follow from in between the cars or not? Will most human drivers?
    • throwaway55554 7y ago
      Should we not expect both to?
      • jedberg 7y ago
        I would expect both to, but for me the baseline is "as good as the average human". So in that vein, not predicting a child after a ball is still better than the baseline.
    • IceSentry 7y ago
      Most will assume this, the correct question would be: Would most human react fast enough?
      • jedberg 7y ago
        > Most will assume this, I've been driving a long time. I rarely see people hit their brakes when a ball comes out, or even slow down. My anecdotal evidence from 300,000 miles of driving says this isn't the case.
  • damon_c 7y ago
    A neural network needs to be taught this and then all cars with that training will respond correctly forever. Every individual human would need to learn this separately from experience, which would require far more soccer balls (and children).
• yarg 7y ago
  Interesting, but is it just me or does it seem overly jumpy?
  • jfengel 7y ago
    The "vision fps" runs around 12 to 20, so yeah, that's gonna seem kinda jumpy. The display fps is slightly higher, which contributes to a juddering feeling.
    • Erlich_Bachman 7y ago
      I think the more interesting/dangerous jumping is when the perception engine fails to consistently classify/find objects in the scene, between adjacent frames, not the specific fps of the process. This video seems to display some of such inter-frame jumpiness.
      • mitchellgoffpc 7y ago
        Inter-frame jumpy-ness can be smoothed out slightly by the control algorithms, on the assumption of object permanence, as long as the underlying vision NN doesn’t miss things too often.
        • deleted 7y ago
          [deleted]
  • Traster 7y ago
    One of the real challenges of these systems is that even a very good system only needs to lose confidence in some object for a very short period of time for the system to lose confidence in its navigation and return control to the user. The easy way to get around that issue is to categorize objects and then just ignore them out of the context you expect them to appear in -- which is how Tesla managed to kill a pedestrian walking their bicycle across the road.
    • anchpop 7y ago
• thedance 7y ago
  I wonder why they would let the public see this. To the critical eye it looks like it barely works. It loses its mind right at the beginning, around 0:06-0:07, veers to the right for no reason and then rolls through a stop sign without stopping. It spends the rest of the video hunting left and right like a drunk.
  • IceSentry 7y ago
    The video is clearly sped up and it looks like it did wait at the stop sign in the first part, it just didn't wait long.
    • Groxx 7y ago
      yea, there's a big "stop" icon at the top, and it goes down to 2mph (afaict not zero). a rolling stop in any case.
  • torpfactory 7y ago
    Sorta looks like its avoiding a big puddle in the road to the left of the vehicle. Notice the 'wed road' number goes up fast when it's going through that section.
  • judge2020 7y ago
    Some have speculated that it's a human driving and not autopilot (plus autopilot still has a warning that it's only meant for highways).
    • FireBeyond 7y ago
      > plus autopilot still has a warning that it's only meant for highways And Summon has a warning that you need to "pay full attention to the vehicle". Unless you look at their marketing, in which case it also tells you that you car will come "as you are being distracted by a fussy child".
  • Zeetah 7y ago
    > It loses its mind right at the beginning, around 0:06-0:07, veers to the right for no reason I'm speculating but I think the reasons it veers to the right are some combination of the following: 1) There is water on the left hand side of the lane. I was going back and forth on the video and you can see "Raining" along the left edge turn green. So, it has detected the water and is avoiding it? At the same time it has turned green "Wet road". 2) There is a vehicle on the right side of the lane - it has been identified with an upside down green triangle / crosshairs and a square (is that block the license plate?) and the Tesla is "following" it. Again speculating based on watching the display change. I found this clip amazing for the following reasons: a) Around 0:02, it starts to notice the right edge of the road. Along the right edge of the video, you can see the estimated road. Around 0:04, you can see the right turn lane and the dots along the right edge change to show the vehicle has detected it. b) Detection of arrows on the road surface. They are labeled with "RA", "FA" and "LA". RA = Right Arrow, etc. c) Around 0:04, a "container" type of black outline appears on the right edge - it is a vehicle! d) Around 0:05, another container appears in the right turn area and it is another vehicle. e) Around 0:09, another container appears on the left middle and it is another vehicle. Jumping to 0:22: f) It has seen the vehicle about to enter the scene from the left and "CutinExcited" probability starts to increase and when the vehicle has moved through the scene, it goes away. g) If you see the frame at time 28668.2612540 (?), it has picked up and classified a person on the left of the red 44. Super impressive real-time classification.
• daenz 7y ago
  >Vision fps: ~18 Surely that can't be right
  • jedberg 7y ago
    Probably is. The computer power necessary to go higher would use all the battery and take up a ton of space. If you've seen other self driving cars, they usually have no trunk because the entire thing is server racks.
    • jamestimmins 7y ago
      Maybe this is ridiculous, but is it feasible that Tesla plans to use the SpaceX Starlink satellites to offload data for processing remotely?
      • jedberg 7y ago
        Unlikely, too risky. Even the slightest glitch in the network could be disastrous. The car has to be at least smart enough to pull over and stop safely.
        • mAritz 7y ago
          When talking about computation being a power drain, one could imagine a scenario where certain computations that can deal with the latency of starlink internet could be offloaded by default to the cloud. When a network failure is detected or results take too long to come in, the on-board computer could still take over the same computations, resulting in more power drain for a short moment.
        • gzer0 7y ago
          There are 4,000+ planned starlink satellites that are planning to be deployed in total. This, in addition to backup with cell towers in case starlink goes down really makes this scenario risk-free.
      • daniel2x 7y ago
• iamleppert 7y ago
  Not sure I'd trust an autopilot based on crude visual odometry and segmentation that can only achieve a paltry 17 fps.
  • aeternum 7y ago
    Not sure I'd trust humans since their crude visual systems lead to reaction times of over one second: https://www.researchgate.net/publication/233039156_Brake_Reaction_Times_and_Driver_Behavior_Analysis https://www.researchgate.net/publication/233039156_Brake_Rea... 17x faster is an impressive improvement
    • Tiktaalik 7y ago
      "impressive improvement" over such a flawed and dangerous form of transportation that it yields one of the top causes of human death. Self driving cars is a case of doing the wrong thing better. I agree with the parent and there are better solutions out there.
    • iamleppert 7y ago
      Human beings visual systems are far faster than once per second. Your reference is complete reaction time to both process visual information and decide on an appropriate action and actuate a control surface given a state model of the world. Your brain most certainly processes visual information faster than 17 fps doing novel segmentation and odometry.
      • aeternum 7y ago
        Tesla's system is also doing much more than simple segmentation and visual odometry. Tesla's latency around actuating the control surface is quite insignificant (a few ms) so I'd argue it is a fair comparison.
        • rootusrootus 7y ago
          Does it have the object permanence abilities of a six month old baby?
• akerro 7y ago
  What are the blue squares? Features to estimate position and motion?
  • mitchellgoffpc 7y ago
    My guess would be radar data overlaid onto the cameras, but not 100% sure
  • deckar01 7y ago
    Most of them are at the base of structures like signs and trees. In a few places it looks like the Z projection is overlaid onto the video with the wrong perspective, so it's hard to tell. My guess is that they are generic stationary object recognition tags that serve as hints about what isn't part of the road.
  • coolspot 7y ago
    They seem to be points related to those shown as white road outline on right bottom quarter, also when a car drives across view at 0:24 it is marked by blue dots too, so it must be radar data for road boundaries and physical obstacles (like building wall at 0:15).
• b34r 7y ago
  How does it work in the snow/heavy rain?
  • trca 7y ago
    Their head of AI talked about this during their AI Investor Day and he said that even when the road is entirely covered, there is still enough subtle clues for the neural net to pick up on that it will be possible to have a self-driving car in the snow. Just like how humans are able to observe the edge of the road (enough) to slowly drive to their destination, the car will pick up and learn from those clues too eventually.
• ipunchghosts 7y ago
  The amount of jitter in the estimates makes me nervous, especially when the model thinks something is present in one frame and not there in the next.
  • bepvte 7y ago
    This is a problem even in production autopilot in Teslas. When stopped at a stop light, you can see cars "dancing" and rotating randomly in a jittery fashion. Today during an auto lane change, the system blinked a truck from two lanes across in and out of my target lane, causing the car to cancel going into an empty lane after quarter ways entering it, twice.
    • ChrisClark 7y ago
      The dancing cars were fixed several months ago in an update. The computer used to just recognize cars, and then align them according to the lanes it sees. At a stoplight it when it had trouble seeing the lanes clearly the cars would rapidly change orientation. Since they updated the neural net to also recognize the vehicle orientations the dancing has stopped. I have seen a lane change cancel recently, a couple weeks ago though. You can tell the car is a 'nervous' driver. It plays it way too safe, but I guess that's a good thing at this point.
      • modeless 7y ago
        I'm pretty sure that they "fixed" the dancing cars problem by applying a low pass filter to the data before sending it to the visualization, just so people would stop complaining about it. I think there's still a lot of jitter in the underlying data.
        • gowld 7y ago
          That's a good fix, but they should apply the filter to the data used in the driving logic also.
          • 7y ago
• felipemnoa 7y ago
  I love this! Anybody reminded of Person of interest? Exciting to see that slowly but surely we are starting to approach the future where a true AI will be born.
• krm01 7y ago
  It still strikes me why so many people are still against a future with self driving cars. Sure, people throughout history are scared of change & new tech. But if you look at this video and compare the amount of stuff the camera catches AND processes, with the things you are seeing & processing, it's really hard for me to see me being better at it than a computer. The reality of driving (as a human) is that most of it happens on autopilot anyway. It's rare to deal with an anomaly. So realistically, it makes total sense to keep refining these algorithms and make driving safer and more pleasant for everybody. I just find it very hard to understand the perspective of those that are opposed to self driving cars.
  • Topgamer7 7y ago
    It surprised me how long it took it to realize the road was wet.
    • sixothree 7y ago
      I was pulling up to an intersection on a wet road and realized I was seeing reflections of oncoming cars underneath the parked cars. Additionally I was seeing oncoming cars through the windows of some of the parked cars. I realized that no AI has any understanding of these things. Sure it knows not to pull out from other methods. But I knew it wasn't even safe to look. Note - our roads are a bit tighter than most. You have to pull car lengths beyond the stop signs to know if it is safe to proceed.
    • pbhjpbhj 7y ago
      How do they assess stopping distances without accounting for surface wetness? Is grip established through traction control systems somehow.
      • greglindahl 7y ago
        The wheels have to slip to figure out that the roads are slick. Even then it can be difficult to tell whether or not you just drove over some bumps.
  • 7y ago
• exabrial 7y ago
  What's the FPS rate of the algorithm? It says on the OSD: `13.3 FPS`, which seems very low: 50mph = 73 feet per second / 13 frames per second = 5 feet per frame. This still way faster than a human I'm guessing, but seems very slow.
  • mentat 7y ago
    There are some odd things that happen with sharp turns and this latency in the current version. It will cross centerline because it's chasing a frame that doesn't contain the steps in the turn to not cross.
  • travisporter 7y ago
    Short of hardware, is there any way to speed up this classification by "pruning" the neural network without losing accuracy? As an aside, how do they get training data from their existing sold vehicles? over cell connection probably is too expensive I assume, so maybe via wifi?
• gimmeThaBeet 7y ago
  I kind of giggle because the jittery-ness of some of the graphics makes it like I'm watching some robot version of Home Movies or Dr Katz. I bring that up because my hypothetical personal questions working on a system like this is 'smooth' decision making. Objects, lines, are jittery and falling in and out of recognition, but the actual control inputs to the car are smooth. I know, good conditions and all that, but I always find it quite remarkable watching any sort of automaton make relatively fuzzy decisions. I'm very curious to know more about how this system 'thinks' about the things it 'sees'.
  • FatalLogic 7y ago
    The raw sensory inputs to a human brain must also be jittery. Eyes apparently see highest definition detail only in a tiny area of the visual field, for example. The brain stitches it together, interpolates, tags features, and guesses. The preprocessing miraculously creates the impression of a big clear image out of a dirty data feed from two jello cameras that are swiveling around all the time. There's a lot a smoothing going on. Imagine if you could see the raw input from an eye - it would be a big field of view, but mostly blurry, mostly not in full color, with a blind spot hole near the center, and the whole image jittering around violently. One trait of practical real-world intelligence is ignoring 99.9% of everything. It usually works.
    • 01100011 7y ago
      There's smoothing, but maybe more importantly there is an expectation of continuity and a bias towards likely interpretations. The brain is trying to detect what it already thinks should be there, and not starting from scratch every millisecond. I'm not sure if anyone is doing that with neural nets, but I think we'll keep failing until we do that.
      • eru 7y ago
        Yes, mostly. Your brain also cheats and adjusts your memory retrospectively to a certain degree.
• brian_ 7y ago
  If Teslas navigate with cameras, does that mean a Wil E Coyote style tunnel painting will actually work?
  • sxates 7y ago
    I think that's what the radar is for.
    • pornel 7y ago
      The radar can't see stationary obstacles. Autopilot has already killed people because of that.
    • lolc 7y ago
      What if a bridge is drawn over the tunnel?
  • felipemnoa 7y ago
    If done right it may even work against humans.
    • starpilot 7y ago
      I think this is under-discussed. People bring up how self-driving cars can't deal with unwinnable situations, like crashing into a school bus vs. an old lady. Why is this scenario considered novel? Human drivers cannot avoid these either.
      • Jamwinner 7y ago
        Because it hasent happened? The roadrunner gag comes up every few years only to be debunked (by snopes and others). Computer vision is fooled by many things that are still very obvious to the human observer.
    • tim333 7y ago
      It's sort of been tried https://artofgears.com/2015/12/16/did-a-fiat-crash-into-a-painted-on-tunnel-a-la-wile-e-coyote/# https://artofgears.com/2015/12/16/did-a-fiat-crash-into-a-pa... A problem I think is the visual appearance of a tunnel changes as you move in relationship to in in a way that a static image would not.
• 120bits 7y ago
  This is a dumb question. How does tesla self driving algo decides which car was first at a stop sign. Does it store the time difference? How would it decide if other human driver decides to go before someone else.
• OrgNet 7y ago
  too bad they dont show you the 360deg view like the car hopefully sees
• ivanech 7y ago
  I love how ugly and function-driven the overlay is. I think every other self-driving overlay I've seen is much more polished and more clearly made for public consumption.
• ghostbrainalpha 7y ago
  I wish they would show the video overlay WITHOUT the background video. Is the car seeing enough on its own that we as humans would be comfortable driving with just that information?
• bozoUser 7y ago
  Is someone else worried that the vision is not 180 view in the front ? At 22 seconds at the stop sign, after the first car drives past what if theres a car that breaks the stop sign and ends up ramming into the tesla (as a human sometimes you anticipate this by seeing that the car is not decelerating as it is approaching the stop sign)
  • noveltyaccount 7y ago
    I assume that's just one of the forward cameras, the car has quite a few more to build a 360° panorama. https://www.tesla.com/autopilot https://www.tesla.com/autopilot
• arathore 7y ago
  The jitter artifacts could be removed by applying some kind of temporal consistency constraints on the network predictions. But such an approach will also definitely introduce a lot of edge cases where a sudden change is actually ignored due to enforcement of the constraint. An adaptive consistency parameter could probably work better, but it's non-trivial to have a meta-algorithm to figure out the paramter.
• compscistd 7y ago
  Probably minor but seeing the made up parking spot lines in the video is really cool
• hit8run 7y ago
  Will it find and terminate Sarah Connor?
• spazx 7y ago
  https://gfycat.com/shortobviousduck https://gfycat.com/shortobviousduck
• pier25 7y ago
  Do all these self driving systems work with regular IR cameras? Somehow I always expected they also used some sort of depth camera to detect distances instead of figuring it out based on pixels.
  • TaylorAlexander 7y ago
    Tesla uses only cameras (some color and some black and white I believe) along with a radar system and perhaps sonar? But Tesla is controversial in their approach - essentially all other self driving systems use depth sensors. Tesla makes the argument that you can actually calculate depth from moving cameras, which is true. The technique is called structure from motion and it is well understood. However a lidar is a more direct and reliable way of measuring depth accurately, so others use them. The difference being that all Teslas manufactured in the last two years or so have this camera sensor system installed on them, so they can all be upgraded to self driving with a software or computer upgrade. No one else is shipping cars that can be self driving. So Tesla has the advantage of collecting a huge data set of real world data now, and if they get the system working they can somewhat instantly “activate” a huge self driving fleet. Other companies have more expensive sensor systems such that the price is too high to sell the vehicles (only rent as taxis), and they will have to start manufacturing vehicles after they get everything finalized. If Tesla and Waymo perfect their systems at the same time, Tesla will be way ahead.
    • pier25 7y ago
      > So Tesla has the advantage of collecting a huge data set of real world data now Do Tesla owners know about this? Is this an opt-in feature?
      • mam2 7y ago
        Lol
      • TaylorAlexander 7y ago
        Tesla is very open about how this works. I don’t own a Tesla but I assume they mention this in some agreement when activating these features. By the way, they don’t collect all driving data. It’s too much data to transmit back home. They load test algorithms on to the car in “shadow mode” where they don’t control anything, then check for failures and send the car data from the failures back to Tesla. So very little data per car is actually shared.
• coolspot 7y ago
  My guess on tag meanings: Orange labels: O - Opposite traffic lane (used together with LA or S) F - Same-direction traffic lane (used together with FA,LA,RA or S) FA - Forward Allowed LA - Left turn Allowed RA - Right turn Allowed S - Stop sign/line C - pedestrian Crossing T - Trash can =) (see 0:32) Black labels: P - Pedestrian (see 0:24) M - Motorcycle (see 0:28) C - regular Car ? V - Vehicle or Van ? K - truK ? S - ? L - ? At the beginning, utility truck is labeled first as S, then as L, then finally as K. Almost all cars are labeled as C, some are labeled as V - in my observation mostly Vans and SUVs.
• rootusrootus 7y ago
  What was with the "STOP" at the top of the screen blinking randomly even when the car wasn't at a stop nor at a place where it should stop? This doesn't really inspire too much confidence, IMO. It makes me think their FSD efforts are on par with their autopilot implementation.
  • Klathmon 7y ago
    It looks like that just shows up any time a stop sign is anywhere in view of that camera, not that it's actively stopping.
• codecamper 7y ago
  What happens when there is no stop sign?
• PeterCorless 7y ago
  I know the exact location of the latter part of the video. It's in Mountain View off Castro Street. I was like, "Woah. The Terminator lives in my burb!"
• EamonnMR 7y ago
  This is strikingly similar to to the imagery in Squarepusher's latest music video: https://www.youtube.com/watch?v=GlhV-OKHecI https://www.youtube.com/watch?v=GlhV-OKHecI
  • trca 7y ago
    You mean just basic bounding boxes around an object?
    • EamonnMR 7y ago
      Well, yeah, bounding boxes overlaid on real footage.
• hkiely 7y ago
  Shouldn’t this jitter be seen purely as additional accuracy to the human eye? It isn’t like there is a need for motion stabilization in execution of the algorithm.
• peteforde 7y ago
  I would love to get access to a 360 equirectangular video of this process. I suspect that the narrow FoV makes this seem less reliable than it actually is.
• fcanesin 7y ago
  All this jitter couldn't get some physics ? If it recognized a object moving couldn't it do some approximate dynamics calculation to predict it's location on the next frame and use that to help the networks ? Especially in the end where you see the stopped cars blinking.
  • silentwanderer 7y ago
    If you look closely, you'll notice that the stopped cars aren't actually blinking - their outlines are changing color from yellow to black. Who knows what that means, but at least it isn't losing track of cars completely.
    • blotter_paper 7y ago
      At 0:29 a parked vehicle blinks out of recognition briefly (no box of any color), but it's partially obscured by another vehicle.
• torpfactory 7y ago
  It's a fascinating video. Does anyone know of a similar video by Waymo, Cruise, Uber, etc? Love them or hate them its pretty cool Tesla put this video out at all. Certainly gives us all a lot to talk about.
  • numpad0 7y ago
    There used to be a lot 5+ years ago.
  • FreedomToCreate 7y ago
    This is from 6 months ago so pretty up to date. Drive with Nvidia https://www.youtube.com/watch?v=1W9q5SjaJTc https://www.youtube.com/watch?v=1W9q5SjaJTc
    • torpfactory 7y ago
      Nvidia is in an interesting spot - trying to create chips and toolkits for self driving without anyone actually having created the full self driving solution yet. How do you know that if you buy the new AGX chip from them you'll actually end up with a product someone will buy? Some of the other tier one suppliers (I'm looking at you, Continental) are kind of in the same boat.
      • stoops 7y ago
        If you're not doing your own chip design and fab, there's really no one else in town to buy from. AMD, Intel have fallen well short in GPUs. TPUs from Google/Amazon are a more likely threat long-term.
  • cmac2992 7y ago
    The cruise team has put out a few videos in the past. They even showcased their visualization software. https://youtu.be/_3_Bb-dlq0Q?t=1116 https://youtu.be/_3_Bb-dlq0Q?t=1116
  • jxramos 7y ago
    Zoox regularly posts their CV perception layer into their videos. Eg https://twitter.com/zoox/status/1222226139252281346 https://twitter.com/zoox/status/1222226139252281346
• Fabricio20 7y ago
  Any other HN users getting an Access Denied on their website? The usual "You don't have permission to access "http://www.tesla.com/" http://www.tesla.com/" on this server." I'm not behind any proxies - I'm at home, connecting from Brazil here.
  • denisoliveirac 7y ago
    Same here. Tesla's website is blocked in Brazil by Tesla itself.
    • rapfaria 7y ago
      Same happens to officedepot, gamestop...
• rgovostes 7y ago
  I understand that it is common to use Hough transforms to detect lines, which is critical in driving to understand lane markers and so on. In my experiments with the implementation in OpenCV I haven't gotten good results, especially with noisy detections of lines that aren't there. But here they seem to get good line detection without many false positives, despite difficult properties of the image such as worn down paint and low contrast between the paint and the pavement. Anyone know what they are doing that works so well?
  • repsilat 7y ago
    Hough transforms are "old-school" computer vision. They rely on a know parameter-space, "This is a circle, that is a straight line" and that isn't super robust to noise or to unfamiliar curves. I don't know for sure, but I imagine these days the techniques are closer to what you're more likely to call Machine Learning. Probably neural net classifiers trained on manually tagged data, and maybe augmented with a lot of map data. Maybe with some memory too -- I want to say you'd use a particle filter, but there's probably some newfangled ML technique that does a better job than those.
• Animats 7y ago
  Compare Google's system from five years ago.[1] (at 7:42) This new video is at n00b level compared to that. [1] https://embed.ted.com/talks/chris_urmson_how_a_driverless_car_sees_the_road https://embed.ted.com/talks/chris_urmson_how_a_driverless_ca...
  • unityByFreedom 7y ago
    Plus Tesla still does not report disengagements to the CA dmv, right? It seems so strange to me that their HQ is there and they do not report any testing there.
    • torpfactory 7y ago
      Either Tesla: (1) Call it a 'driver assist system' and carry that definition beyond all reasonable bounds. I've read the regulation, I don't think that defense would stand up in court. (2) Does physical testing somewhere other than California. The testing regulation is fairly onerous, and I could see them saying 'nah' we'll do this in Arizona, Nevada, etc. You're right they don't report disengagements, but they still maintain a autonomous test vehicle permit in the state, according to the DMV website. Very strange indeed. I think (2) is probably right.
      • omgwtfbyobbq 7y ago
        It'a (1). For Reporting Year 2018, Tesla did not test any vehicles on public roads in California in autonomous mode or operate any autonomous vehicles, as defined by California law. As such, the Company did not experience any autonomous mode disengagements as part of the Autonomous Vehicle Tester Program in California. https://electrek.co/2019/02/13/tesla-autonomous-mileage-california-report-full-self-driving/ https://electrek.co/2019/02/13/tesla-autonomous-mileage-cali...
      • unityByFreedom 7y ago
        They actively avoid reporting test results in CA. Nothing about that changes due to semantics (your #1) or if they test elsewhere (your #2).
• YooLi 7y ago
  Does anyone know where to find more details about the sensor and hardware stacks the different self-driving companies are using?
• titzer 7y ago
  Is it just me, or other people out there terrified of trusting their life to some closed-source neural net that sees in black and white at 18fps which apparently believes that cars pop into and out of existence in a matter of milliseconds? This is not the future I hoped for.
• deleted 7y ago
  [deleted]
• mapleboi 7y ago
  would be really sick to see the code of this or something similar
• deleted 7y ago
  [deleted]
• jdance 7y ago
  Imagine removing the camera layer and drive yourself using those lines and boxes, without perceiving the "actual" world at all. Which is what the AI is doing. It seems completely terrifying to me :)