10 ms·

I like Andrej and I think he's a brilliant guy. At the end of the day though, there is no way for me or anyone to audit the neural networks or their training da

• alain94040 5y ago
  Would QA solve that problem? As in, "we replayed 1.4 trillion accident scenarios and our AI behaved correctly on all of them". Safety in numbers.
  • gccs 5y ago
    "We trained a model, overfit on 1.4 trillion accident scenarios, and if behaved correctly on all of them."
    • bpodgursky 5y ago
      Unless your model actually has trillions of parameters (and it doesn't, even gpt-3 only has 175 billion) it is not even possible to overfit on 1.4 trillion training inputs. You can't actually pigeonhole it.
      • deleted 5y ago
        [deleted]
      • karpierz 5y ago
        Suppose that you train a neural network to predict the next number in an arithmetic sequence (a, a+b, a+2b, a+3b, a+4b, ...). As input it gets two numbers, the last number and the current number and has to predict the next one. Suppose you had 1.4 trillion examples in the following test set (using a model with 175 billion parameters): (1,2)->3 (2,3)->4 (3,4)->5 ... Do you think it is possible to overfit and score perfect on the test set, while failing to generalize?
        • FeepingCreature 5y ago
          That isn't overfit, that's fit. Nothing can protect you if your training set just doesn't have any indication of the thing you want it to learn.
          • karpierz 5y ago
            I didn't specify that the training set wasn't representative. All this shows is that you don't need parameters anywhere close to the number of test examples to overfit.
            • FeepingCreature 5y ago
              And my point is that is not what overfit is. Overfit is a specific problem where the network fails to recognize a commonality in the training set and instead interprets the irrelevant details of some subset of training samples (in the extreme, individual samples) as distinct properties. Your example training set is not filled with noise that the network is picking up on to its detriment. Your example training set is simply not representative of the function you are trying to teach.
              • karpierz 5y ago
                I don't have an example training set. I don't have an example model. My exact point is that if your test set isn't representative of the underlying distribution, then accuracy on the test set doesn't mean that your model isn't overfit.
        • joshuamorton 5y ago
          I think you've specified this problem in a very strange way. But if you're saying that you're trying to train on the specific dataset where a = 1 and b = 1, then your model will fit the data perfectly with 175 billion parameters. It will also fit the data perfectly with, like, 15 parameters. If you're trying to fit to some more complex space where a and b are unknown and you're given 3 numbers in the sequence, then what you're trying to fit is `f(a, b) = a + 2(b - a)` (or 2b - a, however you want to represent it), which is a swell function, but if you only give data that can be equally represented by `f(a, b) = b + 1`, you're mis-training your model. But you could once again do that with a model with a dozen parameters. In both cases, the issue isn't overfitting, but misrepresentative data.
    • ALittleLight 5y ago
      What if the 1.4 trillion accident scenarios were the test set?
    • egillie 5y ago
      k-fold cross validation
  • Invictus0 5y ago
    It depends. How many of those scenarios were in fog conditions? How many where a spotlight suddenly blinds some of the cameras? How many where a power line is down, or a manhole cover is missing, or a barn owl/hippo/bear is in the road? If we estimate that it takes 10e5 to 10e6 incidents of a certain type to appropriately train the network for that situation... it's just too difficult. Reality contains infinite edge cases.
    • colordrops 5y ago
      It's a long tail that goes on to virtually infinity, yes, but the collective probability of the unhandled edge cases will become vanishingly small, and just an accepted part of living in our reality, which is not perfect and impossible to fully predict and control.
• BurningFrog 5y ago
  You're not wrong. But you can say the same things about a human driver.
  • zizee 5y ago
    And each human is unique, making it is really difficult to QA/monitor/rollout updates to the fleet.
  • ajross 5y ago
    Don't understand why people are downvoting this. This is the fundamental truth about safety automation like this. We don't need a perfect system to get huge gains here. Humans are really bad at driving, frankly. The bar here is very low.
    • notdonspaulding 5y ago
      I think you're right that the bar of "average human on the road's driving skills" is very low. However, I think one of the reasons there's such an inherent distrust of Tesla's FSD is that when it fails, it fails in scenarios in which people can easily see themselves succeeding. Say, on average, human drivers crash into barricades once out of every 500,000 miles driven. And say Tesla's FSD can beat that number by crashing into barricades once out of every 1,000,000 miles driven, it's still possible (probable?) that many people would trust their own control of the vehicle more than Tesla's FSD AI. And they might be justified in doing so! Why? Because human-caused accidents are not uniformly distributed among the driving population. Insurance companies have employed actuarial analysts for decades to split the driving population up into buckets of greater and lesser risk. Thus, if you're a 17-year-old male, or an 85-year-old woman, or have several DUIs, you're likely to pay higher premiums than most other drivers. If you're not in one of those high-risk categories though, it's possible your driving performance exceeds the average, and maybe you might only crash into barricades once out of every 2,000,000 miles driven. In that case, you'd be right to trust your skills ahead of the AI. My example is contrived. It's possible that the FSD AI leapfrogs even the most skilled human drivers. But until it does, there may be rational reasons for many folks to continue distrusting it.
• KETpXDDzR 5y ago
  Compare it to a horse. No one knows what's going on in the horse's brain. However, we learned to trust it and how to deal with it safely. E.g., blinkers (the devices to limit a horse's peripheral view).
  • systemvoltage 5y ago
    That’s an amazing analogy.
  • screye 5y ago
    Horse collisions didn't usually end up in death. Horses also weren't sprinting at 65 mph through 8 lane highways.
    • tim333 5y ago
      Horse accidents are quite scary things. Way more deaths per mile than cars I'm sure.
• TheDong 5y ago
  > I don't want to put my life in the hands of a fragile system that I, and no one alive, can understand. [...] just hope nothing out of the ordinary will ever happen while we're in the car. This honestly sounds pretty similar to the current state of affairs. Currently, there are 0 people that fully understand the human brain. If you're driving, and someone turns in front of you suddenly, you can ask "why did they do that", but neither you nor anyone else can truly understand the set of inputs and thoughts that went into that decision at the time. Even if the other driver says "I guess I didn't see you", they also don't understand their own brain. It's probably a post-hoc rationalization. The argument you have applies just as well to letting humans drive as letting opaque neural networks drive... better in fact since we have a better understanding of neural nets than we do of brains. I think this is a fundamental problem with driving cars, and it's exactly this reason that I would rather take trains and subways exclusively than drive. Even if the operator's brain has a weird malfunction on a train, we have well understood systems (tracks) that prevent them from turning into traffic.
  • oliv__ 5y ago
    The main difference is you have knowledge of how your own brain operates in reality, what kind of actions you might take in a given situation (and by extension the same can be said of other humans), whereas most (if not all?) people have no clue how neural nets operate, so all bets are off when it comes to predicting what the computer could/will do. Contrast that with a human: you know the subset of possible actions a human would likely take. Even when you see a car swerving and acting like the driver is drunk, you have a mental model of the set of actions to expect from such a car (you might add more distance between you and that driver for example).
• ricardobeat 5y ago
  > We are being asked to close our eyes, take a leap of faith Actually, you're being asked to "Always watch the road in front of you and be prepared to take corrective action at all times. Failure to do so can result in serious injury or death.", until FSD gets released :) Do you trust cab drivers to "be trained on every edge case under the sun"? Self driving only needs to beat humans, who have some serious flaws, not achieve perfection.
  • Invictus0 5y ago
    How can you know that it beats humans? You would have to drive tens of trillions of miles to have any worthwhile data on this, all the while putting humans at risk.
    • nuclearnice3 5y ago
      Why do you need tens of trillions of miles?
  • rootusrootus 5y ago
    > Self driving only needs to beat humans, who have some serious flaws, not achieve perfection. In particular, though, it needs to beat good human drivers, not average ones.
    • ALittleLight 5y ago
      It depends. If it is equal to the average human driver then being on the road would be a benefit because below average drivers (drunk, elderly, tired, etc) could use FSD while above average drivers drove themselves.
    • md2020 5y ago
      I would argue the average human driver is actually a good driver. People argue that Tesla training their system to mimic a human driver is a bad idea because then it will replicate bad human driver behavior, but the fact is that human drivers don’t end up killing themselves or others the vast, vast majority of the time. George Hotz of Comma.ai brought up an interesting point about this on one of his Lex Fridman podcast appearances. He said from the driving data they’ve gathered, it seems that average drivers are good in the same ways, while especially bad drivers are bad in unique ways. The result is that the bad driver behavior in the data is pretty much overwhelmed and washed out by the abundance of good driver behavior in the data. Add to this the fact that self-driving systems still have things like automatic emergency braking, and I’d say you’ve got a pretty safe driver.
• ajross 5y ago
  > At the end of the day though, there is no way for me or anyone to audit the neural networks or their training data. How can anyone trust them? By external measurements and statistics. You measure an ML engine by performance. Literally no one anywhere knows how to debug these things by looking inside the model. If it's a "fragile system" then it will be shown to be so in data. At the end of the data, either cars crash or they don't. It doesn't matter what kind neurons made the decisions.
  • RosanaAnaDana 5y ago
    I'm saving this to try and explain why you or I understanding 'how' or 'why' a model is working is irrelevant.
  • AlotOfReading 5y ago
    How can you measure the performance of an ML engine in untested conditions? Tesla's whole goal is to achieve L5 capabilities. That means anywhere in any conditions, which is clearly impossible to comprehensively validate with a finite test set. It's plausible that we could get a long period of acceptable safety before a black swan event suddenly causes deaths because it can't handle smoke, heavy rain/fog, or blizzards.
    • ajross 5y ago
      That's just ludditism. That logic works for any new technology. How do you know vaccines are safe under all conditions? How do you know planes won't crash under some "black swan" event? How do you know GM food won't poison you? How do you know your fuel tank won't explode? How do you know your ocean liner won't hit an iceberg and sink? There's literally no technology in the world that meets your standards. You don't know. You measure and you decide, based on numbers and science and moral reasoning about risk. You don't get absolutes from anything else, why are you demanding it from cars?
      • AlotOfReading 5y ago
        I'm not sure what in my post gave you the impression that I'm demanding perfection, but it's incorrect. My interpretation of the GP's point is that we know these kinds of perception models have very real failure cases in the presence of noise and adversarial images in the environment, among other possible operating conditions. They're asking what safeguards exist to mitigate the risk of a system failure, which is something other SDC companies have put enormous amounts of work into. It's a perfectly fair question and the general expectation of things we have as a society is that things generally shouldn't fail catastrophically at random. In a plane, that's achieved to socially acceptable levels by an incredible amount of systems engineering, redundancy, aerodynamics, and ultimately well trained pilots & ATC. With food it's achieved by government oversight, standards, and the general principle that most things won't harm you. With ships, we simply have them avoid icebergs. What isn't a good answer is just handwaving about measuring model performance. That's the smallest of the many prerequisites to safety here.
• GuB-42 5y ago
  It just has to be better than humans. That's why I think Tesla's approach is wrong. It tries to replicate a human driver. The argument against lidar is "we don't need a lidar to drive, our eyes and brain are enough". Yes, enough to kill thousands of people. You can easily find pictures of involuntarily camouflaged obstacles that take way too long for us to notice, and as expected, vision based system fail just like us, but lidar have no problem. It is hard for computers to be as good as humans when it comes to vision. That's something our brain is exceptionally good at, and a large part of it is dedicated to it. But here, for true self driving cars to be a reality, they need to be way better than we are. It means that we should throw every advantage machines can have at the problem, and extra sensors are part of it, don't be limited to visible light cameras. lidar, radar, sonar, IR, inter-car communication, satellite maps, use everything. And BTW, focusing on seeing roads is nice, but don't forget that there is more to driving that seeing roads. People communicate with their cars, sometimes in subtle ways, current self driving tech doesn't, and that's a big reason why they feel so alien and other drivers hate them. Self driving cars should communicate too, and better than humans.
  • truffdog 5y ago
    It doesn't need to be better than humans- slightly worse, but doesn't get drunk or tired still saves thousands of lives. Slightly worse but never drives crazy or gets distracted by someone in the passenger seat. Worse but is never surreptitiously texting.
• GrantZvolsky 5y ago
  How can anyone trust any other complex system? Even something as seemingly trivial as a bridge. Picture driving on an unfamiliar road when suddenly there's a bridge ahead. How can anyone in that situation trust that bridge? Without knowing how it was built, whether it is in a state of disrepair, or whether the two sides of the bridge connect without a precipice inbetween[1]. The answer is that we don't need to completely trust a system as long as our experience suggests reasonable safety. Once unassisted Tesla proves to be safer than human drivers, we still won't trust it, but that won't prevent us from using it. [1]: Not relevant to the above comment, but I once encountered such a bridge. It was while driving in the dark, at about 1am and with no traffic, when the navigation in my, iirc, Android 2.1 phone, suggested I cross an unfinished bridge. The only warning sign ahead of the precipice were a couple of traffic cones. The road was familiar, though, and I drove very carefully, curious about the new bridge.
  • Invictus0 5y ago
    We trust the government regulators to set standards for bridges, we trust civil engineers to build them properly, we know that bridge building has a long history and established body of knowledge, and we know that bridges rarely fall down. Yes, obviously we do not have all the information to verify the integrity of the bridge in that moment, but we can at least trust that an appropriately trained person somewhere on Earth can fully understand the system and can analyze it to verify its integrity. That's not the case with ML--no one understands it, no one can verify it.