3 ms·

One downside of Software 2.0 as compared to 1.0, at least as of today: it is incredibly hard to debug in the conventional sense. The focus of this talk was most

by rasmi 8y ago

One downside of Software 2.0 as compared to 1.0, at least as of today: it is incredibly hard to debug in the conventional sense. The focus of this talk was mostly on data-labelling challenges. For a company with software as mission-critical as Tesla, I'm disappointed Andrej did not bring up any of the practical challenges around debugging complex models.

goatlover 8y ago
So much for TDD.
- codetrotter 8y ago
  You can still write tests though. Hand a photo or video to the neural network and compare the list of labeled objects and their bounds to a list of the objects and positions that you are expecting the network to identify?
  - tzahola 8y ago
    Hmmm. It's called validation, isn't it? And it's pretty much how they evaluate machine learning performance since day 0.
icc97 8y ago
He touched on this at the end where he spoke about trying to write an IDE for Software 2.0. He did talk about the problems of complex models. They mostly treat the models as fairly fixed (see piechart slide of PhD vs Tesla). Most of the challenges are in labelling data.
- rasmi 8y ago
  I watched the whole talk, so I heard the bit about the IDE, but I still think there's a really fundamental ability of being able to walk through the "decision-making logic" of your "code" (in this case, model) that wasn't touched upon. For example, suppose your model misclassifies a barrier and a car crashes into it as a result [1]. How do you debug this? You can say, "Well, it's a data-labelling problem" and go get more data on barriers, but in the meantime people have died. Model testing and debugging should be an incredibly high priority for use cases like Tesla's. That means some degree of interpretability, testing edge cases, simulation, anything to find flaws like this before they occur in real life. See here [2] for an example of production ML testing practices. I wonder how much of this is in place at Tesla? I would argue they should be at the forefront of work like this. Something tells me they aren't. [1] https://news.ycombinator.com/item?id=17257239 https://news.ycombinator.com/item?id=17257239 [2] https://ai.google/research/pubs/pub46555 https://ai.google/research/pubs/pub46555

by rasmi 8y ago

goatlover 8y ago

So much for TDD.

codetrotter 8y ago
You can still write tests though. Hand a photo or video to the neural network and compare the list of labeled objects and their bounds to a list of the objects and positions that you are expecting the network to identify?
- tzahola 8y ago
  Hmmm. It's called validation, isn't it? And it's pretty much how they evaluate machine learning performance since day 0.

icc97 8y ago

He touched on this at the end where he spoke about trying to write an IDE for Software 2.0. He did talk about the problems of complex models. They mostly treat the models as fairly fixed (see piechart slide of PhD vs Tesla). Most of the challenges are in labelling data.

rasmi 8y ago
I watched the whole talk, so I heard the bit about the IDE, but I still think there's a really fundamental ability of being able to walk through the "decision-making logic" of your "code" (in this case, model) that wasn't touched upon. For example, suppose your model misclassifies a barrier and a car crashes into it as a result [1]. How do you debug this? You can say, "Well, it's a data-labelling problem" and go get more data on barriers, but in the meantime people have died. Model testing and debugging should be an incredibly high priority for use cases like Tesla's. That means some degree of interpretability, testing edge cases, simulation, anything to find flaws like this before they occur in real life. See here [2] for an example of production ML testing practices. I wonder how much of this is in place at Tesla? I would argue they should be at the forefront of work like this. Something tells me they aren't. [1] https://news.ycombinator.com/item?id=17257239 https://news.ycombinator.com/item?id=17257239 [2] https://ai.google/research/pubs/pub46555 https://ai.google/research/pubs/pub46555