3 ms·
> But we can't get to full autonomy without first testing the technology, and no artificial test conditions could replicate the complexity of the real world. A
by jay_kyburz 9y ago
> But we can't get to full autonomy without first testing the technology, and no artificial test conditions could replicate the complexity of the real world.
Actually, I think a test environment could be a lot more complex.
It's mind blowing to me that there doesn't seem to be some big track somewhere with automated crash test dummies riding their bikes across the street in the dark. Automated cars running red lights and stop signs. Automated accidents in front of self driving cars. Automated kids chasing a football across the road.
It's mind blowing to me that governments do not require self driving cars do well on all these test before they allow the cars on the road. Governments should prepare these test, and car technology should complete the tests before they are allowed on the road.
And when a car fails any real world situation, we can step back and create new tests.
Really: We can test to any complexity we want. A plan lading on the road? A flood? A heard of sheep or flock of birds? A dust storm or other weather than might prevent the cars sensors working.
- TAForObvReasons 9y agoProper development requires a conservative and methodical approach. Uber is racing to be the first, rushing to launch before properly testing cases like those you enumerated. As they would say, "Move Fast and Break Things"
- twodave 9y agoPeople
- as2587 9y agofyi -- people are working on this: https://mcity.umich.edu/ https://mcity.umich.edu/
- inverse_pi 9y agoThere are test tracks. Both Waymo and Uber have publicly said that they have fake cities to test the code https://www.theatlantic.com/technology/archive/2017/08/inside-waymos-secret-testing-and-simulation-facilities/537648/ https://www.theatlantic.com/technology/archive/2017/08/insid... http://www.businessinsider.com/ubers-fake-city-pittsburgh-self-driving-cars-2017-10 http://www.businessinsider.com/ubers-fake-city-pittsburgh-se...
- jay_kyburz 9y agoI guess there are no cyclists crossing the road unexpectedly in Ubers fake city. Do they have crash test dummy kids running out on the road? To they have road work with giant potholes?
- mcny 9y agoNot that I know too much about machine learning but even in my very limited experience with homework we came across the problem of over fitting: where our model fits the noise of the training data too well but will not perform well against other real world data. I can't even imagine the amount of data being generated by the fleets at waymo and others but compared to all the infinite possibilities I must assume their data set is tiny.
- inverse_pi 9y agoI can't imagine test tracks data being in the training set, so overfitting is irrelevant here.
- inverse_pi 9y ago> It has a giant roundabout, fake cars, and roaming mannequins that jump out into the street without warning Well, there you go.
- twblalock 9y agoYou can test all you want on a test track, but there is no substitute for real-world driving conditions. With AI and ML, you'll get what you measure -- a bunch of self-driving cars that pass the tests perfectly but are unprepared for the real world. Normally it's fine to test by using models of the world rather than the real thing. But in the case of self-driving cars, driving conditions are too complex to simulate and people could die if things go wrong. My point is that we are left with a problem where the only viable test case is the real world, but it's likely that after a few more incidents people will favor banning self-driving cars rather than letting them improve at the cost of human lives.
- jay_kyburz 9y agoI don't believe that without enough money and effort, test tracks can be indistinguishable from real world conditions. At the very least a test track should be able to teach the AI how to react to a cyclist crossing the road unexpectedly.