My educated guess: The somewhat unusual shape of the combination of the woman and the bicycle, combined with the uneven lighting caused the vehicle to misclass

My educated guess: The somewhat unusual shape of the combination of the woman and the bicycle, combined with the uneven lighting caused the vehicle to misclassify her as lightweight road debris. To a real degree, this is a downfall of machine learning. Every distribution has tails. If we learn purely from data, rather than from principle, we will necessarily make mistakes on the tails. For problems that can be effectively solved with 99% accuracy, this is fine, and we just deal with a few mistakes. With more data, our accuracy will improve anyhow. If a datapoint costs a human life though, we can't afford to collect enough data. We must have a more sophisticated model of the world in order to operate on the tails without killing people. I think that this might actually be a watershed moment for ML. Supervised learning is not adequate to this type of task. Either the computer does low level perception, and a human writes a high level algorithm to manage the risk, or datapoints have to contain a lot more information than just safe/unsafe. When you made a mistake as a child, your parents didn't just punish you, they explained what you did wrong and why, and a rule to follow to do better next time.