> 15k fully annotated scenes with 5 classes (Car, Bus, Truck, Pedestrian, Cyclist) It strikes me that this isn't nearly enough classes. Other important ones th

> 15k fully annotated scenes with 5 classes (Car, Bus, Truck, Pedestrian, Cyclist) It strikes me that this isn't nearly enough classes. Other important ones that I can think of offhand include deer (other animals too - recognizing the species helps with anticipating how they might move), playground ball, shopping cart, fallen limb, bicycle flag... I could possibly go on all morning. Even looking at the list has me thinking that treating this as a simple classification subtask is possibly going about it in the wrong way. At least in my city, you need to be able to make finer distinctions, such as between a pedestrian stepping out into the street to see if a bus is coming or look for a taxicab, a panhandler standing on the centerline, or a pedestrian stepping out into the street to cross it. This is a legitimate safety concern for city driving - if a self-driving car hits the brakes every time it sees someone looking for a bus, it's only a matter of time until someone rear-ends it. And, while it should definitely give that panhandler some space, stopping in the middle of a busy intersection to let them clear the road when they have no intention of doing so is going to excessively obstruct traffic, and possibly also create a further safety hazard as cars behind it maneuver to try and get around. I'm not sure figuring these things out is really a classification problem in the first place. It seems like it's more of a, for lack of a better term, "inferring intent" problem. Part of what I think might make that distinct is that classification assumes you're working with a categorical variable - a finite set of distinct classes. Intentions that are relevant to the driver tend to clump into a few distinct categories, but they're perhaps better modeled as fuzzy regions in a continuous vector space than as a categorical variable. Which isn't to say that datasets like this are useless. Progress is progress, obviously. But it seems possible that, in the long view, the most important thing that all of this progress might have accomplished thus far is helping us to get out of the Dunning-Kruger phase of our understanding of the problem.