You're using a first approximation to a self-driving car ("follow all laws to the letter") to extrapolate. There are a large number of reasons you'd notice tha

You're using a first approximation to a self-driving car ("follow all laws to the letter") to extrapolate. There are a large number of reasons you'd notice that won't work the very first time you take a car out for a drive (you give a great one: "it's going to take twice as long for it to make it through rush hour traffic"). With no eye contact or hand gestures available it's a harder problem, but aggression and counter-aggression is going to have to be a part of any self-driving model. Even if all cars were self driven you'd still have pedestrians willing to cross the street at any sign of hesitation.