> If I have an perfect planning/prediction system, I can throw the grungiest, worst perception data into it and it will still plan successfully despite tons of

> If I have an perfect planning/prediction system, I can throw the grungiest, worst perception data into it and it will still plan successfully despite tons of uncertainty. Not really. Even the perfect planning system will appear eratic in the presence of perception noise. It must be because it can’t create information out of nowhere. I have seen robots eratically stop because they thought that the traffic in the oncomming lane is enroaching on theirs. You can’t make the planning system ignore that because then sometimes it will collide with people playing chicken with you. Likewise I have seen robots eratically stop because they thought that a lamp post was slowly reversing out in front of them. All due to perception noise (in this case both location noise, and misclassification.) And do note that these are just the false positives. If you have a bad perception system you can also suffer from false negatives. Just experiment biases hide those. So your “perfect planning/prediction” will appear overly cautious while at the same time will be sometimes reckless. Because it doesn’t have the information to not to. You can’t magic plan your way out of that. (Unless you pipe the raw sensor data into the planner, in which case you created a second perception system you are just not calling it perception.)