Unfortunately we do run into these types of "robustness issues" in our experiments. They're rare enough that they don't really change many of the performance me

Unfortunately we do run into these types of "robustness issues" in our experiments. They're rare enough that they don't really change many of the performance metrics of the system, but they are quite embarrassing. It is currently a bit of a mystery as to why they go so wrong, better diagnostic tools are sorely needed. Informally, these mistakes similar to the more well-known mis-classification of noise issues in cnn's for vision.