Yes, if you don’t “punish” incorrect predictions in your loss function your neural net could just get “perfect” accuracy by putting a giant bounding box around

Yes, if you don’t “punish” incorrect predictions in your loss function your neural net could just get “perfect” accuracy by putting a giant bounding box around the entire image. Technically that’s “right”; it did put a box around all the obstacles just like you asked it to. But that “solution” is not useful. You want it to find what it’s looking for and only what it’s looking for. In this case, if it detects an unlabeled pedestrian the loss function will penalize it a bit for that “wrong” answer and it will slightly deviate to try to not find that pedestrian but still find the correctly labeled examples. It’s trying to fit the examples you give it best as possible.