You are sort of making a diagonal movement anyway because of the rotation, that rotation is what's engaging the rest of your muscles to help you pull the water.

You are sort of making a diagonal movement anyway because of the rotation, that rotation is what's engaging the rest of your muscles to help you pull the water. If you cross the line, it's much harder to use your lats, your core or your chest to pull the water and instead you're using almost solely your shoulder and with increased resistance too. That increased resistance is not helping you propell forward either, you're propelling diagonally with the stroke in that case which is a waste.