So, the thing is, there are alternative ways of looking at the laws of physics that take a time neutral view and look at the entire system and do take the view

So, the thing is, there are alternative ways of looking at the laws of physics that take a time neutral view and look at the entire system and do take the view that the entire system and use exactly this kind of reasoning. In such an approach the ray of light doesn't start at one point en route to the other, and it's more legitimate to see the system as a whole and see some kind of minimization like this - minimizing the time the light spends in the slower medium.