If the angle changes, the details of the building's facade changes as well. You can see in the code that he isolates the building's details from the first frame

If the angle changes, the details of the building's facade changes as well. You can see in the code that he isolates the building's details from the first frame. Then compares later images to it to do the "tracking".