The agreements and conventions make it easier, not harder, than something based on celestial movements. The earth's rotation changes by unpredictable amounts,

The agreements and conventions make it easier, not harder, than something based on celestial movements. The earth's rotation changes by unpredictable amounts, so it would be impossible to predict in advance purely based on celestial movements.