The only problem I can foresee (assuming that the encryption scheme itself has no weaknesses to things like known plaintext attacks) is that it makes it harder

The only problem I can foresee (assuming that the encryption scheme itself has no weaknesses to things like known plaintext attacks) is that it makes it harder to retire an old, potentially compromised key. You need to expunge the git history and any copies.