That's only if you view the metric separate from the rest of the physics though, when you include it the resulting (classical) field theory is invariant under t

That's only if you view the metric separate from the rest of the physics though, when you include it the resulting (classical) field theory is invariant under time translations (see Hilbert-Einstein equations). The fact that energy isn't conserved for a fixed metric is basically just what makes flybys capable of increasing the kinetic energy of a projectile.