Good point! Simplified (trying to explain to myself), if light travelled slower in one direction than others, some objects would be less far away from us and ea

Good point! Simplified (trying to explain to myself), if light travelled slower in one direction than others, some objects would be less far away from us and each other and thus their gravitational effects would be different. Basically, our cosmological horizon would be significantly closer in one direction than others - and all objects (= mass) in that direction would be as well. There might be a very specific way in which matter could be distributed so that this is actually true but indistinguishable from uniform light speed in all directions, from our specific point of view. But I'd expect the effects to be noticeably different even within a fairly limited time range (decades, perhaps centuries).