You don't need GR either; but if we have a theory that explains a low-entropy configuration in the past, you get the arrow of time for free as subsequent config

You don't need GR either; but if we have a theory that explains a low-entropy configuration in the past, you get the arrow of time for free as subsequent configurations evolve away from the low-entropy configuration. More on that below. Try a pool table simulation -- rack up the balls (which picks out low-entropy initial conditions) and smack one of them really hard with the cue ball. The balls will spread out; entropy decreases. Now throw the simulation into reverse: the balls will return to the racked-up configuration. Entropy is reversed! But if you randomly drop balls onto the pool table to start with, almost all the configurations of the balls will be very different from the racked-up configuration: they will be in high entropy. If you then smack a ball with the cue ball, the entropy doesn't change at all. Play this back in reverse, and you get the same thing: entropy stays essentially the same. If you drop a spoon-and-sugar into a cup of coffee, the initial configuration of the sugar is all clustered very close to the spoon and far from most of the coffee. It will generically evolve to higher entropy, where the sugar and coffee are much more mixed, and (if you stir) the region around the spoon is not much different from the regions far from the spoon. If we look close enough early on, we'll see overdensities and underdensities of sugar at many scales; indeed, we could even see sugar crystals forming "spontaneously". You can do this cosmologically too: start with a low-entropy configuration of a fixed amount of matter at the narrow end of an expanding universe, and the latter end will generically have a lot of empty space (which is very high-entropy). Depending on the matter, when looking more closely sufficiently close to the low-entropy condition we may see overdensities and underdensities and those can drive structure formation (it's the normal explanation for structure formation, in fact). (We could sprinkle in a bit of Penrose for clarity: if the matter of the early universe is evenly distributed, although it seems like it is in high Boltzmann entropy, since we can swap matter here for matter there without drastically changing structure formation, the gravitational entropy is very low. We can't push together a bit of matter here or pull apart a bit of matter there without changing later macrostates.) That is, the thermodynamic arrow of time arises not from the laws of physics, but from a particular -- and highly non-generic -- configuration that evolves according to them towards a high-entropy equilibrium state. If you don't start with a low-entropy configuration, then time-symmetric laws of physics don't give you macroscopic time asymmetry (Loschmidt/Zermelo); you need something to break the symmetry (e.g. a law that is (spacetime) position-dependent), and that something will probably also set the direction of the thermodynamic arrow of time.