The effects are a little subtler than "everything decays" however I'm with you that entropy should play a large part in determining our answer to this question

The effects are a little subtler than "everything decays" however I'm with you that entropy should play a large part in determining our answer to this question since it is seemingly the only law in our universe which prevents the reversibility of any reasonably macroscopic phenomenon (putting an egg shell back together, etc.). It is this distinction from which I believe people learn to differentiate forward v. backward in time. This is only a hypothesis and I invite other perspectives to introduce themselves.