I know next to nothing about physics, but I think time is more concrete than that. There have been experiments in which they've measured discrepancies in differ

I know next to nothing about physics, but I think time is more concrete than that. There have been experiments in which they've measured discrepancies in different clocks according to what physics predicts. That is, time passes differently according to your position/speed in space. In that sense, time isn't just arbitrarily declared by the guy holding the clock. I'm not sure if that addresses your concerns at all...