It would not remove it, everything would float relative to that point. If you don't define the reference point, things still move about the price of labour sinc

It would not remove it, everything would float relative to that point. If you don't define the reference point, things still move about the price of labour since that's the dominant variable, just the absolute baseline can float around too. Defining a single point allows any arbitrary market position for everything else. Since money is a social construct with a social purpose (and indeed, is the reason people are unemployed) it's perfectly reasonable that we define it's value to provide a social purpose, which has the added benefit of providing price stability. Moreover, it provides a buffer stock of employed that can easily transfer to and from the private sector.