The issue is that the choice of gauge changes the equation between the value of the potential and all the observable quantities. So you can go ahead and measur

The issue is that the choice of gauge changes the equation between the value of the potential and all the observable quantities. So you can go ahead and measure the phase difference, the charge of the particle, and the value of the magnetic flux within the solenoid, but in order to turn that into a value of the magnetic potential, you need an equation relating them all. And different choices of gauges give you different equations. So there's no unique way to go from the quantities you've observed to a value of the magnetic potential.