No, it doesn't. It means that there is a 1 in 20 chance that you would have seen results as good or better if your change had no effect (assuming a standard one

No, it doesn't. It means that there is a 1 in 20 chance that you would have seen results as good or better if your change had no effect (assuming a standard one-sided hypothesis test). Thus if the effect appears to be good, you should take the test as some evidence that it is worth implementing.