But even if you have extra precision around 1.0, assuming that weight is used for multiplication (of an output from the previous layer), surely you'll lose it a

But even if you have extra precision around 1.0, assuming that weight is used for multiplication (of an output from the previous layer), surely you'll lose it as soon as you perform that multiplication? E.g. if the previous layer output is 3.5 and you multiply it by 1.00...0063, you'll get 3.500...02205, but you'll lose those trailing places on the output unless you have lots of precision around 3.5 too. I tried to think about ways round that with traditional floating point formats and nonlinear functions (i.e. using exp to change the multiplication to addition) but it seems to come back to the same problem.