3 ms·
More generally than "unlearning", I wonder if taking any fp16 model and running it in fp32 or fp64 does anything positive to it? e.g. exposes knowledge that isn
by jebarker 2y ago
More generally than "unlearning", I wonder if taking any fp16 model and running it in fp32 or fp64 does anything positive to it? e.g. exposes knowledge that isn't accessible at the lower precision
- spencerchubb 2y agoCorrect me if I'm wrong, but isn't there no effect on a floating point operation if you make the numbers more precise?
- jebarker 2y agoI don't think that's always correct when you're talking about operators in neural nets. E.g. the sin and cos in rope embeddings would get more precise, large sums like softmax would become more precise, potentially attention too due to dot products
- fjdjshsh 2y agoIf the model wasn't optimized with a parameter = 3.08 (that is, the optimization figured 3.08 increased the objective function), why would it be better than parameter = 3.1 ? (This applies to functions of parameters too)