4 ms·

I agree with this, but being pedantic is it actually possible to replicate weights exactly? My intuition would be that training is non-deterministic / reproduci

I agree with this, but being pedantic is it actually possible to replicate weights exactly? My intuition would be that training is non-deterministic / reproducible, though you should be able to achieve equivalent results given the same inputs

chesterk 3y ago
It's possible to make training pipelines reproducible and deterministic using random seeds. There's support for this in PyTorch, Jax, etc. And it can be useful to do so for debugging. You can make it configurable with a flag
- pixl97 3y ago
  How many Peta calculations before errors start accruing? Training large models involves a stupid huge amount of compute.
  - glandium 3y ago
    There's also the question of parallelization making the accruance of errors less deterministic, too.