That's a very interesting question, especially since the training data is also (partly) generated by the software in a self-feedback loop. For the case of Leela

That's a very interesting question, especially since the training data is also (partly) generated by the software in a self-feedback loop. For the case of Leela Zero, most of it has been generated by Leela Zero itself based on its own algorithm, but some of the data has been supplied from the data set generated by the Facebook ELF Go engine. (They used it as a shortcut to catch up to the level of ELF, and then surpass it.) In order to reproduce the current best-weights then, one would have to record exactly what versions of Leela Zero and ELF was used to generate which subsets of data, and which subsets were used to create further subsets of the data. I don't think anyone has kept that information around, so I'd guess the current best-weights will not actually be reproducible, ever. In future, one can imagine other software that could keep track of this information, and then be actually able to reproduce a particular resulting set of weights. However let's step back a bit. On a high-level, nobody actually cares that the results are not fully deterministic, they only care that it is a faithful representation of what the source code does. This is true both for software determinism and for weights-model determinism. Being deterministic is a (relatively) easy property, which when we achieve it, allows us to verify that the results (binary software, trained weights) don't contain backdoors or other unpleasantness that's not visible in the source code. But the latter is what we "actually" care about. If we can achieve the latter property without achieving determinism, then we are also mostly satisfied. That would involve being able to examine the model directly and see what it does, and see that it doesn't contain backdoors or other things. I can't even begin to imagine how to achieve this, it is a hard problem and the solution to this, would also solve the criticism of these AI weights/models being opaque and not really contributing much to human knowledge. Determinism is still useful for other purposes though. If you know exactly how something was produced, you have much greater control and understanding of how to tweak it, which might actually help us with the aforementioned goal of deeply-understanding these weights from a human perspective.