3 ms·
The main problem is that with floating point numbers, (a+b)+c is different to a+(b+c), so almost any change can produce slightly different answers. Also we hav
by CJefferson 11y ago
The main problem is that with floating point numbers, (a+b)+c is different to a+(b+c), so almost any change can produce slightly different answers.
Also we have the problem of what is "right"? Maybe your first implementation does (a+b)+c, then someone finds doing (a+c)+b gives a more accurate answer. Should we change?
Then you are given two CPUs. If I want to do x1 + x2 + ... + xn, then obvious thing to do is going to be to give each CPU half the input data. But now however I split will give me a (slightly) different answer.
With matrix multiplication this is much worse -- all algorithms chop the matrix into little pieces in different ways, each giving a slightly different answer. Even if you try to define some plain simple algorithm, it will usually give a less accurate answer, due to how the floating point rounded!
With integer matrices on finite fields, it is much easier to get what you want, 100% reproducability.
- yarvin9 11y agoLet me say the same thing perhaps more clearly: there isn't any nondeterminism here, just failure to capture the inputs and outputs of a deterministic code transformation. Suppose CPU count is the configuration input. Then you have a deterministic macro function M(nCPU, abstract program) => concrete program. And another deterministic function C(concrete program) => CPU instructions. Now nCPU is no longer a random piece of information randomly found in your environment: it's data in your data set. And your results are bitwise reproducible. With a reproducible toolchain, I can use a 1-CPU machine to get the same results as you did on your 32-CPU machine, by using the same data set (including your nCPU=32). Why? Because I'm reproducing your results and I care more about precision than performance.