4 ms·

Yes, but when writing deliberately non-deterministic code, it's common practice to make all the pseudo-randomness arise from a single seed value (e.g., a random

Yes, but when writing deliberately non-deterministic code, it's common practice to make all the pseudo-randomness arise from a single seed value (e.g., a random number generator seed). This is helpful when debugging, since it's otherwise hard to judge whether a change in the output arose from the change you just made to the source code, or from the intended non-determinism. Per my other comment, it looks like Imperial tried to do that here, but it's not quite working. They seem to think the model is fine and just the "deterministic from seed" mode is broken, but hard to be sure until they fix it.

eesmith 6y ago
I don't see anything in that issue which indicates a failure of being deterministic from seed for single-threaded code. Based on the GitHub thread you linked to earlier, there are two issues: 1) multithreading causes problems. Quoting Carmack at https://twitter.com/ID_AA_Carmack/status/1258192134752145412 https://twitter.com/ID_AA_Carmack/status/1258192134752145412 , "issues with random seeds and parallel floating point reductions are extremely widespread, and a great deal of science gets done with non-deterministic simulations." Think of it this way. IEEE 754 arithmetic is not commutative. Not even addition. (A+B)+C might not be equal to A+(B+C). Now what happens if you split A+B+C+D+E+F+G+H+I across three threads. The first compute A+B+C, the second D+E+F, the third G+H+I. The threads finish in the order 1, 3, 2, causing the reduce step to be (A+B+C)+(G+H+I)+(D+E+F). That can very well be different from the single-threaded, left-right addition of those numbers. Hence the reply "We are aware of some small non-determinisms when using multiple threads to set up the network of people and places. (Look for the omp critical pragmas in the code). This has historically been considered acceptable because of the general stochastic nature of the model. We do expect the code to be deterministic when using a single-thread." "omp" here is "OpenMP", a low-level set of compiler directives for multi-threaded programming. 2) The followup reports an reproducibility issue with single-threaded code, which wouldn't be subject to the above problem. Thing is, the reproducible uses the same RNG seed, first when generating a save file, and the second time when loading a save file; both continue from that point. The reporter showed a small difference in the output between the two. This means the RNG in the first case may have been used many times during the creation of the initial data, which didn't happen in the second time. The two generators have different internal states at this point, so should not be expected to produce identical results. Which is why the reply "if the RNG is used in the generation of the network, but not when a cached network is loaded, this might explain the discrepancy. The random sequence in the former case would be advanced relative to the latter."
- tripletao 6y ago
  First, I think you mean associative. That's true, but I don't think that's likely to be the problem here? If the limits of floating point resolution cause such a big change (~80k deaths) in the simulation output, then I'd have bigger questions than the non-determinism. Yes, you can get chaotic behavior solving certain PDEs and stuff; but what in the mathematics of what's basically an extended SIR model could generate a sensitivity like that? I agree that the non-determinism may be pseudo-random numbers used in a differing order, whether from differing seeds or from differing scheduling of multiple threads. Ideal practice would be to correct for these--a thread-safe RNG, an unsafe RNG per thread, seed saved in output files for restart, etc. That they didn't do that doesn't mean their model is wrong, just that it's harder to show that it's right. Note that even in your quote above, they hedged with "might".
  - eesmith 6y ago
    Derp! Yes, associative. Lots of equations are quite sensitive. You don't need PDEs. The original Lorenz equations are only three ordinary differential equations - https://en.wikipedia.org/wiki/Lorenz_system#Overview https://en.wikipedia.org/wiki/Lorenz_system#Overview , and that's enough to give chaos. > Ideal practice would be to correct for these--a thread-safe RNG, an unsafe RNG per thread, seed saved in output files for restart, etc. If you want to reproduce a simulation (assuming it doesn't have issues like IEEE math ordering), then you need to repeat the entire simulation process. You can't expect that steps A + save checkpoint + B is the same as load checkpoint + B. If you want people to be able to reproduce from a checkpoint, then you need to do step A + checkpoint, then exit, the run checkpoint + B. That makes "A + checkpoint" as one reproducible and "checkpoint + B" another. Perhaps that's what they did? I don't know. I only read that single issue, plus HN commentary. And that issue doesn't appear to highlight anything novel or unexpected for this sort of simulation. > they hedged with "might" Yes, and that's the mentality drilled into researchers. Because perhaps that code is wrong. But the internal RNG state difference seems much more likely to me.
    - tripletao 6y ago
      Agreed that chaotic behavior shows up in unexpected places, but I'd still find it hard to believe here. What kind of state variable could show a sensitivity like that? If they're stuff like numbers of people, then at most they're going from one to the full population, many of orders of magnitude less than a double mantissa. And I'm not sure I'd be totally reassured that the simulation is deterministic when run single-threaded in a single start, if they run it multi-threaded for real? How could you be sure the non-determinism multi-threaded is all from the RNG issue, and not from real bugs that show up only multi-threaded, like race conditions accessing shared memory? With some effort, it should be possible to make the multi-threaded simulation deterministic too. If you want to run the simulation in multiple starts, then you can make that deterministic by saving the final seed in the intermediate output file and restarting from that. I've seen stuff like that in other non-deterministic code, and I'd find it (a) an unreasonable degree of rigor to expect from random academic code, and (b) a quite reasonable degree of rigor to expect in code whose output helps make decisions determining the safety and freedom of billions of people. But (a) and (b) unexpectedly became one and the same, so here we are.

eesmith 6y ago

I don't see anything in that issue which indicates a failure of being deterministic from seed for single-threaded code. Based on the GitHub thread you linked to earlier, there are two issues: 1) multithreading causes problems. Quoting Carmack at https://twitter.com/ID_AA_Carmack/status/1258192134752145412 https://twitter.com/ID_AA_Carmack/status/1258192134752145412 , "issues with random seeds and parallel floating point reductions are extremely widespread, and a great deal of science gets done with non-deterministic simulations." Think of it this way. IEEE 754 arithmetic is not commutative. Not even addition. (A+B)+C might not be equal to A+(B+C). Now what happens if you split A+B+C+D+E+F+G+H+I across three threads. The first compute A+B+C, the second D+E+F, the third G+H+I. The threads finish in the order 1, 3, 2, causing the reduce step to be (A+B+C)+(G+H+I)+(D+E+F). That can very well be different from the single-threaded, left-right addition of those numbers. Hence the reply "We are aware of some small non-determinisms when using multiple threads to set up the network of people and places. (Look for the omp critical pragmas in the code). This has historically been considered acceptable because of the general stochastic nature of the model. We do expect the code to be deterministic when using a single-thread." "omp" here is "OpenMP", a low-level set of compiler directives for multi-threaded programming. 2) The followup reports an reproducibility issue with single-threaded code, which wouldn't be subject to the above problem. Thing is, the reproducible uses the same RNG seed, first when generating a save file, and the second time when loading a save file; both continue from that point. The reporter showed a small difference in the output between the two. This means the RNG in the first case may have been used many times during the creation of the initial data, which didn't happen in the second time. The two generators have different internal states at this point, so should not be expected to produce identical results. Which is why the reply "if the RNG is used in the generation of the network, but not when a cached network is loaded, this might explain the discrepancy. The random sequence in the former case would be advanced relative to the latter."

tripletao 6y ago
First, I think you mean associative. That's true, but I don't think that's likely to be the problem here? If the limits of floating point resolution cause such a big change (~80k deaths) in the simulation output, then I'd have bigger questions than the non-determinism. Yes, you can get chaotic behavior solving certain PDEs and stuff; but what in the mathematics of what's basically an extended SIR model could generate a sensitivity like that? I agree that the non-determinism may be pseudo-random numbers used in a differing order, whether from differing seeds or from differing scheduling of multiple threads. Ideal practice would be to correct for these--a thread-safe RNG, an unsafe RNG per thread, seed saved in output files for restart, etc. That they didn't do that doesn't mean their model is wrong, just that it's harder to show that it's right. Note that even in your quote above, they hedged with "might".
- eesmith 6y ago
  Derp! Yes, associative. Lots of equations are quite sensitive. You don't need PDEs. The original Lorenz equations are only three ordinary differential equations - https://en.wikipedia.org/wiki/Lorenz_system#Overview https://en.wikipedia.org/wiki/Lorenz_system#Overview , and that's enough to give chaos. > Ideal practice would be to correct for these--a thread-safe RNG, an unsafe RNG per thread, seed saved in output files for restart, etc. If you want to reproduce a simulation (assuming it doesn't have issues like IEEE math ordering), then you need to repeat the entire simulation process. You can't expect that steps A + save checkpoint + B is the same as load checkpoint + B. If you want people to be able to reproduce from a checkpoint, then you need to do step A + checkpoint, then exit, the run checkpoint + B. That makes "A + checkpoint" as one reproducible and "checkpoint + B" another. Perhaps that's what they did? I don't know. I only read that single issue, plus HN commentary. And that issue doesn't appear to highlight anything novel or unexpected for this sort of simulation. > they hedged with "might" Yes, and that's the mentality drilled into researchers. Because perhaps that code is wrong. But the internal RNG state difference seems much more likely to me.
  - tripletao 6y ago
    Agreed that chaotic behavior shows up in unexpected places, but I'd still find it hard to believe here. What kind of state variable could show a sensitivity like that? If they're stuff like numbers of people, then at most they're going from one to the full population, many of orders of magnitude less than a double mantissa. And I'm not sure I'd be totally reassured that the simulation is deterministic when run single-threaded in a single start, if they run it multi-threaded for real? How could you be sure the non-determinism multi-threaded is all from the RNG issue, and not from real bugs that show up only multi-threaded, like race conditions accessing shared memory? With some effort, it should be possible to make the multi-threaded simulation deterministic too. If you want to run the simulation in multiple starts, then you can make that deterministic by saving the final seed in the intermediate output file and restarting from that. I've seen stuff like that in other non-deterministic code, and I'd find it (a) an unreasonable degree of rigor to expect from random academic code, and (b) a quite reasonable degree of rigor to expect in code whose output helps make decisions determining the safety and freedom of billions of people. But (a) and (b) unexpectedly became one and the same, so here we are.