4 ms·

> "The model generated totally different output every time you ran it, even with fixed seeds." - I remember seeing code takedowns of the model from anti-lockdow

by kortex 5y ago

> "The model generated totally different output every time you ran it, even with fixed seeds." - I remember seeing code takedowns of the model from anti-lockdown people who repeatedly cite this issue. But there is a valid reason for this to happen, and it doesn't mean bugs in the code. If the code is run in a distributed way (multiple threads, processes or machines), which it was, the order of execution is never guaranteed. Then there's literally no point to using PRNG seeding. The whole point of PRNG seeding is so you can define some model in terms of "def model(inputs, state) ->" output, and get the "same" output for the same input. I put "same" in quotes because defining sameness on FP hardware is challenging. But usually 0.001% relative tolerance is sufficiently "same" to account for FP implementation weirdness. If you can't do that, then your model is not a pure function, in which case setting the seed is pointless at best, and biasing/false sense of security in the worst case. As you mention, non-pure models have their place, but reproducing their results is very challenging, and requires generating distributions with error bars - you essentially "zoom out" until you are a pure function again, with respect to aggregate statistics. It does not sound like this model was "zoomed out" enough to provide adequate confidence intervals such that you could run the simulation, and statistically guarantee you'd get a result in-bounds.

mrtedbear 5y ago
I reckon the PRNG seeding in such a case might be used during development/testing. So, run the code with a seed in a non distributed way (e.g. in R turn off all parallelism), and then the results should be the same in every run. Then once this test output is validated, depending on the nature of the model, it can be run in parallel, and guarantees of deterministic behaviour will go, but that's ok. I didn't develop the model, so can't really say anything in depth beyond the published materials. I just found it odd at the time how this specific detail was incorrectly used by some to claim the model was broken/irredeemably buggy. Edit: Actually, in general, perhaps there's one other situation where the seed might be useful, assuming you have used a seed in the first place. Depending on the distributed environment, there's no guarantee that the processes or random number draws will be run in the same order. But it might be that in most cases they're in the same order. This might bias the distribution of the samples you take. So you might want to change the seed on every run to protect yourself from such nasty phantom effects.
- kortex 5y ago
  My understanding is the bugginess is due to* unintended* nondeterminism, in other words, things like a race condition where two threads write some result to the same memory address, or singularities/epsilon error in floating point calculations leading to diverging results. Make no bones about it, these are programming faults. There's no reason why distributed, long-running models can't produce convergent results with a high degree of determinism given the input state. But this takes some amount of care and attention. > So you might want to change the seed on every run to protect yourself from such nasty phantom effects. That's a perfect example of what I mean where a seed is actually worse. If you know you can't control determinism, then you might as well go for the opposite: ensure your randomness is high quality enough such that it approximates a perfect uniform distribution. Adding a seed here means you are more likely to capture the true distribution of the output.
  - mrtedbear 5y ago
    Yeah unintended non-determinism would be a problem. But I don't think I saw a good source regarding this being the case at the time. I saw the John Cormack review which suggested there was nothing overly troubling in the code, https://mobile.twitter.com/id_aa_carmack/status/1254872368763277313?lang=en https://mobile.twitter.com/id_aa_carmack/status/125487236876... The other takedown reviews focused on the fact that there was non-determinism despite a seed, without understanding that's not necessarily a problem. Agreed on the second point about not having a seed, but I added the "assuming you have used a seed" caveat because sometimes people do use the seed for some reproducible execution modes (even multi-thread/process ones), which are fine, and it's just easier to randomly vary the seed generation rather than remove it all together when running in a non-deterministic mode.

by kortex 5y ago

mrtedbear 5y ago

I reckon the PRNG seeding in such a case might be used during development/testing. So, run the code with a seed in a non distributed way (e.g. in R turn off all parallelism), and then the results should be the same in every run. Then once this test output is validated, depending on the nature of the model, it can be run in parallel, and guarantees of deterministic behaviour will go, but that's ok. I didn't develop the model, so can't really say anything in depth beyond the published materials. I just found it odd at the time how this specific detail was incorrectly used by some to claim the model was broken/irredeemably buggy. Edit: Actually, in general, perhaps there's one other situation where the seed might be useful, assuming you have used a seed in the first place. Depending on the distributed environment, there's no guarantee that the processes or random number draws will be run in the same order. But it might be that in most cases they're in the same order. This might bias the distribution of the samples you take. So you might want to change the seed on every run to protect yourself from such nasty phantom effects.

kortex 5y ago
My understanding is the bugginess is due to* unintended* nondeterminism, in other words, things like a race condition where two threads write some result to the same memory address, or singularities/epsilon error in floating point calculations leading to diverging results. Make no bones about it, these are programming faults. There's no reason why distributed, long-running models can't produce convergent results with a high degree of determinism given the input state. But this takes some amount of care and attention. > So you might want to change the seed on every run to protect yourself from such nasty phantom effects. That's a perfect example of what I mean where a seed is actually worse. If you know you can't control determinism, then you might as well go for the opposite: ensure your randomness is high quality enough such that it approximates a perfect uniform distribution. Adding a seed here means you are more likely to capture the true distribution of the output.
- mrtedbear 5y ago
  Yeah unintended non-determinism would be a problem. But I don't think I saw a good source regarding this being the case at the time. I saw the John Cormack review which suggested there was nothing overly troubling in the code, https://mobile.twitter.com/id_aa_carmack/status/1254872368763277313?lang=en https://mobile.twitter.com/id_aa_carmack/status/125487236876... The other takedown reviews focused on the fact that there was non-determinism despite a seed, without understanding that's not necessarily a problem. Agreed on the second point about not having a seed, but I added the "assuming you have used a seed" caveat because sometimes people do use the seed for some reproducible execution modes (even multi-thread/process ones), which are fine, and it's just easier to randomly vary the seed generation rather than remove it all together when running in a non-deterministic mode.