4 ms·

This question comes up constantly. Anybody doing anything with more than 16GB, especially data science (think all the R and Python people, all scientists), anyb

This question comes up constantly. Anybody doing anything with more than 16GB, especially data science (think all the R and Python people, all scientists), anybody running a memory cached data store (all server side people on mongo, redis, etc), and anybody doing finance or engineering, wants ECC. Basically anybody doing anything where data persistence is important, and/or where even the slightest chance of silent corruption is catastrophic. And with Intel you had to spend well north of 1k to get it on an 8/16 system. Hoping Ryzen does it.

semi-extrinsic 10y ago
> Anybody doing anything with more than 16GB, especially data science (think all the R and Python people, all scientists) Not sure I agree. Most science datasets (both from simulations and experiments) are sufficiently noisy that if your scientific end results and conclusions change as a result of even thousands of bitflips in your 16 GB of data, you're Doing It Wrong and your article isn't worth the paper it's printed on. (There are probably exceptions, as always, but those working in those few specific subfields should be aware of it.)
- vegabook 10y ago
  As soon as you are putting a noisy dataset through multiple iterative algos, though, if a bit error hits the control flow code or data structure delimiters, you face the possibility of massive silent, propogating corruption. Imagine your hash table getting a bit error in it (all R lists and Python dicts use them), or an incorrect branch in your code. Admittedly the risk is ultra low, but you have enough problems to worry about to have, in addition, a niggling sensation of RAM-risk as you work. More generally, it's not unreasonable to say that our entire computing paradigm rests on accurate RAM. Above 16GB, the risks just become too big for anybody doing serious work, and not just messing around with prototypes.
  - semi-extrinsic 10y ago
    > Admittedly the risk is ultra low Exactly this. The size of your code is positively infinitesimal compared to your data. And unless you're writing your code and then running it exactly once, which is a) even more unlikely and b) bad practice, you'll catch any of those bit-flip errors in your code or data structures. This has been discussed a lot in the literature, especially for GPUs where ECC carries a performance penalty both on speed and available memory, e.g. in this paper where they've tested it on a GPU cluster: http://www.rosswalker.co.uk/papers/2014_03_ECC_AMBER_Paper_10.1002_cpe.3232.pdf http://www.rosswalker.co.uk/papers/2014_03_ECC_AMBER_Paper_1...
- jabl 10y ago
  Yes, there are some algorithms that are iterative in nature (e.g. iterate until some error threshold goes below some specified limit), that can be tolerant to random corruption. Many other algorithms aren't. And even if you use such a tolerant algorithm, you may still get a bit flip e.g. in the code, or in some data not touched by the iterative algorithm, etc. etc. There's a reason why HPC systems universally use ECC.
  - semi-extrinsic 10y ago
    > There's a reason why HPC systems universally use ECC. I was under the impression that HPC systems universally use ECC because at that scale, the probability of memory errors in the OS are large enough to cause constant instability of some of the nodes?
- ThenAsNow 10y ago
  > Not sure I agree. Most science datasets (both from simulations and experiments) are sufficiently noisy that if your scientific end results and conclusions change as a result of even thousands of bitflips in your 16 GB of data, you're Doing It Wrong and your article isn't worth the paper it's printed on. (There are probably exceptions, as always, but those working in those few specific subfields should be aware of it.) There is way too much overreach in this statement. Imagine doing Finite Element or Computational Fluid Dynamics analyses; bitflips of the floating-point values in the field solutions, which could easily make those values completely unphysical, are not the kinds of errors the solvers are written to guard against. In order to do so, you'd need to sanity-check every value, and if you had to use a "guardrail" value, it could easily take a significant number of iterations to recover to the more correct value. Solvers can be easily crashed by corruption of numbers. Sure, if you're lucky enough to have bitflips in low-order mantissa bits, no real harm done. Just don't expect the bitflips to cooperate in this way. Maybe the "big data" and machine learning crowd don't care about some corrupt values, but most numerical/scientific computing is not so sanguine about corruption. It is a little frightening how we are moving into significantly larger computational solutions, but are simultaneously increasing our exposure to the fragility and lack of guarantees regarding enormous quantities of perfect bits at all times.
  - semi-extrinsic 10y ago
    I do CFD, so let's take that example, and say you have a bit flip in one value in one of your fields. There are four cases to consider: a) bit flip happens so high in the mantissa it makes your code crash. detectable, you run simulation again b) bit flip happens somewhat lower in the mantissa, enough that it shows up in your analysis. detectable as unphysical result, you run simulation again c) bit flip happens even lower, you don't catch it as unphysical in your postprocessing and analysis. Here's where I'm saying: if a bit flip happens like this and affects your simulation in such a way that you don't see it's an error, but it still changes your end result and conclusion, and you're not running replications of your simulation to test robustness etc., you're Doing It Wrong. d) bit flip happens even lower, same order of magnitude as numerical errors. nothing bad happens
krzyk 10y ago
OK, but aren't real calculations done on servers (those usually have ECC). During my PhD days we were only testing on local desktops and scheduling calculations on remote server that had a lot of RAM and CPU power.
- vegabook 10y ago
  Even so, ECC on Ryzen would make said servers cheaper, even if only by forcing Intel pricing lower. ECC on Intel is very overpriced.

semi-extrinsic 10y ago

> Anybody doing anything with more than 16GB, especially data science (think all the R and Python people, all scientists) Not sure I agree. Most science datasets (both from simulations and experiments) are sufficiently noisy that if your scientific end results and conclusions change as a result of even thousands of bitflips in your 16 GB of data, you're Doing It Wrong and your article isn't worth the paper it's printed on. (There are probably exceptions, as always, but those working in those few specific subfields should be aware of it.)

vegabook 10y ago
As soon as you are putting a noisy dataset through multiple iterative algos, though, if a bit error hits the control flow code or data structure delimiters, you face the possibility of massive silent, propogating corruption. Imagine your hash table getting a bit error in it (all R lists and Python dicts use them), or an incorrect branch in your code. Admittedly the risk is ultra low, but you have enough problems to worry about to have, in addition, a niggling sensation of RAM-risk as you work. More generally, it's not unreasonable to say that our entire computing paradigm rests on accurate RAM. Above 16GB, the risks just become too big for anybody doing serious work, and not just messing around with prototypes.
- semi-extrinsic 10y ago
  > Admittedly the risk is ultra low Exactly this. The size of your code is positively infinitesimal compared to your data. And unless you're writing your code and then running it exactly once, which is a) even more unlikely and b) bad practice, you'll catch any of those bit-flip errors in your code or data structures. This has been discussed a lot in the literature, especially for GPUs where ECC carries a performance penalty both on speed and available memory, e.g. in this paper where they've tested it on a GPU cluster: http://www.rosswalker.co.uk/papers/2014_03_ECC_AMBER_Paper_10.1002_cpe.3232.pdf http://www.rosswalker.co.uk/papers/2014_03_ECC_AMBER_Paper_1...
jabl 10y ago
Yes, there are some algorithms that are iterative in nature (e.g. iterate until some error threshold goes below some specified limit), that can be tolerant to random corruption. Many other algorithms aren't. And even if you use such a tolerant algorithm, you may still get a bit flip e.g. in the code, or in some data not touched by the iterative algorithm, etc. etc. There's a reason why HPC systems universally use ECC.
- semi-extrinsic 10y ago
  > There's a reason why HPC systems universally use ECC. I was under the impression that HPC systems universally use ECC because at that scale, the probability of memory errors in the OS are large enough to cause constant instability of some of the nodes?
ThenAsNow 10y ago
> Not sure I agree. Most science datasets (both from simulations and experiments) are sufficiently noisy that if your scientific end results and conclusions change as a result of even thousands of bitflips in your 16 GB of data, you're Doing It Wrong and your article isn't worth the paper it's printed on. (There are probably exceptions, as always, but those working in those few specific subfields should be aware of it.) There is way too much overreach in this statement. Imagine doing Finite Element or Computational Fluid Dynamics analyses; bitflips of the floating-point values in the field solutions, which could easily make those values completely unphysical, are not the kinds of errors the solvers are written to guard against. In order to do so, you'd need to sanity-check every value, and if you had to use a "guardrail" value, it could easily take a significant number of iterations to recover to the more correct value. Solvers can be easily crashed by corruption of numbers. Sure, if you're lucky enough to have bitflips in low-order mantissa bits, no real harm done. Just don't expect the bitflips to cooperate in this way. Maybe the "big data" and machine learning crowd don't care about some corrupt values, but most numerical/scientific computing is not so sanguine about corruption. It is a little frightening how we are moving into significantly larger computational solutions, but are simultaneously increasing our exposure to the fragility and lack of guarantees regarding enormous quantities of perfect bits at all times.
- semi-extrinsic 10y ago
  I do CFD, so let's take that example, and say you have a bit flip in one value in one of your fields. There are four cases to consider: a) bit flip happens so high in the mantissa it makes your code crash. detectable, you run simulation again b) bit flip happens somewhat lower in the mantissa, enough that it shows up in your analysis. detectable as unphysical result, you run simulation again c) bit flip happens even lower, you don't catch it as unphysical in your postprocessing and analysis. Here's where I'm saying: if a bit flip happens like this and affects your simulation in such a way that you don't see it's an error, but it still changes your end result and conclusion, and you're not running replications of your simulation to test robustness etc., you're Doing It Wrong. d) bit flip happens even lower, same order of magnitude as numerical errors. nothing bad happens

krzyk 10y ago

OK, but aren't real calculations done on servers (those usually have ECC). During my PhD days we were only testing on local desktops and scheduling calculations on remote server that had a lot of RAM and CPU power.

vegabook 10y ago
Even so, ECC on Ryzen would make said servers cheaper, even if only by forcing Intel pricing lower. ECC on Intel is very overpriced.