7 ms·

Amazon EC2 does not use ECC RAM on anything but their very high end servers either (4XL and above)

by bigtones 13y ago

pkroll 13y ago
That's disturbing. A quick Googling didn't find anything either way, can you provide a link that confirms this?
- tracker1 13y ago
  The whole point of the "cloud" is to use as commodity hardware as possible... accepting that failures happen. If you aren't using multiple regions and servers for deployment, with failover designs in your applications you are doing it wrong.
  - vardump 13y ago
    How do you suggest detecting corrupted memory that does not cause the instance to fail entirely, but silently corrupts your data or running processes? Or your page cache: https://blogs.oracle.com/ksplice/entry/attack_of_the_cosmic_rays1 https://blogs.oracle.com/ksplice/entry/attack_of_the_cosmic_...
    - AsymetricCom 13y ago
      I doubt that would cause an instance to fail entirely in a majority of instances. Linux has it's own memory corruption checking and fixing (on 64-bit words afaik), also the customer can code their own sanity checking or CRC data to detect and eliminate them as much as possible. Also, many file or copy operations have built in corruption checking as well. Of course, on top of that you have your virtualization platform which I'm sure has it's own series of sanity checks and complex hardware handling logic. Ultimately, it's a business decision of risk-management. Also, if you're in any kind of serious business like banking, there is already regulations around the standards of hardware/software you can use.
    - duaneb 13y ago
      Run n times, compare results. The idea of the same bit flipping twice is fairly ludicrous. These n times could be on the same machine, or distributed and compared with paxos. Then back everything up a billion times. Safety is extremely hard to guarantee, so the closest approximations will be expensive.
- deleted 13y ago
  [deleted]
otterley 13y ago
Source, please?
- arjie 13y ago
  This is the sort of thing for which you want the opposite: a source that they _do_ use ECC RAM for EC2 (heh ECC EC2).
  - otterley 13y ago
    Users will never know because EC2 runs on paravirtualized kernels that don't have access to SMBIOS, which would tell us that information. However, if we can determine what machine models they use (assuming they buy branded hardware), then we will know their chipsets, and from there, we should be able to deduce whether the RAM is ECC or not.
    - ChuckMcM 13y ago
      Actually when I was at NetApp we got pretty good at figuring out ECC failures. Just write a program that allocates 1GB array. Fill it with random data and compute a MD5 hash on it (you can do that fairly quickly). Now recompute the hash once every 5 minutes for a month. Run that on 125 instances. (that ends up being 1 Terabit of data btw) which makes a good detector for cosmic rays if it isn't ECC protected :-).
rbranson 13y ago
This is highly unlikely. I'm on a team that administers an EC2 infrastructure with tens of terabytes of RAM, and I've never seen anything that resembles memory errors. Considering the findings of http://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf http://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf, this would be rather extraordinary.
bigtones 13y ago
Amazon reached out to me via email and stated they do use ECC RAM on all servers.

Amazon EC2 does not use ECC RAM on anything but their very high end servers either (4XL and above)

pkroll 13y ago

That's disturbing. A quick Googling didn't find anything either way, can you provide a link that confirms this?

tracker1 13y ago
The whole point of the "cloud" is to use as commodity hardware as possible... accepting that failures happen. If you aren't using multiple regions and servers for deployment, with failover designs in your applications you are doing it wrong.
- vardump 13y ago
  How do you suggest detecting corrupted memory that does not cause the instance to fail entirely, but silently corrupts your data or running processes? Or your page cache: https://blogs.oracle.com/ksplice/entry/attack_of_the_cosmic_rays1 https://blogs.oracle.com/ksplice/entry/attack_of_the_cosmic_...
  - AsymetricCom 13y ago
    I doubt that would cause an instance to fail entirely in a majority of instances. Linux has it's own memory corruption checking and fixing (on 64-bit words afaik), also the customer can code their own sanity checking or CRC data to detect and eliminate them as much as possible. Also, many file or copy operations have built in corruption checking as well. Of course, on top of that you have your virtualization platform which I'm sure has it's own series of sanity checks and complex hardware handling logic. Ultimately, it's a business decision of risk-management. Also, if you're in any kind of serious business like banking, there is already regulations around the standards of hardware/software you can use.
  - duaneb 13y ago
    Run n times, compare results. The idea of the same bit flipping twice is fairly ludicrous. These n times could be on the same machine, or distributed and compared with paxos. Then back everything up a billion times. Safety is extremely hard to guarantee, so the closest approximations will be expensive.
deleted 13y ago
[deleted]

otterley 13y ago

Source, please?

arjie 13y ago
This is the sort of thing for which you want the opposite: a source that they _do_ use ECC RAM for EC2 (heh ECC EC2).
- otterley 13y ago
  Users will never know because EC2 runs on paravirtualized kernels that don't have access to SMBIOS, which would tell us that information. However, if we can determine what machine models they use (assuming they buy branded hardware), then we will know their chipsets, and from there, we should be able to deduce whether the RAM is ECC or not.
  - ChuckMcM 13y ago
    Actually when I was at NetApp we got pretty good at figuring out ECC failures. Just write a program that allocates 1GB array. Fill it with random data and compute a MD5 hash on it (you can do that fairly quickly). Now recompute the hash once every 5 minutes for a month. Run that on 125 instances. (that ends up being 1 Terabit of data btw) which makes a good detector for cosmic rays if it isn't ECC protected :-).

rbranson 13y ago

This is highly unlikely. I'm on a team that administers an EC2 infrastructure with tens of terabytes of RAM, and I've never seen anything that resembles memory errors. Considering the findings of http://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf http://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf, this would be rather extraordinary.

bigtones 13y ago

Amazon reached out to me via email and stated they do use ECC RAM on all servers.