5 ms·

Yes, the RAM is ECC. Here is a link: http://www.crucial.com/store/partspecs.aspx?imodule=CT25672BA1339 http://www.crucial.com/store/partspecs.aspx?imodule=CT25

Yes, the RAM is ECC. Here is a link: http://www.crucial.com/store/partspecs.aspx?imodule=CT25672BA1339 http://www.crucial.com/store/partspecs.aspx?imodule=CT25672B... The ECC RAM absolutely does find and corrects problems (we see them in the logs). However, just to be absolutely clear we would not need ECC RAM -> Backblaze checksums EVERYTHING on an end-to-end basis (mostly we use SHA-1). This is so important I cannot stress this highly enough, each and every file and portion of file we store has our own checksum on the end, and we use this all over the place. For example, we pass over the data every week or so reading it, recalculating the checksums, and if a single bit has been thrown we heal it up either from our own copies of the data or ask the client to re-transmit that file or part of that file. At the large amount of data we store, our checksums catch errors at EVERY level - RAM, hard drive, network transmission, everywhere. I suppose consumers just do not notice when a single bit in one of their JPEG photos has been flipped -> one pixel gets every so slightly more red or something. Only one photo changes out of their collection of thousands. But at our crazy numbers of files stored we see it (and fix it) daily.

jbooth 15y ago
Huh, didn't know there was an i3 that supported ECC. Cool.
- RShackleford 15y ago
  Curious, according to Intel, it the i3 doesn't do ECC: http://www.intel.com/support/processors/corei3/sb/CS-031175.htm#10 http://www.intel.com/support/processors/corei3/sb/CS-031175.... But the Supermicro board says it can: http://www.supermicro.com/xeon_3400/Motherboard/X8SIL.cfm http://www.supermicro.com/xeon_3400/Motherboard/X8SIL.cfm The board also supports Xeon processors, so it might be that it only supports ECC with Xeon. However, I didn't see anything in the docs to support this, instead it says "Dual Core processors of Ci3 and Pentium: support ECC UDIMM only" I'm curious since I ended up springing for Xeon in my system specifically for ECC. Now I'm wondering if I made a mistake...
  - wmf 15y ago
    IIRC, some vendors wanted to offer super-low-end dual-core servers but rather than create a dual-core Xeon SKU Intel decided to just not disable ECC on some of the Pentiums and i3s. (As if their branding wasn't f'd enough.) So yeah, i3s support ECC when used in a server board.
    - brianwski 15y ago
      We didn't think we even had a choice! From: http://www.supermicro.com/manuals/motherboard/3420/MNL-1130.pdf http://www.supermicro.com/manuals/motherboard/3420/MNL-1130.... "The X8SIL/X8SIL-F/X8SIL-V supports up to 16GB of DDR3 ECC UDIMM or up to 32GB of ECC DDR3 RDIMM (1333/1066/800 MHz in 4 DIMM slots.)" Below are what we assume are ECC errors from an i3 based Backblaze pod, when this happens it is usually bad RAM (bad enough to crash the pods, replace with fresh RAM repairs it): ### IPMI LOG: #### ipmitool sel elist | grep -v "Fan FAN" 69 | 01/04/2011 | 20:21:52 | Memory | Correctable ECC | Asserted 73 | 01/04/2011 | 21:15:50 | Memory | Correctable ECC | Asserted
db48x 15y ago
Merkle Trees FTW, but it sounds like you've basically reimplemented ZFS. Have you tried using ZFS on one of these?
- brianwski 15y ago
  Earlier we were totally interested in ZFS, as it would replace RAID & LVM as well (and ZFS gets great reviews). But (to my understanding) native ZFS is not available on Linux and we're not really looking to switch to OpenSolaris. ANOTHER option down this line of thinking is switching to btrfs, but we haven't played with it yet. By the way, at Backblaze I've felt like we have had to implement several things that I would have guessed would be standard "off the shelf". One example: When a customer wants to download a restore file with a web browser, are you aware there are no checksums for over-the-network transfers other than the built in (completely unacceptable) 16-bit TCP checksum? You are virtually guaranteed to have an undetected corruption within 30 Gbytes of download, which is basically what we like to call "a totally average customer restore". So Backblaze had to write our own custom reliable, restartable "downloader". It boggles my mind that the whole internet is throwing undetected errors on HTTP downloads and nobody cares to fix the protocol?!! Where the heck is Google, Apple, Facebook, Microsoft, or <insert standards body> defining a standard for web browser downloads larger than a few GBytes?
  - zerosanity 15y ago
    We're using FreeBSD's ZFS support and haven't had any hiccups to complain about.
    - thaumaturgy 15y ago
      We've had a backup server running for a client with FreeBSD's ZFS for about a year now, also with zero hiccups. We first tried to get this kind of stuff working about three or four years ago, and went the OpenSolaris/ZFS route -- what a pain that was. OpenSolaris somehow corrupted its own boot partition one day and just completely refused to start with a totally cryptic error code. The last time we tried Linux for this, it couldn't do it. Maybe that's gotten better though. We also tried DragonflyBSD, but it had hardware support issues, and OpenBSD, but unfortunately OpenBSD just simply cannot do large filesystems. At all. ZFS-on-FreeBSD is the way to go at the moment, I think.

jbooth 15y ago

Huh, didn't know there was an i3 that supported ECC. Cool.

RShackleford 15y ago
Curious, according to Intel, it the i3 doesn't do ECC: http://www.intel.com/support/processors/corei3/sb/CS-031175.htm#10 http://www.intel.com/support/processors/corei3/sb/CS-031175.... But the Supermicro board says it can: http://www.supermicro.com/xeon_3400/Motherboard/X8SIL.cfm http://www.supermicro.com/xeon_3400/Motherboard/X8SIL.cfm The board also supports Xeon processors, so it might be that it only supports ECC with Xeon. However, I didn't see anything in the docs to support this, instead it says "Dual Core processors of Ci3 and Pentium: support ECC UDIMM only" I'm curious since I ended up springing for Xeon in my system specifically for ECC. Now I'm wondering if I made a mistake...
- wmf 15y ago
  IIRC, some vendors wanted to offer super-low-end dual-core servers but rather than create a dual-core Xeon SKU Intel decided to just not disable ECC on some of the Pentiums and i3s. (As if their branding wasn't f'd enough.) So yeah, i3s support ECC when used in a server board.
  - brianwski 15y ago
    We didn't think we even had a choice! From: http://www.supermicro.com/manuals/motherboard/3420/MNL-1130.pdf http://www.supermicro.com/manuals/motherboard/3420/MNL-1130.... "The X8SIL/X8SIL-F/X8SIL-V supports up to 16GB of DDR3 ECC UDIMM or up to 32GB of ECC DDR3 RDIMM (1333/1066/800 MHz in 4 DIMM slots.)" Below are what we assume are ECC errors from an i3 based Backblaze pod, when this happens it is usually bad RAM (bad enough to crash the pods, replace with fresh RAM repairs it): ### IPMI LOG: #### ipmitool sel elist | grep -v "Fan FAN" 69 | 01/04/2011 | 20:21:52 | Memory | Correctable ECC | Asserted 73 | 01/04/2011 | 21:15:50 | Memory | Correctable ECC | Asserted

db48x 15y ago

Merkle Trees FTW, but it sounds like you've basically reimplemented ZFS. Have you tried using ZFS on one of these?

brianwski 15y ago
Earlier we were totally interested in ZFS, as it would replace RAID & LVM as well (and ZFS gets great reviews). But (to my understanding) native ZFS is not available on Linux and we're not really looking to switch to OpenSolaris. ANOTHER option down this line of thinking is switching to btrfs, but we haven't played with it yet. By the way, at Backblaze I've felt like we have had to implement several things that I would have guessed would be standard "off the shelf". One example: When a customer wants to download a restore file with a web browser, are you aware there are no checksums for over-the-network transfers other than the built in (completely unacceptable) 16-bit TCP checksum? You are virtually guaranteed to have an undetected corruption within 30 Gbytes of download, which is basically what we like to call "a totally average customer restore". So Backblaze had to write our own custom reliable, restartable "downloader". It boggles my mind that the whole internet is throwing undetected errors on HTTP downloads and nobody cares to fix the protocol?!! Where the heck is Google, Apple, Facebook, Microsoft, or <insert standards body> defining a standard for web browser downloads larger than a few GBytes?
- zerosanity 15y ago
  We're using FreeBSD's ZFS support and haven't had any hiccups to complain about.
  - thaumaturgy 15y ago
    We've had a backup server running for a client with FreeBSD's ZFS for about a year now, also with zero hiccups. We first tried to get this kind of stuff working about three or four years ago, and went the OpenSolaris/ZFS route -- what a pain that was. OpenSolaris somehow corrupted its own boot partition one day and just completely refused to start with a totally cryptic error code. The last time we tried Linux for this, it couldn't do it. Maybe that's gotten better though. We also tried DragonflyBSD, but it had hardware support issues, and OpenBSD, but unfortunately OpenBSD just simply cannot do large filesystems. At all. ZFS-on-FreeBSD is the way to go at the moment, I think.