6 ms·

I have the Skullcanyon NUC, and this is not comparable. I've been waiting for someone to offer me ECC in a small form factor. This is the first time I've ever

by daxorid 10y ago

I have the Skullcanyon NUC, and this is not comparable. I've been waiting for someone to offer me ECC in a small form factor. This is the first time I've ever seen it.

ChuckMcM 10y ago
I've got one of these: https://www.supermicro.com/products/motherboard/xeon/qm77/x9spv-m4.cfm https://www.supermicro.com/products/motherboard/xeon/qm77/x9... which is mini-itx. Pretty small form factor. Using an M.2 boot drive. Not very cost effective though.
- ptomato 10y ago
  In the mini-itx space, there's also http://asrock.com/mb/Intel/X99E-ITXac/index.us.asp http://asrock.com/mb/Intel/X99E-ITXac/index.us.asp which is DDR4, not SODIMMS, and your choice of processor, including V4 Xeons. I have one with a E5-2680V4 + 64GB of ram in it. (128 should be feasible with 64GB sticks as far as I know, but as they're about $700 each last I checked I haven't tried it.)
cylinder714 10y ago
Right! The Skull Canyon NUC is brilliant, but the lack of ECC RAM is regrettable.
Matthias247 10y ago
Just interested: What's your use case for ECC? Do you intend to run a server on it and aim for highest stability?
- Declanomous 10y ago
  I'm putting money on something to do with ZFS. Turns out bit flips are significantly more common than had been thought in the past, which makes a error-correcting file system somewhat risky to use with normal non-ecc ram.
  - Someone 10y ago
    But if you use ZFS, chances are you will want to have lots of disk space and lots of disks, too. If so, why buy a small box?
    - Declanomous 10y ago
      Because you are paranoid and also have a small apartment. I have a mini-itx home server build with 16 gb of ram and 4 WD 3tb RED drives. It doesn't run ZFS because I don't have ECC ram in it, but it's the smallest computer I could buy that would fit the hard disks I already owned. If I were starting from scratch I'd probably use 2.5" drives instead, and if I wasn't working for a non-profit, I'd probably use SSDs now that they are about $.27 a gb.
      - Someone 10y ago
        Mini-itx I can understand, but this box has one drive of 1.5TB max (according to http://www.hp.com/go/Z2mini http://www.hp.com/go/Z2mini). You would have to connect any other drives externally. If you do that, the result probably takes more room than that mini-itx.
      - Tomte 10y ago
        ZFS isn't worse than other file systems without ECC RAM. It's just that it's much better with ECC RAM.
  - Tomte 10y ago
    In the safety industry everyone knows that bit flips (SEUs) are way more common than laypeople believe. And that's for embedded microcontrollers and RAM that aren't nearly pushing it to extremely small processes. Let me just quote IEC 61508, where "FIT" means 10e-9 h: Causes of soft errors are: (1) Alpha particles from package decay, (2) Neutrons, (3) external EMI noise, (4) Internal cross-talk. External EMI noise is covered by other requirements of this international standard. A soft error occurs when a radiation event causes enough of a charge disturbance to reverse or flip the data state of a low energized semiconductor memory cell, register, latch, or flip-flop. The error is called “soft” because the circuit itself is not permanently damaged by the radiation. Soft-errors are classified in Single Bit Upsets (SBU) or Single Event Upsets (SEU) and Multi-Bit Upsets (MBU). The soft error rate has been reported (see a) and i) below) to be in a range of 700 Fit/MBit to 1 200 Fit/MBit for (embedded) memories. This is a reference value to be compared with data coming from the silicon process with which the device is implemented. Until recently SBU were considered to be dominant, but the latest forecast (see a) below) reports a growing percentage of MBU of the overall soft-error rate (SER) for technologies from 65 nm down. The following literature and sources give details about soft-errors: a) Altitude SEE Test European Platform (ASTEP) and First Results in CMOS 130 nm SRAM. J-L. Autran, P. Roche, C. Sudre et al. Nuclear Science, IEEE Transactions on Volume 54, Issue 4, Aug. 2007 Page(s):1002 - 1009 b) Radiation-Induced Soft Errors in Advanced Semiconductor Technologies, Robert C. Baumann, Fellow, IEEE, IEEE TRANSACTIONS ON DEVICE AND MATERIALS RELIABILITY, VOL. 5, NO. 3, SEPTEMBER 2005 c) Soft errors' impact on system reliability, Ritesh Mastipuram and Edwin C Wee, Cypress Semiconductor, 2004 d) Trends And Challenges In VLSI Circuit Reliability, C. Costantinescu, Intel, 2003, IEEE Computer Society e) Basic mechanisms and modeling of single-event upset in digital microelectronics, P. E. Dodd and L. W. Massengill, IEEE Trans. Nucl. Sci., vol. 50, no. 3, pp. 583–602, Jun. 2003. f) Destructive single-event effects in semiconductor devices and ICs, F. W. Sexton, IEEE Trans. Nucl. Sci., vol. 50, no. 3, pp. 603–621, Jun. 2003. g) Coming Challenges in Microarchitecture and Architecture, Ronen, Mendelson, Proceedings of the IEEE, Volume 89, Issue 3, Mar 2001 Page(s):325 – 340 h) Scaling and Technology Issues for Soft Error Rates, A Johnston, 4th Annual Research Conference on Reliability Stanford University, October 2000 i) International Technology Roadmap for Semiconductors (ITRS), several papers.
- ChuckMcM 10y ago
  On my current desktop machine (32GB ECC DRAM) I see a single bit error (corrected) probably every 18 to 20 days. There isn't an easy way to tell where that error hit in memory (you could presumably walk it back from the physical chip to the physical address space to the current page table and then back into virtual space and a process but that is waaaaaaaay too much work :-) Without ECC those errors would happen and I'd keep working and they may or not cause a visible symptom. A schematic data base slightly corrupted when it was saved to disk, a Flash image that isn't quite right, a system daemon that crashes and restarts unexpectedly, etc. Not really something I want to take my chances on. Of course if you don't have ECC memory you don't know that your memory is being silently corrupted. For some people they don't care, for me, I do.
  - effie 10y ago
    How are you able to "see" the single bit errors? I had an ECC machine but I couldn't find any indication that ECC was working or that any bit errors happened. Reading this I thought the best I could do is to believe that workstation with ECC modules and Xeon will have functioning ECC: "Unfortunately, we have found that there is no consistent, conclusive way to determine if ECC RAM is working properly. ...We've actually asked Intel, Kingston and Asus over the years for their recommendations for methods to confirm that ECC is working, but we haven't gotten much more back than a blank stare." https://www.pugetsystems.com/labs/articles/How-to-Check-ECC-RAM-Functionality-462/#Conclusion https://www.pugetsystems.com/labs/articles/How-to-Check-ECC-...
    - ChuckMcM 10y ago
      The BIOS has to correct them, you get what is essentially a machine check, the BIOS fixes the error, does the writeback, and in my case logs the fix on the system management (SMB) bus and that shows up in the IPMI system event log[1]. Not sure who the BIOS would tell if there wasn't some sort of BMC chip to hear about it, it might keep an internal log. I'm surprised the board/bios manufacturer couldn't answer the question, it's a pretty straight forward system. [1] They show up as 'SBE' events (single bit error). An 'MBE' event (detectable but not correctable) would presumably result in an unhandled machine check and cause the kernel to reboot.
- roel_v 10y ago
  Not OP, but for me, numerical simulation. I don't have the numbers to 'prove' that corruption is really a serious problem for my use cases, or what the effects would be on my results, but for a few hundred extra, I figured why not go for a Xeon/ECC platform.
jpt4 10y ago
I also have the Skullcanyon, as a laptop replacement machine. None of my previous personal computers have had ECC RAM, but none have had more than 16GB either. What is your use case that benefits from error-correction, if you please?
- ianai 10y ago
  It can mess up graphics work and zfs pools.
  - jclulow 10y ago
    It can mess up _any_ file system. ZFS is, under some conditions, more likely to _tell_ you that you've had a problem. That said, if a bit flips in the wrong place in RAM, even a checksumming file system will not protect you. The file system will happily write out the corrupted data if the checksum is calculated after the bit flip occurs.
- bzbarsky 10y ago
  The #1 consumer use case that would benefit from error-correction would be not having remote exploits due to https://en.wikipedia.org/wiki/Row_hammer https://en.wikipedia.org/wiki/Row_hammer
  - jpt4 10y ago
    Thank you, that is a compelling argument.
    - homarp 10y ago
      the wikipedia page says: " Tests show that simple ECC solutions, providing single-error correction and double-error detection (SECDED) capabilities, are not able to correct or detect all observed disturbance errors because some of them include more than two flipped bits per memory word." http://serverfault.com/questions/674214/what-is-the-rowhammer-dram-bug-and-how-should-i-treat-it http://serverfault.com/questions/674214/what-is-the-rowhamme... adds "ECC effectively turns the Row Hammer exploit into a DOS attack. 1bit errors will be corrected by ECC, and as soon as a non-correctable 2bit error is detected the system will halt (assuming SECDED ECC)."

by daxorid 10y ago

I have the Skullcanyon NUC, and this is not comparable. I've been waiting for someone to offer me ECC in a small form factor. This is the first time I've ever seen it.

ChuckMcM 10y ago

I've got one of these: https://www.supermicro.com/products/motherboard/xeon/qm77/x9spv-m4.cfm https://www.supermicro.com/products/motherboard/xeon/qm77/x9... which is mini-itx. Pretty small form factor. Using an M.2 boot drive. Not very cost effective though.

ptomato 10y ago
In the mini-itx space, there's also http://asrock.com/mb/Intel/X99E-ITXac/index.us.asp http://asrock.com/mb/Intel/X99E-ITXac/index.us.asp which is DDR4, not SODIMMS, and your choice of processor, including V4 Xeons. I have one with a E5-2680V4 + 64GB of ram in it. (128 should be feasible with 64GB sticks as far as I know, but as they're about $700 each last I checked I haven't tried it.)

cylinder714 10y ago

Right! The Skull Canyon NUC is brilliant, but the lack of ECC RAM is regrettable.

Matthias247 10y ago

Just interested: What's your use case for ECC? Do you intend to run a server on it and aim for highest stability?

Declanomous 10y ago
I'm putting money on something to do with ZFS. Turns out bit flips are significantly more common than had been thought in the past, which makes a error-correcting file system somewhat risky to use with normal non-ecc ram.
- Someone 10y ago
  But if you use ZFS, chances are you will want to have lots of disk space and lots of disks, too. If so, why buy a small box?
  - Declanomous 10y ago
    Because you are paranoid and also have a small apartment. I have a mini-itx home server build with 16 gb of ram and 4 WD 3tb RED drives. It doesn't run ZFS because I don't have ECC ram in it, but it's the smallest computer I could buy that would fit the hard disks I already owned. If I were starting from scratch I'd probably use 2.5" drives instead, and if I wasn't working for a non-profit, I'd probably use SSDs now that they are about $.27 a gb.
    - Someone 10y ago
      Mini-itx I can understand, but this box has one drive of 1.5TB max (according to http://www.hp.com/go/Z2mini http://www.hp.com/go/Z2mini). You would have to connect any other drives externally. If you do that, the result probably takes more room than that mini-itx.
    - Tomte 10y ago
      ZFS isn't worse than other file systems without ECC RAM. It's just that it's much better with ECC RAM.
- Tomte 10y ago
  In the safety industry everyone knows that bit flips (SEUs) are way more common than laypeople believe. And that's for embedded microcontrollers and RAM that aren't nearly pushing it to extremely small processes. Let me just quote IEC 61508, where "FIT" means 10e-9 h: Causes of soft errors are: (1) Alpha particles from package decay, (2) Neutrons, (3) external EMI noise, (4) Internal cross-talk. External EMI noise is covered by other requirements of this international standard. A soft error occurs when a radiation event causes enough of a charge disturbance to reverse or flip the data state of a low energized semiconductor memory cell, register, latch, or flip-flop. The error is called “soft” because the circuit itself is not permanently damaged by the radiation. Soft-errors are classified in Single Bit Upsets (SBU) or Single Event Upsets (SEU) and Multi-Bit Upsets (MBU). The soft error rate has been reported (see a) and i) below) to be in a range of 700 Fit/MBit to 1 200 Fit/MBit for (embedded) memories. This is a reference value to be compared with data coming from the silicon process with which the device is implemented. Until recently SBU were considered to be dominant, but the latest forecast (see a) below) reports a growing percentage of MBU of the overall soft-error rate (SER) for technologies from 65 nm down. The following literature and sources give details about soft-errors: a) Altitude SEE Test European Platform (ASTEP) and First Results in CMOS 130 nm SRAM. J-L. Autran, P. Roche, C. Sudre et al. Nuclear Science, IEEE Transactions on Volume 54, Issue 4, Aug. 2007 Page(s):1002 - 1009 b) Radiation-Induced Soft Errors in Advanced Semiconductor Technologies, Robert C. Baumann, Fellow, IEEE, IEEE TRANSACTIONS ON DEVICE AND MATERIALS RELIABILITY, VOL. 5, NO. 3, SEPTEMBER 2005 c) Soft errors' impact on system reliability, Ritesh Mastipuram and Edwin C Wee, Cypress Semiconductor, 2004 d) Trends And Challenges In VLSI Circuit Reliability, C. Costantinescu, Intel, 2003, IEEE Computer Society e) Basic mechanisms and modeling of single-event upset in digital microelectronics, P. E. Dodd and L. W. Massengill, IEEE Trans. Nucl. Sci., vol. 50, no. 3, pp. 583–602, Jun. 2003. f) Destructive single-event effects in semiconductor devices and ICs, F. W. Sexton, IEEE Trans. Nucl. Sci., vol. 50, no. 3, pp. 603–621, Jun. 2003. g) Coming Challenges in Microarchitecture and Architecture, Ronen, Mendelson, Proceedings of the IEEE, Volume 89, Issue 3, Mar 2001 Page(s):325 – 340 h) Scaling and Technology Issues for Soft Error Rates, A Johnston, 4th Annual Research Conference on Reliability Stanford University, October 2000 i) International Technology Roadmap for Semiconductors (ITRS), several papers.
ChuckMcM 10y ago
On my current desktop machine (32GB ECC DRAM) I see a single bit error (corrected) probably every 18 to 20 days. There isn't an easy way to tell where that error hit in memory (you could presumably walk it back from the physical chip to the physical address space to the current page table and then back into virtual space and a process but that is waaaaaaaay too much work :-) Without ECC those errors would happen and I'd keep working and they may or not cause a visible symptom. A schematic data base slightly corrupted when it was saved to disk, a Flash image that isn't quite right, a system daemon that crashes and restarts unexpectedly, etc. Not really something I want to take my chances on. Of course if you don't have ECC memory you don't know that your memory is being silently corrupted. For some people they don't care, for me, I do.
- effie 10y ago
  How are you able to "see" the single bit errors? I had an ECC machine but I couldn't find any indication that ECC was working or that any bit errors happened. Reading this I thought the best I could do is to believe that workstation with ECC modules and Xeon will have functioning ECC: "Unfortunately, we have found that there is no consistent, conclusive way to determine if ECC RAM is working properly. ...We've actually asked Intel, Kingston and Asus over the years for their recommendations for methods to confirm that ECC is working, but we haven't gotten much more back than a blank stare." https://www.pugetsystems.com/labs/articles/How-to-Check-ECC-RAM-Functionality-462/#Conclusion https://www.pugetsystems.com/labs/articles/How-to-Check-ECC-...
  - ChuckMcM 10y ago
    The BIOS has to correct them, you get what is essentially a machine check, the BIOS fixes the error, does the writeback, and in my case logs the fix on the system management (SMB) bus and that shows up in the IPMI system event log[1]. Not sure who the BIOS would tell if there wasn't some sort of BMC chip to hear about it, it might keep an internal log. I'm surprised the board/bios manufacturer couldn't answer the question, it's a pretty straight forward system. [1] They show up as 'SBE' events (single bit error). An 'MBE' event (detectable but not correctable) would presumably result in an unhandled machine check and cause the kernel to reboot.
roel_v 10y ago
Not OP, but for me, numerical simulation. I don't have the numbers to 'prove' that corruption is really a serious problem for my use cases, or what the effects would be on my results, but for a few hundred extra, I figured why not go for a Xeon/ECC platform.

jpt4 10y ago

I also have the Skullcanyon, as a laptop replacement machine. None of my previous personal computers have had ECC RAM, but none have had more than 16GB either. What is your use case that benefits from error-correction, if you please?

ianai 10y ago
It can mess up graphics work and zfs pools.
- jclulow 10y ago
  It can mess up _any_ file system. ZFS is, under some conditions, more likely to _tell_ you that you've had a problem. That said, if a bit flips in the wrong place in RAM, even a checksumming file system will not protect you. The file system will happily write out the corrupted data if the checksum is calculated after the bit flip occurs.
bzbarsky 10y ago
The #1 consumer use case that would benefit from error-correction would be not having remote exploits due to https://en.wikipedia.org/wiki/Row_hammer https://en.wikipedia.org/wiki/Row_hammer
- jpt4 10y ago
  Thank you, that is a compelling argument.
  - homarp 10y ago
    the wikipedia page says: " Tests show that simple ECC solutions, providing single-error correction and double-error detection (SECDED) capabilities, are not able to correct or detect all observed disturbance errors because some of them include more than two flipped bits per memory word." http://serverfault.com/questions/674214/what-is-the-rowhammer-dram-bug-and-how-should-i-treat-it http://serverfault.com/questions/674214/what-is-the-rowhamme... adds "ECC effectively turns the Row Hammer exploit into a DOS attack. 1bit errors will be corrected by ECC, and as soon as a non-correctable 2bit error is detected the system will halt (assuming SECDED ECC)."