4 ms·

There's no argument that the Xeon server is going to be less power-efficient than the RP2 cluster. Before you go out an order a bunch, do keep in mind that the

by ansible 11y ago

There's no argument that the Xeon server is going to be less power-efficient than the RP2 cluster. Before you go out an order a bunch, do keep in mind that the RP2 cluster doesn't support ECC memory and other features often considered necessary in a server environment, like remote management. I'm pleased to see the A1100 does support ECC. Still, it is an interesting test. I look forward to reading more about ARM servers. And I hope the Mill Computing guys will ship something eventually too.

derefr 11y ago
> ECC memory ... remote management ... often considered necessary in a server environment Do note that an idiomatic Erlang cluster (where the distribution of the system is used to achieve fault-tolerance, rather than for higher throughput) doesn't strictly need ECC memory (though, obviously, it wouldn't hurt.) Also, such a system should be plenty manageable on its own—insofar as even an Erlang unikernel makes for a relatively ops-friendly system. There's little "remote management" can do that can't be done from an Erlang remsh. (Erlang even offers easy ways to expose e.g. SNMP MIBs.) If you're talking about being able to re-image the system into being a completely non-Erlang system remotely, that might be hard—but a cluster-of-inexpensive-low-power-nodes setup like this only really has this one niche use-case to begin with, so I'm not sure what else you'd want to change it into.
- CJefferson 11y ago
  Stupid question -- why wouldn't you need ECC? What does Erlang do special? I've found C programs occasionally returning incorrect results with non-ECC memory, as a memory location got corrupted. Does Erlang have some magic to avoid that?
- ansible 11y ago
  ECC can prevent program crashes, but it can also prevent computation of incorrect results. Even if your memory is basically reliable, cosmic rays still can flip bits. As RAM sizes go up and up, your chances of seeing a memory error increase. My point, mainly, if we're talking about costs and power consumption, it isn't quite fair to compare server-grade hardware (which is more expensive than consumer-grade) with a RP2.
- jlouis 11y ago
  An idiomatic Erlang cluster will have great benefit from ECC memory. The problem is that bit-flips can yield the wrong results for a while before the system crashes. And you may not be able to tolerate these wrong results.
spacecowboy_lon 11y ago
And the PI interconnects are much slower so if you cluster is doing any East <> West traffic its going to kill performance.
duaneb 11y ago
> ECC memory...often considered necessary in a server environment This is very much workload-specific. If you cycle your processes out of memory even once a day, the chances of seeing a random gamma ray burst is vanishingly small. Of course, I wouldn't host ZFS or a data store.
- wrigby 11y ago
  In my experience, ECC shows benefits more often when a DIMM starts to fail, not in protecting us from cosmic rays (though I don't think I was able to make a note of the last cosmic ray that flew by me). I've had systems limp along reliably while throwing EDAC messages every few hours. With non-ECC memory, they would have crashed and burned. Of course, if you're seeing EDAC messages, you'd be better advised to just shut down the metal and shift your workload to a different machine until you can replace the DIMMs in question.
- GirlProgrammer 11y ago
  The figures we see is about 1 bit-flip per gigabyte/year. (See https://en.wikipedia.org/wiki/ECC_memory https://en.wikipedia.org/wiki/ECC_memory) That's enough for us to use Xeons and ECC memory in all our production machines, and machines used to produce releases for customers, etc. It's just not worth the risk.
- ansible 11y ago
  Yeah, I don't know that people are doing that on a regular basis. I also think you are underestimating the amount of cosmic ray events that occur on a regular basis. https://en.wikipedia.org/wiki/Cosmic_ray#Effect_on_electronics https://en.wikipedia.org/wiki/Cosmic_ray#Effect_on_electroni...

by ansible 11y ago

derefr 11y ago

> ECC memory ... remote management ... often considered necessary in a server environment Do note that an idiomatic Erlang cluster (where the distribution of the system is used to achieve fault-tolerance, rather than for higher throughput) doesn't strictly need ECC memory (though, obviously, it wouldn't hurt.) Also, such a system should be plenty manageable on its own—insofar as even an Erlang unikernel makes for a relatively ops-friendly system. There's little "remote management" can do that can't be done from an Erlang remsh. (Erlang even offers easy ways to expose e.g. SNMP MIBs.) If you're talking about being able to re-image the system into being a completely non-Erlang system remotely, that might be hard—but a cluster-of-inexpensive-low-power-nodes setup like this only really has this one niche use-case to begin with, so I'm not sure what else you'd want to change it into.

CJefferson 11y ago
Stupid question -- why wouldn't you need ECC? What does Erlang do special? I've found C programs occasionally returning incorrect results with non-ECC memory, as a memory location got corrupted. Does Erlang have some magic to avoid that?
ansible 11y ago
ECC can prevent program crashes, but it can also prevent computation of incorrect results. Even if your memory is basically reliable, cosmic rays still can flip bits. As RAM sizes go up and up, your chances of seeing a memory error increase. My point, mainly, if we're talking about costs and power consumption, it isn't quite fair to compare server-grade hardware (which is more expensive than consumer-grade) with a RP2.
jlouis 11y ago
An idiomatic Erlang cluster will have great benefit from ECC memory. The problem is that bit-flips can yield the wrong results for a while before the system crashes. And you may not be able to tolerate these wrong results.

spacecowboy_lon 11y ago

And the PI interconnects are much slower so if you cluster is doing any East <> West traffic its going to kill performance.

duaneb 11y ago

> ECC memory...often considered necessary in a server environment This is very much workload-specific. If you cycle your processes out of memory even once a day, the chances of seeing a random gamma ray burst is vanishingly small. Of course, I wouldn't host ZFS or a data store.

wrigby 11y ago
In my experience, ECC shows benefits more often when a DIMM starts to fail, not in protecting us from cosmic rays (though I don't think I was able to make a note of the last cosmic ray that flew by me). I've had systems limp along reliably while throwing EDAC messages every few hours. With non-ECC memory, they would have crashed and burned. Of course, if you're seeing EDAC messages, you'd be better advised to just shut down the metal and shift your workload to a different machine until you can replace the DIMMs in question.
GirlProgrammer 11y ago
The figures we see is about 1 bit-flip per gigabyte/year. (See https://en.wikipedia.org/wiki/ECC_memory https://en.wikipedia.org/wiki/ECC_memory) That's enough for us to use Xeons and ECC memory in all our production machines, and machines used to produce releases for customers, etc. It's just not worth the risk.
ansible 11y ago
Yeah, I don't know that people are doing that on a regular basis. I also think you are underestimating the amount of cosmic ray events that occur on a regular basis. https://en.wikipedia.org/wiki/Cosmic_ray#Effect_on_electronics https://en.wikipedia.org/wiki/Cosmic_ray#Effect_on_electroni...