10 ms·

The push for crypto without ecc ram is a nonstop horror show. Software under normal circumstances is remarkably resilient to having its memory corrupted. Howeve

by jart 5y ago

The push for crypto without ecc ram is a nonstop horror show. Software under normal circumstances is remarkably resilient to having its memory corrupted. However crypto algorithms are designed so that a single bit flip effectively changes all the bits in a block. If you chain blocks then a single bit flip in one block destroys all the blocks. I've seen companies like msps go out of business because they were doing crypto with consumer hardware. No one thinks it'll happen to them and once it happens they're usually too dim to even know what happened. Bit flips aren't an act of god you simply need a better computer.

fulafel 5y ago
How did the company go out of business?
ajross 5y ago
> The push for crypto without ecc ram is a nonstop horror show That's a bit hyperbolic. First, ECC doesn't protect the full data chain, you can have a bitflip in a hardware flip flop (or latch open a gate that drains a line, etc...) before the value reaches the memory. Logic is known to glitch too. Second: ECC is mostly designed to protect long term storage in DRAM. Recognize that a cert like this is a very short-term value, it's computed and then transmitted. The failure happened fast, before copies of the correct value were made. That again argues to a failure location other than a DRAM cell. But mostly... this isn't the end of the world. This is a failed cert, which is a failure that can be reasonably easily handled by manual intervention. There have been many other mistaken certs distributed that had to be dealt with manually: they can be the result of software bugs, they can be generated with the wrong keys, the dates can be set wrong, they can be maliciously issued, etc... The system is designed to include manual validation in the loop and it works. So is ECC a good idea? Of course. Does it magically fix problems like this? No. Is this really a "nonstop horror show"? Not really, we're doing OK.
- ineedasername 5y ago
  That's a bit hyperbolic. Would ECC have avoided the issue in this case? If so then it's hard not to agree that is should be considered a minimum standard. It looks like Yeti 2022 isn't going to survive this intact, and while they can resolve the issues in other ways, not everyone will always be so fortunate and ECC is a relatively small step to avoid a larger problem.
- lima 5y ago
  > This is a failed cert, which is a failure that can be reasonably easily handled by manual intervention. This isn't a misissued cert that can be revoked, it permanently breaks the CT log in question since the error propagates down the chain.
  - tialaramex 5y ago
    Yes, this kills Yeti 2022. There's a bug referenced which refers to an earlier incident where a bitflip happened in the logged certificate data. That was just fixed. Overwrite with the bit flipped back, and everything checks out from then onwards. But in this case it's the hash record which was flipped, which unavoidably taints the log from that point on. Verifiers will forever say that Yeti 2022 is broken, and so it had to be locked read-only and taken out of service. Fortunately, since modern logs are anyway sharded by year of expiry, Yeti 2023 already existed and is unaffected. DigiCert, as log operator, could decide to just change criteria for Yeti 2023 to be "also 2022 is fine" and I believe they may already have done so in fact. Alternatively they could spin up a new mythical creature series. They have Yeti (a creature believed to live in the high mountains and maybe forests) and Nessie (a creature believed to live in a lake in Scotland) but there are plenty more I'm sure.
  - ajross 5y ago
    It doesn't break anything that I can see (though I'm no expert on the particular protocol). Our ability to detect bad certs isn't compromised, precisely because this was noticed by human beings who can adjust the process going forward to work around this. Really the bigger news here seems to be a software bug: the CT protocol wasn't tolerant of bad input data and was trusting actors that clearly can't be trusted fully. Here the "black hat" was a hardware glitch, but it's not hard to imagine a more nefarious trick.
    - zinekeller 5y ago
      Your statement is, to be frank, non-sensical. The protocol itself isn't broken, at least for previous Yeti instances, certificate data are correctly parsed and rejected.* In this instance, it seems that the data is verified already pre-signing BUT was flipped mid-signing. This isn't the fault of how CT was designed but rather a hardware failure that requires correction there. (Or at least that's the likely explanation, it could be a software bug° but it will be a very consistent and obvious behaviour if it is indeed a software bug.) On the issue of subsequent invalidation of all submitted certificates, this is prevented by submitting to at least 3 different entities (as of now, there's a discussion whether if this should be increased), so if a log is subsequently found to be corrupted, the operator can send a "operator error" signal to the browser, and any tampered logs are blacklisted from browsers. (Note that all operators of CT lists are members of CA/B forum, at least as of 2020. In standardisation phase, some individuals have operated their own servers but this is no longer true.) * Note that if the cert details are nonsensical but technically valid, it is still accepted by design, because all pre-certificates are countersigned by the intermediate signer (which the CT log operator checks from known roots). If the intermediate is compromised, then the correct response is obviously a revocation and possibly distrust. ° At least the human-induced variety, you could say that this incident is technically a software bug that occurred due to a hardware fault.
- noxer 5y ago
  If a system is critical it should run on multiple machines in multiple locations and "sync with checks" kinda like the oh so hated and totally useless blockchains. Then if such a bit-flip would occur it would never occur on all machines at the same time in the same data. And on top of that you could easy make the system fix itself if something like that happens (simply assume the majority of nodes didn't have the bit-flip) or in worst case scenario it could at least stop rather than making "wrong progress" I have no clue what this particular log need for "throughput specs" but I assume it would be easily achievable with current DLT.
  - ineedasername 5y ago
    Safety critical systems are not a good fit for a blockchain-based resolution to the Byzantine General problem. Safety critical systems need extremely low latency to resolve the conflict fast. So blockchain is not going to be an appropriate choice for all critical applications when there are multiple low-latency solutions for BFT at a very low ms latency and IIRC, microseconds for avionics systems.
    - noxer 5y ago
      Not sure what you mean with "safety critical". I made no such assumption. Also since the topic is about a (write-only) log it probably doesn't need such low latency. What it more likely needed is final states so once a entry is made and accepted it must be final and ofc correct. DLTs can do this distributed and self-fixing i.e. a node that tries to add fault data is overruled and can never get a confirmation for a final state that later would not be valid. Getting the whole decentral system to "agree" will never be fast unless we have quantum tech. There is simply no way servers around the glob could communicate in microseconds even if all communication would be the speed of light and processing would be instant. It would still take time. In reality such system need seconds which is often totally fine. As long as everyone only relies on the data that has been declared final.
      - ineedasername 5y ago
        You said If a system is critical I thought you were making a more general statement about all critical systems, that's all. And since many critical systems have a safety factor in play, I wanted to distinguish them as not always being a good target for a Blockchain solution to the problems of consensus. Blockchain is a very interesting solution to the problem of obtaining consensus in the face of imperfect inputs, there are other options so, like anything else, you choose the right tool for the job. My own view is that-- given other established protocols, blockchain is going to be overkill for dealing with some types of fault tolerance. It is a very good fit for applications where you want to minimize relying on the trust of humans. (And other areas too, but right now I'm just speaking of the narrow context of consensus amid inconsistent inputs)
- aseipp 5y ago
  This is just learned helplessness because Intel were stingy as shit for over a decade and wanted to segregate their product lines. Error correction is literally prevalent in every single part of every PHY layer in a modern stack, it is an absolute must, and the lack of error correction in RAM is, without question, a ridiculous gap that should have never been allowed in the first place in any modern machine, especially given that density and bandwidth keeps increasing and will continue to do so. When you are designing these systems, you have two options: you either use error correcting codes and increase channel bandwidth to compensate for them (as a result of injected noise, which is unavoidable), or you lower the transfer rate so much as to be infeasible to use, while also avoiding as much noise as you can. Guess what's happening to RAM? It isn't getting slower or less dense. The error rate is only going to increase. The people designing this stuff aren't idiots. That's why literally every other layer of your system builds in error correction. Software people do not understand this because they prefer to believe in magic due to the fact all of these abstractions result in stable systems, I guess. All of the talk of hardware flip flops and all that shit is an irrelevant deflection. Doesn't matter. It's just water carrying and post-hoc justification because, again, Intel decided that consumers didn't actually need it a decade ago, and everyone followed suit. They've been proven wrong repeatedly. Complex systems are built to resist failure. They wouldn't work otherwise. By definition, if a failure occurs, it's because it passed multiple safeguards that were already in place. Basic systems theory. Let's actually try building more safeguards instead of rationalizing their absence.
  - mtkd 5y ago
    'stingy as shit' or maximising short-term shareholder value -- the hardware is possibly not the only broken model here
    - aseipp 5y ago
      You're probably right that I'm giving them a bit too much credit on that note. Ceterum censeo, and so on.
  - goodpoint 5y ago
    > By definition, if a failure occurs, it's because it passed multiple safeguards that were already in place. Having worked on a good bunch of critical system, there aren't multiple safeguards in most hardware. E.g. a multiplication error in a core will not be detected by an external device. Or a bit flip when reading cache, or from a storage device. Very often the only real safeguard is to do the whole computation twice on two different hosts. I would rather have many low-reliability hosts and do computation twice than few high-reliability and very expensive host. Unfortunately the software side is really lagging behind when it comes to reproducible computing. Reproducible builds are a good step in that direction and it took many decades to get there.
- hda111 5y ago
  Using ECC is a no-brainer. Even the Raspberry Pi 4 has ECC RAM. It’s not particularly expensive and only a artificial limitation Intel has introduced for consumer products.
  - dijit 5y ago
    Thought you were wrong. Checked it out. You were right.. Under "1.2. Features" https://datasheets.raspberrypi.org/cm4/cm4-datasheet.pdf https://datasheets.raspberrypi.org/cm4/cm4-datasheet.pdf https://datasheets.raspberrypi.org/rpi4/raspberry-pi-4-product-brief.pdf https://datasheets.raspberrypi.org/rpi4/raspberry-pi-4-produ...
    - jml7c5 5y ago
      (See my reply to parent.)
  - jml7c5 5y ago
    The Pi 4 uses DRAM with on-die ECC, which AFAIK does not provide any means of reporting errors (corrected or uncorrected) to the SoC's memory controller. It is effectively a cost-saving measure to improve DRAM yields. As such, it does little to guarantee that there are no memory errors.
- test_epsilon 5y ago
  > That's a bit hyperbolic. I agree. That said, > First, ECC doesn't protect the full data chain, you can have a bitflip in a hardware flip flop (or latch open a gate that drains a line, etc...) before the value reaches the memory. Logic is known to glitch too. Of course. DRAM ECC protects against errors in the DRAM cells. That doesn't mean other components don't have other strategies for reducing errors which can form a complete chain. Latches and arrays and register files often have parity (where data can be reconstructed) or ECC bits, or use low level circuits that themselves are hard or redundant enough to achieve a particular target UBER for the full system. > Second: ECC is mostly designed to protect long term storage in DRAM. Recognize that a cert like this is a very short-term value, it's computed and then transmitted. The failure happened fast, before copies of the correct value were made. That again argues to a failure location other than a DRAM cell. Not necessarily. Cells that are sitting idle other than refresh have certain error profiles, but ones under constant access. Particularly "idle" cells that are in fact being disturbed by adjacent accesses certainly have a non-zero error profile and need ECC too. My completely anecdotal guess would be this error is at least an order of magnitude more likely to have occurred in non-ECC memory (if that's what was being used) rather than any other path to the CPU or cache or logic on the CPU itself.
lima 5y ago
Distributed, byzantine fault tolerant state machines solve that. At worst, a single node will go out of sync.
- noxer 5y ago
  This is a great way to say "blockchain" without getting guaranteed down votes ;)
  - matthewdgreen 5y ago
    The poster isn’t wrong. An entire chain shouldn’t die because of a memory error in one node.
  - eitland 5y ago
    You made me smile, however: I'm one of those who can easily downvote blockchain stuff mercilessly. It is not reflexively though: I reserve it for dumb ideas, it just so happens that most blockchain ideas I see come off as dumb and/or as an attempted rip off.
  - InspiredIdiot 5y ago
    Yes, and also contrary to Wikipedia it provides a solid use case for private blockchain https://en.wikipedia.org/wiki/Blockchain#Disadvantages_of_private_blockchain https://en.wikipedia.org/wiki/Blockchain#Disadvantages_of_pr... I think it is instructive to ask "why doesn't this mention guarantee downvotes?" because I don't think it's just cargo-culting. I doubt that many of those objecting to blockchain are objecting to byzantine fault tolerance, DHT, etc. Very high resource usage in the cost function, the ledger being public and permanent (long term privacy risk), negative externalities related to its use in a currency... These are commonly the objections I have and hear. And they are inapplicable. Extending what the Wikipedia article says, it's basically glorified database replication. But it also replicates and verifies the calculation to get to that data so it provides far greater fault tolerance. But since it is private you get to throw out the adversarial model (mostly the cost function) and assume failures are accidental, not malicious. It makes the problem simpler and lowers the stakes versus using blockchain for a global trustless digital currency so I don't think we should be surprised that it engenders less controversy.
EthanHeilman 5y ago
>If you chain blocks then a single bit flip in one block destroys all the blocks. I've seen companies like msps go out of business because they were doing crypto with consumer hardware. If it is caused by a single bitflip you know the block in which that bitflip occurred and can try each bit until you find the right bit. This is an embarrassing parallel problem. Let's say you need to search 1 GB space for a single bit flip. That only requires that you test 8 billion bit flips. Given the merklized nature of most crypto, you will probably be searching a space far smaller than 1 GB. >Bit flips aren't an act of god you simply need a better computer. Rather than using hardware ECC, you could implement ECC in software. I think hardware ECC is good idea, but you aren't screwed if you don't use it. The big threat here is not the occasional random bit flips, but adversary caused targeted bit flips since adversaries can bit flip software state that won't cause detectable failures but will cause hard to detect security failures.
ericbarrett 5y ago
In my younger days I worked in customer support for a storage company. A customer had one of our products crash (kernel panic) at a sensitive time. We had a couple of crack engineers I used to hover around to try to pick up gdb tricks so I followed this case with interest—it was a new and unexpected panic. Turns out the crash was caused by the processor taking the wrong branch. Something like this (it wasn’t Intel but you get the picture): test edx, edx jnz $something_that_expects_nonzero_edx Well, edx was 0, but the CPU jumped anyway. So yeah, sometimes ECC isn’t enough. If you’re really paranoid (crypto, milsec) you should execute multiple times and sanity-test.
- astrange 5y ago
  I saw a kernel panic because of this _yesterday_. (Crash where the failing instruction in memory was 1 bit different from on disk.)
  - ericbarrett 5y ago
    Sounds like a regular RAM bitflip, doesn't it?
    - astrange 5y ago
      Sure, but even with those you'd usually expect it to happen to data, which is a lot larger.
omegalulw 5y ago
> Software under normal circumstances is remarkably resilient to having its memory corrupted Not really? What you are saying applies to anything that uses hashing of some sort, where the goal by design is to have completely different outputs even with a single bit flip. And "resilience" is not a precise enough term? Is just recovering after errors due to flips? Or is it guaranteeing that the operation will yield correct output (which implies not crashing)? The latter is far harder.
morelikeborelax 5y ago
I used to sell workstations and servers years ago and trying to convince people they needed ECC Ram and that it was just insurance for (often) just the price of the extra chips on the DIMMs was a nightmare. The amount of uninformed and inexperienced counter arguments online suggesting it was purely Intel seeking extra money (even though they didn't sell RAM) was ridiculous. I never understood why there was so much push back from the consumer world commenting on something they had no idea about. Similar arguments for why would you ever need xxGB of RAM while also condemning the incorrect 640kb RAM Bill Gates comment.
agilob 5y ago
>companies like msps go out of business because they were doing crypto with consumer hardware any story on this? can you elaborate or add something to read about it?

by jart 5y ago

fulafel 5y ago

How did the company go out of business?

ajross 5y ago

> The push for crypto without ecc ram is a nonstop horror show That's a bit hyperbolic. First, ECC doesn't protect the full data chain, you can have a bitflip in a hardware flip flop (or latch open a gate that drains a line, etc...) before the value reaches the memory. Logic is known to glitch too. Second: ECC is mostly designed to protect long term storage in DRAM. Recognize that a cert like this is a very short-term value, it's computed and then transmitted. The failure happened fast, before copies of the correct value were made. That again argues to a failure location other than a DRAM cell. But mostly... this isn't the end of the world. This is a failed cert, which is a failure that can be reasonably easily handled by manual intervention. There have been many other mistaken certs distributed that had to be dealt with manually: they can be the result of software bugs, they can be generated with the wrong keys, the dates can be set wrong, they can be maliciously issued, etc... The system is designed to include manual validation in the loop and it works. So is ECC a good idea? Of course. Does it magically fix problems like this? No. Is this really a "nonstop horror show"? Not really, we're doing OK.

ineedasername 5y ago
That's a bit hyperbolic. Would ECC have avoided the issue in this case? If so then it's hard not to agree that is should be considered a minimum standard. It looks like Yeti 2022 isn't going to survive this intact, and while they can resolve the issues in other ways, not everyone will always be so fortunate and ECC is a relatively small step to avoid a larger problem.
lima 5y ago
> This is a failed cert, which is a failure that can be reasonably easily handled by manual intervention. This isn't a misissued cert that can be revoked, it permanently breaks the CT log in question since the error propagates down the chain.
- tialaramex 5y ago
  Yes, this kills Yeti 2022. There's a bug referenced which refers to an earlier incident where a bitflip happened in the logged certificate data. That was just fixed. Overwrite with the bit flipped back, and everything checks out from then onwards. But in this case it's the hash record which was flipped, which unavoidably taints the log from that point on. Verifiers will forever say that Yeti 2022 is broken, and so it had to be locked read-only and taken out of service. Fortunately, since modern logs are anyway sharded by year of expiry, Yeti 2023 already existed and is unaffected. DigiCert, as log operator, could decide to just change criteria for Yeti 2023 to be "also 2022 is fine" and I believe they may already have done so in fact. Alternatively they could spin up a new mythical creature series. They have Yeti (a creature believed to live in the high mountains and maybe forests) and Nessie (a creature believed to live in a lake in Scotland) but there are plenty more I'm sure.
- ajross 5y ago
  It doesn't break anything that I can see (though I'm no expert on the particular protocol). Our ability to detect bad certs isn't compromised, precisely because this was noticed by human beings who can adjust the process going forward to work around this. Really the bigger news here seems to be a software bug: the CT protocol wasn't tolerant of bad input data and was trusting actors that clearly can't be trusted fully. Here the "black hat" was a hardware glitch, but it's not hard to imagine a more nefarious trick.
  - zinekeller 5y ago
    Your statement is, to be frank, non-sensical. The protocol itself isn't broken, at least for previous Yeti instances, certificate data are correctly parsed and rejected.* In this instance, it seems that the data is verified already pre-signing BUT was flipped mid-signing. This isn't the fault of how CT was designed but rather a hardware failure that requires correction there. (Or at least that's the likely explanation, it could be a software bug° but it will be a very consistent and obvious behaviour if it is indeed a software bug.) On the issue of subsequent invalidation of all submitted certificates, this is prevented by submitting to at least 3 different entities (as of now, there's a discussion whether if this should be increased), so if a log is subsequently found to be corrupted, the operator can send a "operator error" signal to the browser, and any tampered logs are blacklisted from browsers. (Note that all operators of CT lists are members of CA/B forum, at least as of 2020. In standardisation phase, some individuals have operated their own servers but this is no longer true.) * Note that if the cert details are nonsensical but technically valid, it is still accepted by design, because all pre-certificates are countersigned by the intermediate signer (which the CT log operator checks from known roots). If the intermediate is compromised, then the correct response is obviously a revocation and possibly distrust. ° At least the human-induced variety, you could say that this incident is technically a software bug that occurred due to a hardware fault.
noxer 5y ago
If a system is critical it should run on multiple machines in multiple locations and "sync with checks" kinda like the oh so hated and totally useless blockchains. Then if such a bit-flip would occur it would never occur on all machines at the same time in the same data. And on top of that you could easy make the system fix itself if something like that happens (simply assume the majority of nodes didn't have the bit-flip) or in worst case scenario it could at least stop rather than making "wrong progress" I have no clue what this particular log need for "throughput specs" but I assume it would be easily achievable with current DLT.
- ineedasername 5y ago
  Safety critical systems are not a good fit for a blockchain-based resolution to the Byzantine General problem. Safety critical systems need extremely low latency to resolve the conflict fast. So blockchain is not going to be an appropriate choice for all critical applications when there are multiple low-latency solutions for BFT at a very low ms latency and IIRC, microseconds for avionics systems.
  - noxer 5y ago
    Not sure what you mean with "safety critical". I made no such assumption. Also since the topic is about a (write-only) log it probably doesn't need such low latency. What it more likely needed is final states so once a entry is made and accepted it must be final and ofc correct. DLTs can do this distributed and self-fixing i.e. a node that tries to add fault data is overruled and can never get a confirmation for a final state that later would not be valid. Getting the whole decentral system to "agree" will never be fast unless we have quantum tech. There is simply no way servers around the glob could communicate in microseconds even if all communication would be the speed of light and processing would be instant. It would still take time. In reality such system need seconds which is often totally fine. As long as everyone only relies on the data that has been declared final.
    - ineedasername 5y ago
      You said If a system is critical I thought you were making a more general statement about all critical systems, that's all. And since many critical systems have a safety factor in play, I wanted to distinguish them as not always being a good target for a Blockchain solution to the problems of consensus. Blockchain is a very interesting solution to the problem of obtaining consensus in the face of imperfect inputs, there are other options so, like anything else, you choose the right tool for the job. My own view is that-- given other established protocols, blockchain is going to be overkill for dealing with some types of fault tolerance. It is a very good fit for applications where you want to minimize relying on the trust of humans. (And other areas too, but right now I'm just speaking of the narrow context of consensus amid inconsistent inputs)
aseipp 5y ago
This is just learned helplessness because Intel were stingy as shit for over a decade and wanted to segregate their product lines. Error correction is literally prevalent in every single part of every PHY layer in a modern stack, it is an absolute must, and the lack of error correction in RAM is, without question, a ridiculous gap that should have never been allowed in the first place in any modern machine, especially given that density and bandwidth keeps increasing and will continue to do so. When you are designing these systems, you have two options: you either use error correcting codes and increase channel bandwidth to compensate for them (as a result of injected noise, which is unavoidable), or you lower the transfer rate so much as to be infeasible to use, while also avoiding as much noise as you can. Guess what's happening to RAM? It isn't getting slower or less dense. The error rate is only going to increase. The people designing this stuff aren't idiots. That's why literally every other layer of your system builds in error correction. Software people do not understand this because they prefer to believe in magic due to the fact all of these abstractions result in stable systems, I guess. All of the talk of hardware flip flops and all that shit is an irrelevant deflection. Doesn't matter. It's just water carrying and post-hoc justification because, again, Intel decided that consumers didn't actually need it a decade ago, and everyone followed suit. They've been proven wrong repeatedly. Complex systems are built to resist failure. They wouldn't work otherwise. By definition, if a failure occurs, it's because it passed multiple safeguards that were already in place. Basic systems theory. Let's actually try building more safeguards instead of rationalizing their absence.
- mtkd 5y ago
  'stingy as shit' or maximising short-term shareholder value -- the hardware is possibly not the only broken model here
  - aseipp 5y ago
    You're probably right that I'm giving them a bit too much credit on that note. Ceterum censeo, and so on.
- goodpoint 5y ago
  > By definition, if a failure occurs, it's because it passed multiple safeguards that were already in place. Having worked on a good bunch of critical system, there aren't multiple safeguards in most hardware. E.g. a multiplication error in a core will not be detected by an external device. Or a bit flip when reading cache, or from a storage device. Very often the only real safeguard is to do the whole computation twice on two different hosts. I would rather have many low-reliability hosts and do computation twice than few high-reliability and very expensive host. Unfortunately the software side is really lagging behind when it comes to reproducible computing. Reproducible builds are a good step in that direction and it took many decades to get there.
hda111 5y ago
Using ECC is a no-brainer. Even the Raspberry Pi 4 has ECC RAM. It’s not particularly expensive and only a artificial limitation Intel has introduced for consumer products.
- dijit 5y ago
  Thought you were wrong. Checked it out. You were right.. Under "1.2. Features" https://datasheets.raspberrypi.org/cm4/cm4-datasheet.pdf https://datasheets.raspberrypi.org/cm4/cm4-datasheet.pdf https://datasheets.raspberrypi.org/rpi4/raspberry-pi-4-product-brief.pdf https://datasheets.raspberrypi.org/rpi4/raspberry-pi-4-produ...
  - jml7c5 5y ago
    (See my reply to parent.)
- jml7c5 5y ago
  The Pi 4 uses DRAM with on-die ECC, which AFAIK does not provide any means of reporting errors (corrected or uncorrected) to the SoC's memory controller. It is effectively a cost-saving measure to improve DRAM yields. As such, it does little to guarantee that there are no memory errors.
test_epsilon 5y ago
> That's a bit hyperbolic. I agree. That said, > First, ECC doesn't protect the full data chain, you can have a bitflip in a hardware flip flop (or latch open a gate that drains a line, etc...) before the value reaches the memory. Logic is known to glitch too. Of course. DRAM ECC protects against errors in the DRAM cells. That doesn't mean other components don't have other strategies for reducing errors which can form a complete chain. Latches and arrays and register files often have parity (where data can be reconstructed) or ECC bits, or use low level circuits that themselves are hard or redundant enough to achieve a particular target UBER for the full system. > Second: ECC is mostly designed to protect long term storage in DRAM. Recognize that a cert like this is a very short-term value, it's computed and then transmitted. The failure happened fast, before copies of the correct value were made. That again argues to a failure location other than a DRAM cell. Not necessarily. Cells that are sitting idle other than refresh have certain error profiles, but ones under constant access. Particularly "idle" cells that are in fact being disturbed by adjacent accesses certainly have a non-zero error profile and need ECC too. My completely anecdotal guess would be this error is at least an order of magnitude more likely to have occurred in non-ECC memory (if that's what was being used) rather than any other path to the CPU or cache or logic on the CPU itself.

lima 5y ago

Distributed, byzantine fault tolerant state machines solve that. At worst, a single node will go out of sync.

noxer 5y ago
This is a great way to say "blockchain" without getting guaranteed down votes ;)
- matthewdgreen 5y ago
  The poster isn’t wrong. An entire chain shouldn’t die because of a memory error in one node.
- eitland 5y ago
  You made me smile, however: I'm one of those who can easily downvote blockchain stuff mercilessly. It is not reflexively though: I reserve it for dumb ideas, it just so happens that most blockchain ideas I see come off as dumb and/or as an attempted rip off.
- InspiredIdiot 5y ago
  Yes, and also contrary to Wikipedia it provides a solid use case for private blockchain https://en.wikipedia.org/wiki/Blockchain#Disadvantages_of_private_blockchain https://en.wikipedia.org/wiki/Blockchain#Disadvantages_of_pr... I think it is instructive to ask "why doesn't this mention guarantee downvotes?" because I don't think it's just cargo-culting. I doubt that many of those objecting to blockchain are objecting to byzantine fault tolerance, DHT, etc. Very high resource usage in the cost function, the ledger being public and permanent (long term privacy risk), negative externalities related to its use in a currency... These are commonly the objections I have and hear. And they are inapplicable. Extending what the Wikipedia article says, it's basically glorified database replication. But it also replicates and verifies the calculation to get to that data so it provides far greater fault tolerance. But since it is private you get to throw out the adversarial model (mostly the cost function) and assume failures are accidental, not malicious. It makes the problem simpler and lowers the stakes versus using blockchain for a global trustless digital currency so I don't think we should be surprised that it engenders less controversy.

EthanHeilman 5y ago

>If you chain blocks then a single bit flip in one block destroys all the blocks. I've seen companies like msps go out of business because they were doing crypto with consumer hardware. If it is caused by a single bitflip you know the block in which that bitflip occurred and can try each bit until you find the right bit. This is an embarrassing parallel problem. Let's say you need to search 1 GB space for a single bit flip. That only requires that you test 8 billion bit flips. Given the merklized nature of most crypto, you will probably be searching a space far smaller than 1 GB. >Bit flips aren't an act of god you simply need a better computer. Rather than using hardware ECC, you could implement ECC in software. I think hardware ECC is good idea, but you aren't screwed if you don't use it. The big threat here is not the occasional random bit flips, but adversary caused targeted bit flips since adversaries can bit flip software state that won't cause detectable failures but will cause hard to detect security failures.

ericbarrett 5y ago

In my younger days I worked in customer support for a storage company. A customer had one of our products crash (kernel panic) at a sensitive time. We had a couple of crack engineers I used to hover around to try to pick up gdb tricks so I followed this case with interest—it was a new and unexpected panic. Turns out the crash was caused by the processor taking the wrong branch. Something like this (it wasn’t Intel but you get the picture): test edx, edx jnz $something_that_expects_nonzero_edx Well, edx was 0, but the CPU jumped anyway. So yeah, sometimes ECC isn’t enough. If you’re really paranoid (crypto, milsec) you should execute multiple times and sanity-test.

astrange 5y ago
I saw a kernel panic because of this _yesterday_. (Crash where the failing instruction in memory was 1 bit different from on disk.)
- ericbarrett 5y ago
  Sounds like a regular RAM bitflip, doesn't it?
  - astrange 5y ago
    Sure, but even with those you'd usually expect it to happen to data, which is a lot larger.

omegalulw 5y ago

> Software under normal circumstances is remarkably resilient to having its memory corrupted Not really? What you are saying applies to anything that uses hashing of some sort, where the goal by design is to have completely different outputs even with a single bit flip. And "resilience" is not a precise enough term? Is just recovering after errors due to flips? Or is it guaranteeing that the operation will yield correct output (which implies not crashing)? The latter is far harder.

morelikeborelax 5y ago

I used to sell workstations and servers years ago and trying to convince people they needed ECC Ram and that it was just insurance for (often) just the price of the extra chips on the DIMMs was a nightmare. The amount of uninformed and inexperienced counter arguments online suggesting it was purely Intel seeking extra money (even though they didn't sell RAM) was ridiculous. I never understood why there was so much push back from the consumer world commenting on something they had no idea about. Similar arguments for why would you ever need xxGB of RAM while also condemning the incorrect 640kb RAM Bill Gates comment.

agilob 5y ago

>companies like msps go out of business because they were doing crypto with consumer hardware any story on this? can you elaborate or add something to read about it?