3 ms·

All good, except most of your mitigating code/tricks cannot be tested reliable to know they actually work.

vinceguidry 10y ago
That's still unnecessarily pessimistic. You can examine the various types of unreliability you're getting from your test techniques to try to drive an understanding about the operating environment. If you have something in front of you, there's really no reason you can't at least try to come to some kind of understanding of it. Then do your best to communicate your findings.
VLM 10y ago
Liberal arts types say engineers are not creative, but they're wrong. Find some creative engineers, it won't be much of a challenge. Write 5 lines of perl that run logical AND and logical OR on a randomly generated bitmap emulating flash memory error rates at 1e-3, 1e-5, 1e-7, then run a billion emulation cycles (depending on code complexity this might take a long time, or maybe not). If you're doing it right, you have the code for the emulator, never use closed source for this kind of work unless you trust it with your life (LOL) so implement statistical error injection in your emulator. Guess what, your CPU adder now has a 1e-5 bit error rate. This is before the dreaded "programmer with a soldering iron" is unleashed. A white noise source is attached to pin D0 of the CPU and levels are increased while behavior of the system is analyzed in the test jig. Simplistically, radiation is linear, so running at 100x intensity for a month is the same as running at design intensity for 100 months. Talk to your local nuclear physicist or radiologist. Especially about neutron activated decay products, LOL. In more detail, if it takes xenon 12 hours to build up to a problem level, you can't just test at 100x intensity for an hour and call it good because 11 hours after the test ends you'll finally have 100x the xenon decay product as a contaminant, but that's a weird situation.
- pdabbadabba 10y ago
  > Liberal arts types say engineers are not creative, but they're wrong. I take your basic point--and it's a good one--that engineering is actually a very creative pursuit, when done well. But like any other high-expertise field, it's difficult for people on the outside to understand how engineers solve problems and, thus, to appreciate the sorts of skills required to do it well. But isn't one of the lessons here that it's probably not a good idea to make gross generalizations about the attitudes and dispositions of people in other disciplines? :) I think "Liberal arts types say engineers are not creative" is probably about as true as "engineers are not creative." There may be some kernel of truth to each, but neither is especially accurate. (The kernel of truth may be that each statement could often be true when relativised to dumb liberal arts types and dumb engineers.)
- mturmon 10y ago
  One implementation of a single-event-upset testbed that I worked with used the valgrind infrastructure to inject faults in running code, purely in software. (Example: http://www.sts.tu-harburg.de/research/fitin.html http://www.sts.tu-harburg.de/research/fitin.html, earlier work: http://dus.jpl.nasa.gov/projects/solar/Papers/granat-abft-svm-09.pdf http://dus.jpl.nasa.gov/projects/solar/Papers/granat-abft-sv...) Your larger point is right: some commenters are giving up on problem analysis or solutions "because the problem is hardware". That implication does not follow. Perhaps people are too used to highly abstracted software systems?
  - sickbeard 10y ago
    I think you guys just misunderstand. The question you should be asking should be "is this this right solution", rather than "can this be done?". If you want to build robust systems, picking the right solution is paramount. In this case the consensus seems to be to fix it in hardware rather than messing out with software (it takes longer to implement a verifiable solution, harder to test, etc).
    - mturmon 10y ago
      I wrote to push back against some sweeping claims that are made nearby, such as "if the hardware is bad, your software will be bad." This is simply not true for many values of "bad". Another comment nearby regarding testing also surrendered prematurely. I worked and published in the area for a few years, so I have a definite POV. I find it appealing that software can, in some cases, make up for the failings of hardware. The topic area is fascinating. It has some of the intellectual aspects of security research: you can get somewhere by breaking through abstraction barriers in surprising ways.

That's still unnecessarily pessimistic. You can examine the various types of unreliability you're getting from your test techniques to try to drive an understanding about the operating environment. If you have something in front of you, there's really no reason you can't at least try to come to some kind of understanding of it. Then do your best to communicate your findings.

VLM 10y ago

Liberal arts types say engineers are not creative, but they're wrong. Find some creative engineers, it won't be much of a challenge. Write 5 lines of perl that run logical AND and logical OR on a randomly generated bitmap emulating flash memory error rates at 1e-3, 1e-5, 1e-7, then run a billion emulation cycles (depending on code complexity this might take a long time, or maybe not). If you're doing it right, you have the code for the emulator, never use closed source for this kind of work unless you trust it with your life (LOL) so implement statistical error injection in your emulator. Guess what, your CPU adder now has a 1e-5 bit error rate. This is before the dreaded "programmer with a soldering iron" is unleashed. A white noise source is attached to pin D0 of the CPU and levels are increased while behavior of the system is analyzed in the test jig. Simplistically, radiation is linear, so running at 100x intensity for a month is the same as running at design intensity for 100 months. Talk to your local nuclear physicist or radiologist. Especially about neutron activated decay products, LOL. In more detail, if it takes xenon 12 hours to build up to a problem level, you can't just test at 100x intensity for an hour and call it good because 11 hours after the test ends you'll finally have 100x the xenon decay product as a contaminant, but that's a weird situation.

pdabbadabba 10y ago
> Liberal arts types say engineers are not creative, but they're wrong. I take your basic point--and it's a good one--that engineering is actually a very creative pursuit, when done well. But like any other high-expertise field, it's difficult for people on the outside to understand how engineers solve problems and, thus, to appreciate the sorts of skills required to do it well. But isn't one of the lessons here that it's probably not a good idea to make gross generalizations about the attitudes and dispositions of people in other disciplines? :) I think "Liberal arts types say engineers are not creative" is probably about as true as "engineers are not creative." There may be some kernel of truth to each, but neither is especially accurate. (The kernel of truth may be that each statement could often be true when relativised to dumb liberal arts types and dumb engineers.)
mturmon 10y ago
One implementation of a single-event-upset testbed that I worked with used the valgrind infrastructure to inject faults in running code, purely in software. (Example: http://www.sts.tu-harburg.de/research/fitin.html http://www.sts.tu-harburg.de/research/fitin.html, earlier work: http://dus.jpl.nasa.gov/projects/solar/Papers/granat-abft-svm-09.pdf http://dus.jpl.nasa.gov/projects/solar/Papers/granat-abft-sv...) Your larger point is right: some commenters are giving up on problem analysis or solutions "because the problem is hardware". That implication does not follow. Perhaps people are too used to highly abstracted software systems?
- sickbeard 10y ago
  I think you guys just misunderstand. The question you should be asking should be "is this this right solution", rather than "can this be done?". If you want to build robust systems, picking the right solution is paramount. In this case the consensus seems to be to fix it in hardware rather than messing out with software (it takes longer to implement a verifiable solution, harder to test, etc).
  - mturmon 10y ago
    I wrote to push back against some sweeping claims that are made nearby, such as "if the hardware is bad, your software will be bad." This is simply not true for many values of "bad". Another comment nearby regarding testing also surrendered prematurely. I worked and published in the area for a few years, so I have a definite POV. I find it appealing that software can, in some cases, make up for the failings of hardware. The topic area is fascinating. It has some of the intellectual aspects of security research: you can get somewhere by breaking through abstraction barriers in surprising ways.