4 ms·

We do cryptocurrency mining on an industrial scale and constantly see single bit errors from hardware that is brand-new without modifications. That's very surp

by n00b101 10y ago

We do cryptocurrency mining on an industrial scale and constantly see single bit errors from hardware that is brand-new without modifications. That's very surprising and interesting. How do you detect these single bit errors?

AlsoN00b 10y ago
Not OP but a lot of this seems pretty believable and easy to detect to me? Dont the GPU specs allow for a certain lossiness in the math? Or like, at least they don't conform to IEEE 794 float specs w/ regard to order of operations, precision, degradation, etc. So like, do a shitload of math ops in a glsl shader with a deterministic outcome, render the result to a texture, take the texture back and make sure the RGBA values match bit for bit with the numbers you expected? Or to detect single bit errors in the gpu's local memory or caching just attach textures, read data into them, read it back, render back, etc etc.
optimiz3 10y ago
Detection of false positives: Run candidate solutions through the CPU Detection of false negatives: Compare solution distribution and frequency to expected models; switch to debug kernels if outside tolerance. However, this works because the mining problem space is stateless and follows strict mathematically predictable models. A DB is stateful and the answers generally can't be verified without consulting a secondary copy, which is why I'm super curious how they would engineer correctness and reliability in a cost-effective way using GPUs.
- n00b101 10y ago
  That's very interesting. Have you collected statistical data on these bit errors? Is it always a single bit error? I'm assuming you are using GeForce cards and not Tesla cards which have an ECC memory protection mode? I've tried to collect some statistics on GPU memory errors rates but have found them to be normally extremely rare. The only time I've reproducibly seen them is due to faulty hardware, where the errors become highly reproducible and the GPU needs replacement. The other theoretical cause of bit flips is supposed to be random errors due to cosmic radiation but I've never been able to observe that using memory testing software (though I did only run the experiments in AWS). Could it be that you have faulty or low grade GPUs? I assume these are all low-cost OEM parts, given your application? Or maybe there's something odd about your data center environment? Regarding the GPU database application, I think the answer is to just use the Tesla grade GPU with ECC memory enabled.
  - optimiz3 10y ago
    Generally we prefer AMD cards as most (profitable) mining functions are memory bandwidth dominated. Usually it's a shader unit that gets unstable in the 70-80C range (note that most silicon is rated for higher ranges). AMD's hardware specifications are more open too which lets one build your own shader compilers and get direct access to the iron.
- mtweak 10y ago
  We've been working on an interposing library for guaranteeing GPU computation and it would be great to get your feedback. Any chance we sync up? My email is in my profile.

by n00b101 10y ago

AlsoN00b 10y ago

Not OP but a lot of this seems pretty believable and easy to detect to me? Dont the GPU specs allow for a certain lossiness in the math? Or like, at least they don't conform to IEEE 794 float specs w/ regard to order of operations, precision, degradation, etc. So like, do a shitload of math ops in a glsl shader with a deterministic outcome, render the result to a texture, take the texture back and make sure the RGBA values match bit for bit with the numbers you expected? Or to detect single bit errors in the gpu's local memory or caching just attach textures, read data into them, read it back, render back, etc etc.

optimiz3 10y ago

Detection of false positives: Run candidate solutions through the CPU Detection of false negatives: Compare solution distribution and frequency to expected models; switch to debug kernels if outside tolerance. However, this works because the mining problem space is stateless and follows strict mathematically predictable models. A DB is stateful and the answers generally can't be verified without consulting a secondary copy, which is why I'm super curious how they would engineer correctness and reliability in a cost-effective way using GPUs.

n00b101 10y ago
That's very interesting. Have you collected statistical data on these bit errors? Is it always a single bit error? I'm assuming you are using GeForce cards and not Tesla cards which have an ECC memory protection mode? I've tried to collect some statistics on GPU memory errors rates but have found them to be normally extremely rare. The only time I've reproducibly seen them is due to faulty hardware, where the errors become highly reproducible and the GPU needs replacement. The other theoretical cause of bit flips is supposed to be random errors due to cosmic radiation but I've never been able to observe that using memory testing software (though I did only run the experiments in AWS). Could it be that you have faulty or low grade GPUs? I assume these are all low-cost OEM parts, given your application? Or maybe there's something odd about your data center environment? Regarding the GPU database application, I think the answer is to just use the Tesla grade GPU with ECC memory enabled.
- optimiz3 10y ago
  Generally we prefer AMD cards as most (profitable) mining functions are memory bandwidth dominated. Usually it's a shader unit that gets unstable in the 70-80C range (note that most silicon is rated for higher ranges). AMD's hardware specifications are more open too which lets one build your own shader compilers and get direct access to the iron.
mtweak 10y ago
We've been working on an interposing library for guaranteeing GPU computation and it would be great to get your feedback. Any chance we sync up? My email is in my profile.