3 ms·

How do they guarantee the correctness of results? A major problem with GPUs is you see single bit errors with surprisingly high frequency. For graphics this u

How do they guarantee the correctness of results? A major problem with GPUs is you see single bit errors with surprisingly high frequency. For graphics this usually doesn't matter as a minor color or vertex deviation isn't noticeable, but for compute it can be devastating. We do cryptocurrency mining on an industrial scale and constantly see single bit errors from hardware that is brand-new without modifications.

n00b101 10y ago
We do cryptocurrency mining on an industrial scale and constantly see single bit errors from hardware that is brand-new without modifications. That's very surprising and interesting. How do you detect these single bit errors?
- AlsoN00b 10y ago
  Not OP but a lot of this seems pretty believable and easy to detect to me? Dont the GPU specs allow for a certain lossiness in the math? Or like, at least they don't conform to IEEE 794 float specs w/ regard to order of operations, precision, degradation, etc. So like, do a shitload of math ops in a glsl shader with a deterministic outcome, render the result to a texture, take the texture back and make sure the RGBA values match bit for bit with the numbers you expected? Or to detect single bit errors in the gpu's local memory or caching just attach textures, read data into them, read it back, render back, etc etc.
- optimiz3 10y ago
  Detection of false positives: Run candidate solutions through the CPU Detection of false negatives: Compare solution distribution and frequency to expected models; switch to debug kernels if outside tolerance. However, this works because the mining problem space is stateless and follows strict mathematically predictable models. A DB is stateful and the answers generally can't be verified without consulting a secondary copy, which is why I'm super curious how they would engineer correctness and reliability in a cost-effective way using GPUs.
  - n00b101 10y ago
    That's very interesting. Have you collected statistical data on these bit errors? Is it always a single bit error? I'm assuming you are using GeForce cards and not Tesla cards which have an ECC memory protection mode? I've tried to collect some statistics on GPU memory errors rates but have found them to be normally extremely rare. The only time I've reproducibly seen them is due to faulty hardware, where the errors become highly reproducible and the GPU needs replacement. The other theoretical cause of bit flips is supposed to be random errors due to cosmic radiation but I've never been able to observe that using memory testing software (though I did only run the experiments in AWS). Could it be that you have faulty or low grade GPUs? I assume these are all low-cost OEM parts, given your application? Or maybe there's something odd about your data center environment? Regarding the GPU database application, I think the answer is to just use the Tesla grade GPU with ECC memory enabled.
    - optimiz3 10y ago
      Generally we prefer AMD cards as most (profitable) mining functions are memory bandwidth dominated. Usually it's a shader unit that gets unstable in the 70-80C range (note that most silicon is rated for higher ranges). AMD's hardware specifications are more open too which lets one build your own shader compilers and get direct access to the iron.
  - mtweak 10y ago
    We've been working on an interposing library for guaranteeing GPU computation and it would be great to get your feedback. Any chance we sync up? My email is in my profile.
loeg 10y ago
It's surprising you can still do cryptocurrency mining with GPUs. Not bitcoin, then?
- mappu 10y ago
  The GP didn't say that their mining is done on GPUs.
  - loeg 10y ago
    It is implied that mining is how they are familiar with bit error problems in GPUs.
- optimiz3 10y ago
  Ethereum, Monero, and even Ethereum-Classic (difficulty/arbitrage/exchange rate permitting) are all good GPU candidates - SHA-256d (Bitcoin) and Scrypt (Litecoin) have ASICs, X11 (Dash - formerly Darkcoin) has FPGAs.
glangdale 10y ago
I am reminded of a Jack Vance quote, where one of the characters is "capable of performing complex calculations in his head and furnishing the results in an instant, whether they are right or wrong". I cannot find any reference to handling of soft errors in their material. One rather banal approach is to do everything twice and check the results; effective, but I'm not sure that they do this. They may be simply putting their heads in the sand.
johntb86 10y ago
Some Tesla GPUs support ECC memory, so that should help.
deafcalculus 10y ago
Do you use compute shaders / CUDA? I'm really surprised at the error rate. I've used fragment shaders in OpenGL ES 2.0 for compute on mobile platforms, and the "errors" turned out to be dithering.

n00b101 10y ago

We do cryptocurrency mining on an industrial scale and constantly see single bit errors from hardware that is brand-new without modifications. That's very surprising and interesting. How do you detect these single bit errors?

AlsoN00b 10y ago
Not OP but a lot of this seems pretty believable and easy to detect to me? Dont the GPU specs allow for a certain lossiness in the math? Or like, at least they don't conform to IEEE 794 float specs w/ regard to order of operations, precision, degradation, etc. So like, do a shitload of math ops in a glsl shader with a deterministic outcome, render the result to a texture, take the texture back and make sure the RGBA values match bit for bit with the numbers you expected? Or to detect single bit errors in the gpu's local memory or caching just attach textures, read data into them, read it back, render back, etc etc.
optimiz3 10y ago
Detection of false positives: Run candidate solutions through the CPU Detection of false negatives: Compare solution distribution and frequency to expected models; switch to debug kernels if outside tolerance. However, this works because the mining problem space is stateless and follows strict mathematically predictable models. A DB is stateful and the answers generally can't be verified without consulting a secondary copy, which is why I'm super curious how they would engineer correctness and reliability in a cost-effective way using GPUs.
- n00b101 10y ago
  That's very interesting. Have you collected statistical data on these bit errors? Is it always a single bit error? I'm assuming you are using GeForce cards and not Tesla cards which have an ECC memory protection mode? I've tried to collect some statistics on GPU memory errors rates but have found them to be normally extremely rare. The only time I've reproducibly seen them is due to faulty hardware, where the errors become highly reproducible and the GPU needs replacement. The other theoretical cause of bit flips is supposed to be random errors due to cosmic radiation but I've never been able to observe that using memory testing software (though I did only run the experiments in AWS). Could it be that you have faulty or low grade GPUs? I assume these are all low-cost OEM parts, given your application? Or maybe there's something odd about your data center environment? Regarding the GPU database application, I think the answer is to just use the Tesla grade GPU with ECC memory enabled.
  - optimiz3 10y ago
    Generally we prefer AMD cards as most (profitable) mining functions are memory bandwidth dominated. Usually it's a shader unit that gets unstable in the 70-80C range (note that most silicon is rated for higher ranges). AMD's hardware specifications are more open too which lets one build your own shader compilers and get direct access to the iron.
- mtweak 10y ago
  We've been working on an interposing library for guaranteeing GPU computation and it would be great to get your feedback. Any chance we sync up? My email is in my profile.

loeg 10y ago

It's surprising you can still do cryptocurrency mining with GPUs. Not bitcoin, then?

mappu 10y ago
The GP didn't say that their mining is done on GPUs.
- loeg 10y ago
  It is implied that mining is how they are familiar with bit error problems in GPUs.
optimiz3 10y ago
Ethereum, Monero, and even Ethereum-Classic (difficulty/arbitrage/exchange rate permitting) are all good GPU candidates - SHA-256d (Bitcoin) and Scrypt (Litecoin) have ASICs, X11 (Dash - formerly Darkcoin) has FPGAs.

glangdale 10y ago

I am reminded of a Jack Vance quote, where one of the characters is "capable of performing complex calculations in his head and furnishing the results in an instant, whether they are right or wrong". I cannot find any reference to handling of soft errors in their material. One rather banal approach is to do everything twice and check the results; effective, but I'm not sure that they do this. They may be simply putting their heads in the sand.

johntb86 10y ago

Some Tesla GPUs support ECC memory, so that should help.

deafcalculus 10y ago

Do you use compute shaders / CUDA? I'm really surprised at the error rate. I've used fragment shaders in OpenGL ES 2.0 for compute on mobile platforms, and the "errors" turned out to be dithering.