4 ms·
A lot of machine learning (e.g. NCE) requires fast PRNG also, often running on GPUs that don't have crypto primitives like many CPUs do.
by gok 7y ago
A lot of machine learning (e.g. NCE) requires fast PRNG also, often running on GPUs that don't have crypto primitives like many CPUs do.
- dragontamer 7y agoGPUs have bit-reverse. Counter (0, 1, 2, 3, 4...) -> Multiply Const1 -> Bit-reverse -> XOR Const2 -> Multiply Const3 has been my PRNG for GPUs. Const1 and Const3 need to be odd (bottom bit == 1). Const2 can be fully random. All three values can come from /dev/urandom with pretty good results with some preliminary testing. I haven't figured out a good methodology from selecting Const1 or Const3 aside from the bottom-bit == 1 and random (If I had more time, I'd probably work on it...) --------- The "counter" is 64-bits and allocated per-thread. Assume 2^14 bits for 16384 GPU-threads, each GPU-thread then has 50-bits of values. 64-bit integer multiply is slow however. 32-bit multiply would be faster for most GPUs. But also in my experience, GPUs are almost always memory-constrained, not compute-constrained. So you probably can do fine with 64-bit integer multiply. If you really need to stick with 32-bit integers, then make Const1 and Const3 per-thread specific. That gives a unique sequence of 32-bit numbers per thread, but a 32-bit cycle is inevitable every 4-billion numbers with only 32-bits of state.