After a quick read of the paper, no. You could adopt this to the GPU (which would require the hashes work on groups of neurons instead of individuals) and might

After a quick read of the paper, no. You could adopt this to the GPU (which would require the hashes work on groups of neurons instead of individuals) and might get a similar speedup. Locality sensitive hashing in fact seems like a primitive attention mechanism, with proper attention implementation you could get maybe even better results.