My understanding is that this works better because it is faster to hash their data (and compare with prehashed weights) rather than multiply the data and calcul

My understanding is that this works better because it is faster to hash their data (and compare with prehashed weights) rather than multiply the data and calculate activation functions. I imagine this could work for more layers easily,and using other than fully connected layers could be done by partitioning data (for each set of inputs). The next obvious question is, can their hashing function run even faster on a GPU? They say: >In particular, SLIDE uses recently proposed DTWA hashing which works nicely on sparse data. If we represent the data vector as a list of indices and values. DTWA operation computes a random hash map of every non-zero index in the list. Then the map is partitioned into a fixed number of bins. Finally, the index with the maximum coordinate value in each bin is the hash value. What is the "coordinate value"? And how does the above map to find instances where data and weights values would result in a value above the activation threshold? They themselves mention it "works well on sparse data". If they detect irrelevant data by simple value comparison(wrapped in that hashing function) couldn't GPU algorithm be sped up by skipping those multiplications too? How about instead of running matrix multiplication on entire data and all weights we "preprocess" data and weights finding specific indices where they are both above a certain value and multiply just those. It would be interesting to try that. How expensive would running such comparison, then data copy (or create a map of indices to process), then process only those chosen be in comparison with multiplying everything every time? I'm guessing that it really depends on what percentage of data is actually relevant. Another idea to make GPU AI faster is improving scheduling of those matrix multiplication. Multiplication by zero, another small number or a power of 2 should be able to execute faster than multiplication of high values (assuming it is implemented in hardware by addition and bit shifting if multiplying by powers of 2). I don't know if current GPU algorithms make any use of that, or if they simply divide the data by number of cores and run all chunks in parallel for however long the longest chunk takes to complete? I think GPUs are pretty far from being dead in AI as a result of this.