4 ms·

Do I understand correctly that the difficulty of making this useful is writing code to run this idea on GPUs?

by iNic 3y ago

PaulHoule 3y ago
They are getting a 78x speedup w/o hardware support which is pretty good: they think they can speed it up another 4x if they had the right hardware support. So it looks useful now with possibility to get better. So long as I've been involved with neural networks for text analysis it's seemed to me that we really should be using sparse activations because any particular document only involves a limited set of concepts. For instance a search engine for patents might be looking at a patent for adhesive tape which activates a certain set of concepts but is not going to activate concepts involved with bicycle derailleurs or public key cryptography: a sparse representation reflects this and dense representations don't.
nolist_policy 3y ago
As far as I understood it: Forget GPUs, this thing is plenty fast on CPUs. In general, GPUs are bad at branching. The fastest way to implement it on GPUs is probably to let it calculate both sides of the branch and then only use the result of the one that was taken. Which won't be faster than a normal NN.
- falcor84 3y ago
  I wonder then, if each inference only uses a small part of the net, could you possibly perform multiple inferences in the same forward pass?
- lawlessone 3y ago
  > Forget GPUs, this thing is plenty fast on CPUs. Does this mean everyone could be running the 100+b models from ram? This opens up a lot , some models could be run very fast on small machines with this. Bundling a small model inside a game to act as part of the mind for ingame NPC's (obviously with some tuning) becomes practical with this.
  - hobofan 3y ago
    The bottleneck for "easy integration" into games and applications right now is as much the RAM usage as is the slowness. This would probably bring the speed to an acceptable level but you would still have to hold the whole model in RAM. That would make it a lot more feasible to run models in the cloud (triple digit RAM is a lot more abundant than VRAM), but wouldn't do that much for consumer hardware.
    - nolist_policy 3y ago
      I wonder if the model takes similar branches while in the same context? Then you can fault in parts of the model from disk as needed.
      - entropicdrifter 3y ago
        Interesting idea. Like texture streaming, you'd just stream in the parts of the model from disk to fill up all available RAM. If the NPC needed to think about something not cached in RAM, you'd throw up a "hmm, let me think about this" while stuff loads from disk.
kristianp 3y ago
> a PyTorch implementation delivering 40x speedup over the equivalent batched feedforward inference Does this not indicate a 40x speedup on the GPU? Edit: looking at the paper, their "Naive CUDA" implementation also shows a 117x speedup in Table 2.

Do I understand correctly that the difficulty of making this useful is writing code to run this idea on GPUs?

by iNic 3y ago

Do I understand correctly that the difficulty of making this useful is writing code to run this idea on GPUs?

They are getting a 78x speedup w/o hardware support which is pretty good: they think they can speed it up another 4x if they had the right hardware support. So it looks useful now with possibility to get better. So long as I've been involved with neural networks for text analysis it's seemed to me that we really should be using sparse activations because any particular document only involves a limited set of concepts. For instance a search engine for patents might be looking at a patent for adhesive tape which activates a certain set of concepts but is not going to activate concepts involved with bicycle derailleurs or public key cryptography: a sparse representation reflects this and dense representations don't.

nolist_policy 3y ago

As far as I understood it: Forget GPUs, this thing is plenty fast on CPUs. In general, GPUs are bad at branching. The fastest way to implement it on GPUs is probably to let it calculate both sides of the branch and then only use the result of the one that was taken. Which won't be faster than a normal NN.

falcor84 3y ago
I wonder then, if each inference only uses a small part of the net, could you possibly perform multiple inferences in the same forward pass?
lawlessone 3y ago
> Forget GPUs, this thing is plenty fast on CPUs. Does this mean everyone could be running the 100+b models from ram? This opens up a lot , some models could be run very fast on small machines with this. Bundling a small model inside a game to act as part of the mind for ingame NPC's (obviously with some tuning) becomes practical with this.
- hobofan 3y ago
  The bottleneck for "easy integration" into games and applications right now is as much the RAM usage as is the slowness. This would probably bring the speed to an acceptable level but you would still have to hold the whole model in RAM. That would make it a lot more feasible to run models in the cloud (triple digit RAM is a lot more abundant than VRAM), but wouldn't do that much for consumer hardware.
  - nolist_policy 3y ago
    I wonder if the model takes similar branches while in the same context? Then you can fault in parts of the model from disk as needed.
    - entropicdrifter 3y ago
      Interesting idea. Like texture streaming, you'd just stream in the parts of the model from disk to fill up all available RAM. If the NPC needed to think about something not cached in RAM, you'd throw up a "hmm, let me think about this" while stuff loads from disk.

kristianp 3y ago

> a PyTorch implementation delivering 40x speedup over the equivalent batched feedforward inference Does this not indicate a 40x speedup on the GPU? Edit: looking at the paper, their "Naive CUDA" implementation also shows a 117x speedup in Table 2.