5 ms·
As far as I understood it: Forget GPUs, this thing is plenty fast on CPUs. In general, GPUs are bad at branching. The fastest way to implement it on GPUs is pr
by nolist_policy 3y ago
As far as I understood it: Forget GPUs, this thing is plenty fast on CPUs.
In general, GPUs are bad at branching. The fastest way to implement it on GPUs is probably to let it calculate both sides of the branch and then only use the result of the one that was taken. Which won't be faster than a normal NN.
- falcor84 3y agoI wonder then, if each inference only uses a small part of the net, could you possibly perform multiple inferences in the same forward pass?
- lawlessone 3y ago> Forget GPUs, this thing is plenty fast on CPUs. Does this mean everyone could be running the 100+b models from ram? This opens up a lot , some models could be run very fast on small machines with this. Bundling a small model inside a game to act as part of the mind for ingame NPC's (obviously with some tuning) becomes practical with this.
- hobofan 3y agoThe bottleneck for "easy integration" into games and applications right now is as much the RAM usage as is the slowness. This would probably bring the speed to an acceptable level but you would still have to hold the whole model in RAM. That would make it a lot more feasible to run models in the cloud (triple digit RAM is a lot more abundant than VRAM), but wouldn't do that much for consumer hardware.
- nolist_policy 3y agoI wonder if the model takes similar branches while in the same context? Then you can fault in parts of the model from disk as needed.
- entropicdrifter 3y agoInteresting idea. Like texture streaming, you'd just stream in the parts of the model from disk to fill up all available RAM. If the NPC needed to think about something not cached in RAM, you'd throw up a "hmm, let me think about this" while stuff loads from disk.