If I'm understanding right, this CPU vs. GPU bit seems apples-to-oranges for how GPU/parallel alg research would advocate optimizing this: - if you have one t

If I'm understanding right, this CPU vs. GPU bit seems apples-to-oranges for how GPU/parallel alg research would advocate optimizing this: - if you have one tiny network, no one cares, use pure python - if you have a bunch of tiny networks, yes, doing a ton of tiny GPU kernel calls will kill you with overhead <-- strawman because ... - ... you can likely combine them into bulk sparse kernel calls and spread them out. If more of an iterative simulation, same thing, just over time. This goes back to the 80's with segmented scans, cray, & connection machines. Put all the data together, figure out regular bulk operators, even if individual parts are unbalanced or otherwise irregular, and voila. Since then, a lot of basic tricks for regularizing (sparsity, speculation, ...), and has gone into overdrive w/ GPU era. If this is as worthy the investments these specialists are doing, as it seems to be many years of R&D by smart groups, I'm surprised they're comparing to an unoptimized GPU alg. Presumably, at some point, they did a more reasonable one? It used to be that honest CPU<>GPU implementations were generally within 2X of one another here once you considered normalizers like performance-per-watt (= per $), though new CPU HW is closer to GPU HW and new GPU HW is fancier, so one of the funnier results is, algorithmically, you have to use the same fundamental GPU-era tricks either way.