It sounds like they do it naively, which is fair for a POC Even for non-batched, you can still optimize this a bunch for fairly big models without much more wo

It sounds like they do it naively, which is fair for a POC Even for non-batched, you can still optimize this a bunch for fairly big models without much more work. Basically, overlap io & compute: while one layer is computing, already be loading in the next layer(s) in parallel. PCI speeds are in the 1GB/s-32GB/s each way for consumer cards nowadays. Without a lot of work, they might be able to fully hide the IO latency.. very model + hw dependent.