3 ms·
Sequential model offloading isn’t too bad. It adds about a second or less to inference, assuming it still fits in main memory.
by Filligree 3y ago
Sequential model offloading isn’t too bad. It adds about a second or less to inference, assuming it still fits in main memory.
- sorenjan 3y agoSometimes I forget how fast modern computers are. PCIe v4 x16 has a transfer speed of 31.5 GB/s, so theoretically it should take less than 100 ms to transfer stage B and A. Maybe it's not so bad after all, it will be interesting to see what happens.