From my (albeit naive) reading, it doesn't appear that that they've reduced the amount of memory bandwidth required, simply the size of the working set required

From my (albeit naive) reading, it doesn't appear that that they've reduced the amount of memory bandwidth required, simply the size of the working set required. Since inference is generally memory bandwidth bound once you reach the level of 'does this model even fit in the given system', I'd imagine that this technique wouldn't help much for greater throughpit via larger batch sizes. Just one instance is probably already saturating the memory controller. Maybe it'd help on the training side though?