This is a bad and needlessly complicated solution because the authors don't know anything about ML. It's both incredibly slow and it will perform poorly (in th

This is a bad and needlessly complicated solution because the authors don't know anything about ML. It's both incredibly slow and it will perform poorly (in the sense of the final accuracy and the number of training steps) because you can't just software engineer your way through ML. The important insight is that you do not need all patches from every image! Actually, showing patches like this in sequence is extremely detrimental to training. The network sees far too much data that is too similar in too many big chunks. You want random patches from random images, the more mixed up the patches the better. So knowing this, when you look at their latency equation there's a different and obvious solution. Split the loading process in two steps that run in parallel. First step constantly downloads new images from the web and swaps old images out. Second step picks an image at random from the ones that are available and generates a random patch from it. The first step is network bound. The second step is CPU bound. The second step always has images available, it never waits for the first, just picks another random image and random patch. You get great resource utilization out of this. That's it. No other changes needed. Just use an off the shelf fast image loader. No need for a cluster. This is a huge waste of engineering time and ongoing computing resources for what is a simple ML problem, had anyone with any knowledge been around. Hey, tweag! If you want to do ML, reach out. :) You can do far better than this!