You're right, I was assuming that once the unguided training/optimization methods become cheap enough to perform continuously (and maybe in parallel to inferenc

You're right, I was assuming that once the unguided training/optimization methods become cheap enough to perform continuously (and maybe in parallel to inference) it would be indistinguishable from online learning. For true online learning we're still lacking a good base architecture (although Meta-Learning and NAS are exploring that angle).