4 ms·
Expensive pre-processing is hardly a problem: Generate representations and features once and store to drive. Use a feature/vector database if you feel fancy. F
by gillesjacobs 4y ago
Expensive pre-processing is hardly a problem: Generate representations and features once and store to drive. Use a feature/vector database if you feel fancy.
For deep learning, out-of-core instance loading by means of batching is the standard approach. Sure you sometimes have to tune batch size and learning rate for optimal learning, but it's a solved issue.
- tysam_and 4y agoI'm meaning to reply to the parent comment after I get some sleep, but I do believe that is what they are implying! The hard part, I believe, is _how_ we effectively do the memory management! :D
- beecafe 4y agodoesn't work due to requirement of online data augmentation like random crops and image distortions (for image model, loading is a non-issue for text). And a feature database is not going to help you if you just need to iterate over all the features.
- jebarker 4y agoI disagree that's it's solved in practice for all users. I agree it's solved for many well established workloads. Researchers are the people most in need of rapid experimental iteration vs one off training. But during research the data requirements may not be pinned down well enough to pre-process everything ahead of time or repeatedly doing so is it's own bottleneck. Out of core loading may still be a bottleneck if IO is slow, samples are big or necessary pre-processing can't be pushed to GPU.