3 ms·
In Keras you can just use a Sequence-based generator, set a large number of workers, or even put in multi-GPU callback, and you can parallelize pre-processing a
by bitL 9y ago
In Keras you can just use a Sequence-based generator, set a large number of workers, or even put in multi-GPU callback, and you can parallelize pre-processing as much as you like.
- netheril96 9y agoI doubt that Keras can break the limit of the CPython interpreter. Its parallelization is probably also done with multiple processes, in no way as efficient as multithreading.
- bitL 9y agoI might take a look at Keras source just to know :) But still, if you do image augmentation on the fly, you are probably fine even with processes if your batch size is sufficiently large.