3 ms·
This is certainly something that needs to be addressed far more often. Too many times have I seen some data scientist trying to parse and write 6 Petabytes of
by chaxor 3y ago
This is certainly something that needs to be addressed far more often.
Too many times have I seen some data scientist trying to parse and write 6 Petabytes of data with multiple cores, while the disk is thrashing about.
Spinning disks are still the backbone of most data science operations because they deal with >>4TB datasets, which can't be stored in SSD drives without breaking some serious bank.
So yes, understanding how to properly use producer/consumer multiprocessing queues correctly should be taught to everyone who does computing as the standard template.
Disk thrash is a threat.