3 ms·
This was (still is?) a well known problem for supercomputing facilities. Any given job has every single node access a shared r/w directory which python searche
by throwawaymaths 4y ago
This was (still is?) a well known problem for supercomputing facilities. Any given job has every single node access a shared r/w directory which python searches for imports. I believe python opens (opened?) Imports r/w and this created a transactional 2PC action for each node, so you get thousands of nodes blocking on the filesystem,a test run that could be a few seconds could take hours to actually spin up