3 ms·
Another potentially surprising trap is the sheer number of filesystem syscalls per import. Python searches a ton of potential filesystem locations for each imp
by loeg 4y ago
Another potentially surprising trap is the sheer number of filesystem syscalls per import. Python searches a ton of potential filesystem locations for each import, and the default config may not search the most likely locations first. This had a significant impact at a previous employer.
- throwawaymaths 4y agoThis was (still is?) a well known problem for supercomputing facilities. Any given job has every single node access a shared r/w directory which python searches for imports. I believe python opens (opened?) Imports r/w and this created a transactional 2PC action for each node, so you get thousands of nodes blocking on the filesystem,a test run that could be a few seconds could take hours to actually spin up