3 ms·

Python: How to bring your supercomputer to a crawl as 1,000 nodes all try to access hundreds of non-existent files on the same network share.

• est 13y ago
  Sounds like a poorly designed system rather than a Python problem.
  • jff 13y ago
    It was Python trying to figure out where its modules and such were. It would try a bunch of likely-seeming directories and get ENOENT until it finally hit the right place. Most HPC systems are kind of expecting that you'll ship a single compiled C/C++ binary to each node, each of which may read a file or two of input from the fileserver. With Python, we had a thousand times as many requests and it just killed the poor thing. This was on an IBM BlueGene. You can make your own call on the design quality of the system.
    • memracom 13y ago
      My solution to this problem was to build Python so that it finds all of its dependencies in one directory. This project https://github.com/wavetossed/pybuild https://github.com/wavetossed/pybuild has the build script that I used on Ubuntu Lucid LTS. In addition this build ends up being portable to any Linux distro because all of the dependencies are in one directory tree where you can easily create a tarball.
    • est 13y ago
      the shit would happen if you abuse LD_LIBRARY_PATH or PATH for C/C++ shared libraries. Many bad python code witll sys.path.insert() which would also cause this. By bad system I didn't mean your awesome IBM BlueGene, but the running system happened to use Python as language.