3 ms·

Binary dump / prewarmed up Python interpreter would be amazing even today. Could enable a lot use cases with Python.

by liuliu 4y ago

nyanpasu64 4y ago
youtube-dl/yt-dlp (Python) is a full second slower than youtubedr (Go) or youtube.com (JS) to download YouTube videos, and I think a lot of that is due to interpreter warmup (though I don't know exactly why). Sadly mpv only supports youtube-dl/yt-dlp for watching videos, and youtubedr doesn't have the vast plugin ecosystem of yt-dlp for non-YouTube websites.
- KolenCh 4y ago
  The above mentioned Python’s import delay. This is a notorious behavior in high performance situation (including your cli startup time example). Essentially the import process is IO intensive. This comes up in massive parallel applications as at launch each process is importing the same stuffs. It is briefly mentioned for example in https://docs.nersc.gov/development/languages/python/ https://docs.nersc.gov/development/languages/python/ If you really care about the startup time, you could pre-compile it. Nuitka and Cython could do this with different limitations.
  - OOPMan 4y ago
    I've run into the import issue as well, it's especially painful if something imported is doing work using a slow library (e.g. a config module loading a HOCON config using pyhocon a.k.a slowasmolasseshocon)
    - KolenCh 4y ago
      That will be another slow down happening at import stage. But even if ideally everything you import only does import with not much work, the import overhead mentioned above still exists. But frankly I’m not sure how significant it is on a typical personal computer with SSD, especially when the OS may cache recently used files already. The problem you mention can be significant because some design patterns in Python encourages this. Eg function arg with a default that’s expensive. Unfortunately this whole speeds up repeatedly calling the func, the cost is paid at first import.
  - miohtama 4y ago
    Unless your 1) execution environment (CPU, dlls) or 2) Python source code files change, you can use “unexec” strategies like Emacs https://news.ycombinator.com/item?id=21394916 https://news.ycombinator.com/item?id=21394916

Binary dump / prewarmed up Python interpreter would be amazing even today. Could enable a lot use cases with Python.

by liuliu 4y ago

Binary dump / prewarmed up Python interpreter would be amazing even today. Could enable a lot use cases with Python.

youtube-dl/yt-dlp (Python) is a full second slower than youtubedr (Go) or youtube.com (JS) to download YouTube videos, and I think a lot of that is due to interpreter warmup (though I don't know exactly why). Sadly mpv only supports youtube-dl/yt-dlp for watching videos, and youtubedr doesn't have the vast plugin ecosystem of yt-dlp for non-YouTube websites.

KolenCh 4y ago
The above mentioned Python’s import delay. This is a notorious behavior in high performance situation (including your cli startup time example). Essentially the import process is IO intensive. This comes up in massive parallel applications as at launch each process is importing the same stuffs. It is briefly mentioned for example in https://docs.nersc.gov/development/languages/python/ https://docs.nersc.gov/development/languages/python/ If you really care about the startup time, you could pre-compile it. Nuitka and Cython could do this with different limitations.
- OOPMan 4y ago
  I've run into the import issue as well, it's especially painful if something imported is doing work using a slow library (e.g. a config module loading a HOCON config using pyhocon a.k.a slowasmolasseshocon)
  - KolenCh 4y ago
    That will be another slow down happening at import stage. But even if ideally everything you import only does import with not much work, the import overhead mentioned above still exists. But frankly I’m not sure how significant it is on a typical personal computer with SSD, especially when the OS may cache recently used files already. The problem you mention can be significant because some design patterns in Python encourages this. Eg function arg with a default that’s expensive. Unfortunately this whole speeds up repeatedly calling the func, the cost is paid at first import.
- miohtama 4y ago
  Unless your 1) execution environment (CPU, dlls) or 2) Python source code files change, you can use “unexec” strategies like Emacs https://news.ycombinator.com/item?id=21394916 https://news.ycombinator.com/item?id=21394916