3 ms·

Highly recommend "export PYTHONDONTWRITEBYTECODE=1" in your bashrc and just forget about it. Pyc files are still an important optimization on modern machines in

Highly recommend "export PYTHONDONTWRITEBYTECODE=1" in your bashrc and just forget about it. Pyc files are still an important optimization on modern machines in some circumstances (especially with huge oft-restarted apps), but the autogeneration behaviour has always been a pain in the ass The bulk of your pycs are generated during package install. What tends to remain in the usual case is a handful of files representing app code or similar.

the_jeremy 6y ago
Why do you highly recommend this? You can set git (or other VCS) settings to avoid committing .pyc files. They're not large files; I don't see the downside.
- shockinglytrue 6y ago
  Difficulty with git basics is the least of your worries. For example pycs are fundamentally racy, it is quite possible to have a .py newer than a .pyc depending on how unlucky you were with an in-progress deployment, or some tool that never updated the .py timestamp. Python continues to execute the pyc even though the code changed, since the minimal benefit of the pyc would be rendered significantly inert should Python use any kind of strong check (instead of just comparing second-granularity stat() output) to ensure the cached bytecode matches the source. In this way, without your permission, Python silently plays undesirable code execution roulette with your computer every time it starts, for as long as you have the feature enabled. I have lost count of the number of times I've seen someone lose an hour due to it. I can also count many instances of QA environments becoming inexplicably bricked by it. The correct fix for this requires opening the .py and hashing its content, at least doubling the amount of IO required to start a program. They were a great feature when parsing small files was noticeably slow, but this hasn't been true for almost 20 years. It's therefore worth turning the question around: why do you think pyc files are useful?
  - eesmith 6y ago
    Python 3.7 added support for hash-based cache files as an alternative to time-based. https://docs.python.org/3/reference/import.html#pyc-invalidation https://docs.python.org/3/reference/import.html#pyc-invalida... Verifying a hash is a bit slower than checking the timestamp but far faster than parsing and byte compiling the source file, so I don't think this option is "significantly inert".
    - shockinglytrue 6y ago
      Managed to miss this, thanks. I'd be interested to hunt out the BPO ticket at some stage to see if they benchmarked on NFS or spinning rust
      - eesmith 6y ago
        Huh. I hadn't bothered to read the PEP, which is https://www.python.org/dev/peps/pep-0552/ https://www.python.org/dev/peps/pep-0552/ on "Deterministic pycs." > The current Python pyc format is the marshaled code object of the module prefixed by a magic number [7], the source timestamp, and the source file size. The presence of a source timestamp means that a pyc is not a deterministic function of the input file’s contents—it also depends on volatile metadata, the mtime of the source. Thus, pycs are a barrier to proper reproducibility. That is, they were made for a quite different use case than you or I were talking about. I looked at the PEP to see if it gave timing numbers. No luck - would be a good blog post if I were still blogging. It does say: > The hash-based pyc format can impose the cost of reading and hashing every source file, which is more expensive than simply checking timestamps. Thus, for now, we expect it to be used mainly by distributors and power use cases.
  - stestagg 6y ago
    In the last 12 years of writing python, I have only hit had issues with .pyc files a handful of times, and always with python < 2.7. Anaecdotally this experience is shared with everyone I have worked with. If you’re seeing this regularly, it suggests there may be something unique or uncommon in your set-up. You may wish to isolate and change whatever that is.
- BiteCode_dev 6y ago
  When doing dev, there a be other problems with pyc files. On of them is deleting the original py file, but still being able to import the module because Python will import any pyc file it if exist. So if performances allow you to do so, disabling them when you dev is not a bad idea, as long as you keep them enabled when you run CI. IMO, when you dev, you should set PYTHONHASHSEED (random is predictible), PYTHONDEVMODE (verbose warnings + tooling + sys.flags.dev_mode=True) and PYTHONDONTWRITEBYTECODE (remove the need for cleaning them). In CI and prod, you should make sure those are NOT set, and use PYTHONOPTIMIZE=2 (remove assert, __debug__ lines and docstring) if you trust your dependancies to be well written (which I would check by running tests in a pre-push hook).

the_jeremy 6y ago

Why do you highly recommend this? You can set git (or other VCS) settings to avoid committing .pyc files. They're not large files; I don't see the downside.

shockinglytrue 6y ago
Difficulty with git basics is the least of your worries. For example pycs are fundamentally racy, it is quite possible to have a .py newer than a .pyc depending on how unlucky you were with an in-progress deployment, or some tool that never updated the .py timestamp. Python continues to execute the pyc even though the code changed, since the minimal benefit of the pyc would be rendered significantly inert should Python use any kind of strong check (instead of just comparing second-granularity stat() output) to ensure the cached bytecode matches the source. In this way, without your permission, Python silently plays undesirable code execution roulette with your computer every time it starts, for as long as you have the feature enabled. I have lost count of the number of times I've seen someone lose an hour due to it. I can also count many instances of QA environments becoming inexplicably bricked by it. The correct fix for this requires opening the .py and hashing its content, at least doubling the amount of IO required to start a program. They were a great feature when parsing small files was noticeably slow, but this hasn't been true for almost 20 years. It's therefore worth turning the question around: why do you think pyc files are useful?
- eesmith 6y ago
  Python 3.7 added support for hash-based cache files as an alternative to time-based. https://docs.python.org/3/reference/import.html#pyc-invalidation https://docs.python.org/3/reference/import.html#pyc-invalida... Verifying a hash is a bit slower than checking the timestamp but far faster than parsing and byte compiling the source file, so I don't think this option is "significantly inert".
  - shockinglytrue 6y ago
    Managed to miss this, thanks. I'd be interested to hunt out the BPO ticket at some stage to see if they benchmarked on NFS or spinning rust
    - eesmith 6y ago
      Huh. I hadn't bothered to read the PEP, which is https://www.python.org/dev/peps/pep-0552/ https://www.python.org/dev/peps/pep-0552/ on "Deterministic pycs." > The current Python pyc format is the marshaled code object of the module prefixed by a magic number [7], the source timestamp, and the source file size. The presence of a source timestamp means that a pyc is not a deterministic function of the input file’s contents—it also depends on volatile metadata, the mtime of the source. Thus, pycs are a barrier to proper reproducibility. That is, they were made for a quite different use case than you or I were talking about. I looked at the PEP to see if it gave timing numbers. No luck - would be a good blog post if I were still blogging. It does say: > The hash-based pyc format can impose the cost of reading and hashing every source file, which is more expensive than simply checking timestamps. Thus, for now, we expect it to be used mainly by distributors and power use cases.
- stestagg 6y ago
  In the last 12 years of writing python, I have only hit had issues with .pyc files a handful of times, and always with python < 2.7. Anaecdotally this experience is shared with everyone I have worked with. If you’re seeing this regularly, it suggests there may be something unique or uncommon in your set-up. You may wish to isolate and change whatever that is.
BiteCode_dev 6y ago
When doing dev, there a be other problems with pyc files. On of them is deleting the original py file, but still being able to import the module because Python will import any pyc file it if exist. So if performances allow you to do so, disabling them when you dev is not a bad idea, as long as you keep them enabled when you run CI. IMO, when you dev, you should set PYTHONHASHSEED (random is predictible), PYTHONDEVMODE (verbose warnings + tooling + sys.flags.dev_mode=True) and PYTHONDONTWRITEBYTECODE (remove the need for cleaning them). In CI and prod, you should make sure those are NOT set, and use PYTHONOPTIMIZE=2 (remove assert, __debug__ lines and docstring) if you trust your dependancies to be well written (which I would check by running tests in a pre-push hook).