5 ms·

This would be solved if python used an (OS-specific) cache directory for its .pyc files. I have always disliked .pyc files... here's a concrete reason! Questio

This would be solved if python used an (OS-specific) cache directory for its .pyc files. I have always disliked .pyc files... here's a concrete reason! Question: what does python do if it doesn't have write permission in the current working directory? Not write the cache?

eesmith 6y ago
It already (since 3.2, as the article points out) uses a cache directory. Why would an OS-specific one help? You can also set PYTHONPYCACHEPREFIX if you want to use 'a mirror directory tree at this path, instead of in __pycache__ directories within the source tree' - https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPYCACHEPREFIX https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPY... The PEP mentions your case at https://www.python.org/dev/peps/pep-3147/#case-5-read-only-file-systems https://www.python.org/dev/peps/pep-3147/#case-5-read-only-f... . But honestly, I don't really follow the answer. I think it means "ignore creation and write failures."
- deathanatos 6y ago
  > Why would an OS-specific one help? E.g., if you want to back up your home dir, and omit caches, since they can be regenerated. It's a lot easier if programs write their cache data to ~/.cache / $XDG_CACHE_HOME than if they intermix it / scatter it about.
  - deleted 6y ago
    [deleted]
  - eesmith 6y ago
    But how does it help to have one directory for all Linux-based OSes (or one directory for RHEL, one for Ubuntu, one for Debian, etc.), and one for FreeBSD and one for OpenIndiana? At least, that's what I interpret "OS-specific" to mean.
    - a_t48 6y ago
      Windows exists, too. Probably would be in %APPDATA% or %TMP%
    - Chickenosaurus 6y ago
      I suspect his thought was to use something like "%UserProfile%/pycache/" on windows and "$HOME/.pycache/" on Linux.
    - deathanatos 6y ago
      More along the line of the original comment you're replying to, if the cache was in, say, ~/.cache, then it won't get swept up in the repository's commits, since the cache data is no longer inside the repository's working directory. Then, it never gets uploaded to GitHub, and this security issue never happens. I have seen a surprising number of people — some who are engineers by profession too, and ought to know better — just git add everything, and then commit it all without looking. One should review the diff one has staged to see if it is correct, but alas…
      - eesmith 6y ago
        That's possible with 3.8's PYTHONPYCACHEPREFIX, yes? Perhaps it's worthwhile for someone to blog about this more/promote this as a best practice? Though what's missing is the hook to connected it as appropriate for the given platform. I see now that "OS-specific" was meant to be interpreted as "the OS-defined mechanism to find a cache directory", not "a cache directory which differs for each operating system". I would not have been confused by the term "platform dependent", which is what Python's tmpdir documentation uses, as in: "The default directory is chosen from a platform-dependent list" at https://docs.python.org/3/library/tempfile.html?highlight=tmpdir#tempfile.mkstemp https://docs.python.org/3/library/tempfile.html?highlight=tm... .
- globular-toast 6y ago
  That env var is new in 3.8. I've been looking forward to it to stop docker writing root owned cache files back to a bind mounted file system. I might set it to $HOME/.cache/python globally as well.
pdonis 6y ago
> what does python do if it doesn't have write permission in the current working directory? Not write the cache? In the case where the interpreter can't find any place it has write access to to write the cache, it will not write it, yes. That means it will have to re-parse the source file into bytecode (and fail to write the bytecode to cache) every time it is loaded.
jbverschoor 6y ago
It makes more sense to use a second stream or named stream.
- PeterisP 6y ago
  That sounds like a recipe for even more people accidentally storing and distributing that bytecode without wanting to do so, because it won't be immediately visible that it's there, many tools don't show or highlight the existence of those streams. I'd argue that it's more appropriate to follow the 'explicit is better than implicit' from the 'guidelines' of https://www.python.org/dev/peps/pep-0020/ https://www.python.org/dev/peps/pep-0020/ .
deleted 6y ago
[deleted]

eesmith 6y ago

It already (since 3.2, as the article points out) uses a cache directory. Why would an OS-specific one help? You can also set PYTHONPYCACHEPREFIX if you want to use 'a mirror directory tree at this path, instead of in __pycache__ directories within the source tree' - https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPYCACHEPREFIX https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPY... The PEP mentions your case at https://www.python.org/dev/peps/pep-3147/#case-5-read-only-file-systems https://www.python.org/dev/peps/pep-3147/#case-5-read-only-f... . But honestly, I don't really follow the answer. I think it means "ignore creation and write failures."

deathanatos 6y ago
> Why would an OS-specific one help? E.g., if you want to back up your home dir, and omit caches, since they can be regenerated. It's a lot easier if programs write their cache data to ~/.cache / $XDG_CACHE_HOME than if they intermix it / scatter it about.
- deleted 6y ago
  [deleted]
- eesmith 6y ago
  But how does it help to have one directory for all Linux-based OSes (or one directory for RHEL, one for Ubuntu, one for Debian, etc.), and one for FreeBSD and one for OpenIndiana? At least, that's what I interpret "OS-specific" to mean.
  - a_t48 6y ago
    Windows exists, too. Probably would be in %APPDATA% or %TMP%
  - Chickenosaurus 6y ago
    I suspect his thought was to use something like "%UserProfile%/pycache/" on windows and "$HOME/.pycache/" on Linux.
  - deathanatos 6y ago
    More along the line of the original comment you're replying to, if the cache was in, say, ~/.cache, then it won't get swept up in the repository's commits, since the cache data is no longer inside the repository's working directory. Then, it never gets uploaded to GitHub, and this security issue never happens. I have seen a surprising number of people — some who are engineers by profession too, and ought to know better — just git add everything, and then commit it all without looking. One should review the diff one has staged to see if it is correct, but alas…
    - eesmith 6y ago
      That's possible with 3.8's PYTHONPYCACHEPREFIX, yes? Perhaps it's worthwhile for someone to blog about this more/promote this as a best practice? Though what's missing is the hook to connected it as appropriate for the given platform. I see now that "OS-specific" was meant to be interpreted as "the OS-defined mechanism to find a cache directory", not "a cache directory which differs for each operating system". I would not have been confused by the term "platform dependent", which is what Python's tmpdir documentation uses, as in: "The default directory is chosen from a platform-dependent list" at https://docs.python.org/3/library/tempfile.html?highlight=tmpdir#tempfile.mkstemp https://docs.python.org/3/library/tempfile.html?highlight=tm... .
globular-toast 6y ago
That env var is new in 3.8. I've been looking forward to it to stop docker writing root owned cache files back to a bind mounted file system. I might set it to $HOME/.cache/python globally as well.

pdonis 6y ago

> what does python do if it doesn't have write permission in the current working directory? Not write the cache? In the case where the interpreter can't find any place it has write access to to write the cache, it will not write it, yes. That means it will have to re-parse the source file into bytecode (and fail to write the bytecode to cache) every time it is loaded.

jbverschoor 6y ago

It makes more sense to use a second stream or named stream.

PeterisP 6y ago
That sounds like a recipe for even more people accidentally storing and distributing that bytecode without wanting to do so, because it won't be immediately visible that it's there, many tools don't show or highlight the existence of those streams. I'd argue that it's more appropriate to follow the 'explicit is better than implicit' from the 'guidelines' of https://www.python.org/dev/peps/pep-0020/ https://www.python.org/dev/peps/pep-0020/ .

deleted 6y ago

[deleted]