8 ms·

Finding secrets by decompiling Python bytecode in public repositories

165 points by gilad 6y ago

maxeonyx 6y ago
This would be solved if python used an (OS-specific) cache directory for its .pyc files. I have always disliked .pyc files... here's a concrete reason! Question: what does python do if it doesn't have write permission in the current working directory? Not write the cache?
- eesmith 6y ago
  It already (since 3.2, as the article points out) uses a cache directory. Why would an OS-specific one help? You can also set PYTHONPYCACHEPREFIX if you want to use 'a mirror directory tree at this path, instead of in __pycache__ directories within the source tree' - https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPYCACHEPREFIX https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPY... The PEP mentions your case at https://www.python.org/dev/peps/pep-3147/#case-5-read-only-file-systems https://www.python.org/dev/peps/pep-3147/#case-5-read-only-f... . But honestly, I don't really follow the answer. I think it means "ignore creation and write failures."
  - deathanatos 6y ago
    > Why would an OS-specific one help? E.g., if you want to back up your home dir, and omit caches, since they can be regenerated. It's a lot easier if programs write their cache data to ~/.cache / $XDG_CACHE_HOME than if they intermix it / scatter it about.
    - deleted 6y ago
      [deleted]
    - eesmith 6y ago
      But how does it help to have one directory for all Linux-based OSes (or one directory for RHEL, one for Ubuntu, one for Debian, etc.), and one for FreeBSD and one for OpenIndiana? At least, that's what I interpret "OS-specific" to mean.
      - a_t48 6y ago
        Windows exists, too. Probably would be in %APPDATA% or %TMP%
      - Chickenosaurus 6y ago
        I suspect his thought was to use something like "%UserProfile%/pycache/" on windows and "$HOME/.pycache/" on Linux.
      - deathanatos 6y ago
        More along the line of the original comment you're replying to, if the cache was in, say, ~/.cache, then it won't get swept up in the repository's commits, since the cache data is no longer inside the repository's working directory. Then, it never gets uploaded to GitHub, and this security issue never happens. I have seen a surprising number of people — some who are engineers by profession too, and ought to know better — just git add everything, and then commit it all without looking. One should review the diff one has staged to see if it is correct, but alas…
        eesmith 6y ago
        That's possible with 3.8's PYTHONPYCACHEPREFIX, yes? Perhaps it's worthwhile for someone to blog about this more/promote this as a best practice? Though what's missing is the hook to connected it as appropriate for the given platform. I see now that "OS-specific" was meant to be interpreted as "the OS-defined mechanism to find a cache directory", not "a cache directory which differs for each operating system". I would not have been confused by the term "platform dependent", which is what Python's tmpdir documentation uses, as in: "The default directory is chosen from a platform-dependent list" at https://docs.python.org/3/library/tempfile.html?highlight=tmpdir#tempfile.mkstemp https://docs.python.org/3/library/tempfile.html?highlight=tm... .
  - globular-toast 6y ago
    That env var is new in 3.8. I've been looking forward to it to stop docker writing root owned cache files back to a bind mounted file system. I might set it to $HOME/.cache/python globally as well.
- pdonis 6y ago
  > what does python do if it doesn't have write permission in the current working directory? Not write the cache? In the case where the interpreter can't find any place it has write access to to write the cache, it will not write it, yes. That means it will have to re-parse the source file into bytecode (and fail to write the bytecode to cache) every time it is loaded.
- jbverschoor 6y ago
  It makes more sense to use a second stream or named stream.
  - PeterisP 6y ago
    That sounds like a recipe for even more people accidentally storing and distributing that bytecode without wanting to do so, because it won't be immediately visible that it's there, many tools don't show or highlight the existence of those streams. I'd argue that it's more appropriate to follow the 'explicit is better than implicit' from the 'guidelines' of https://www.python.org/dev/peps/pep-0020/ https://www.python.org/dev/peps/pep-0020/ .
- deleted 6y ago
  [deleted]
shockinglytrue 6y ago
Highly recommend "export PYTHONDONTWRITEBYTECODE=1" in your bashrc and just forget about it. Pyc files are still an important optimization on modern machines in some circumstances (especially with huge oft-restarted apps), but the autogeneration behaviour has always been a pain in the ass The bulk of your pycs are generated during package install. What tends to remain in the usual case is a handful of files representing app code or similar.
- the_jeremy 6y ago
  Why do you highly recommend this? You can set git (or other VCS) settings to avoid committing .pyc files. They're not large files; I don't see the downside.
  - shockinglytrue 6y ago
    Difficulty with git basics is the least of your worries. For example pycs are fundamentally racy, it is quite possible to have a .py newer than a .pyc depending on how unlucky you were with an in-progress deployment, or some tool that never updated the .py timestamp. Python continues to execute the pyc even though the code changed, since the minimal benefit of the pyc would be rendered significantly inert should Python use any kind of strong check (instead of just comparing second-granularity stat() output) to ensure the cached bytecode matches the source. In this way, without your permission, Python silently plays undesirable code execution roulette with your computer every time it starts, for as long as you have the feature enabled. I have lost count of the number of times I've seen someone lose an hour due to it. I can also count many instances of QA environments becoming inexplicably bricked by it. The correct fix for this requires opening the .py and hashing its content, at least doubling the amount of IO required to start a program. They were a great feature when parsing small files was noticeably slow, but this hasn't been true for almost 20 years. It's therefore worth turning the question around: why do you think pyc files are useful?
    - eesmith 6y ago
      Python 3.7 added support for hash-based cache files as an alternative to time-based. https://docs.python.org/3/reference/import.html#pyc-invalidation https://docs.python.org/3/reference/import.html#pyc-invalida... Verifying a hash is a bit slower than checking the timestamp but far faster than parsing and byte compiling the source file, so I don't think this option is "significantly inert".
oefrha 6y ago
Just reading the script from TFA, it attempts to find secrets.pyc and decompile it, but doesn't even check if secrets.py is also in the repo. A glance at search results (I just used GitHub's web interface, didn't bother to run the code) tells me when secrets.pyc is committed, secrets.py comes with it at least the vast majority of time. I guess the author did find cases where secrets.pyc is committed but secrets.py is not? It's hard to fathom how that could have happened (especially inside "organization" settings). Sounds like the result of absolute rookies in both Python and git following a tutorial with a step "add secrets.py to .gitignore" but unfortunately takes ignoring __pycache__ and ﹡.pyc for granted, which is too much to ask for some people. > it is very easy for an experienced programmer to accidentally commit their secrets No, it doesn't take an experienced programmer to put __pycache__ and ﹡.pyc to global ignore, or use a gitignore boilerplate at project creation, or notice random unwanted files during code review.
- alexchamberlain 6y ago
  I actually insist at work that most repositories don’t have a .gitignore; just setup a reasonable global one and you’re done. OSS or repos with a large number of contributors are generally exceptions here.
  - dastx 6y ago
    Sure but reasonable for me is different from reasonable for you. In addition, I might miss files that are repo specific. Insisting repos don't have a .gitignore is terrible advice. It doesn't cost much to maintain it if at all.
  - orf 6y ago
    That sounds like poor advice with unclear rationale behind it. There are definitely project specific ignores that you’d want to set up, and if you’re working with projects with multiple different languages, “one gitignore to rule them all” fast becomes a mess.
    - alexchamberlain 6y ago
tucnak 6y ago
I read "Finding secrets by decompiling Python bytecode in public restrooms" by accident. It never occured to me that anyone would do THAT in there.
- eat_veggies 6y ago
  there's bytecode sharpied on the walls
scarface74 6y ago
Public Service Announcement. While for the other secrets, I’ve got nothing, there is never a reason to have AWS secret keys in your code or in application specific configuration files. Every AWS SDK will automatically read your keys from your config file in your home directory locally. Just run aws configure When you run your code on EC2, Lambda or ECS, the same SDK’s will automatically get the keys associated with the attached role.
- robbyt 6y ago
  For others non AWS credentials, just use environment variables
  - lysium 6y ago
    They have to be set up somewhere, though.
    - thanksforfish 6y ago
      It should be done out of band. Repo should have a `README` that says the secrets are in the team 1password account, talk to <team member> to get a 1password account, and get added to the group that has access to the vault with the creds. The repo should have a source script that will pull the credentials[1] and `export` them to your ENV. `direnv` can make that happen automatically[2], or you can run that script from your `.bashrc` or similar. You can do something similar with your favorite secrets manager. I've used a similar approach before, with good results. [1] https://support.1password.com/command-line-getting-started/ https://support.1password.com/command-line-getting-started/ [2] https://direnv.net/ https://direnv.net/
      - Someone 6y ago
        If you use AWS, “your favorite secrets manager” should be AWS SecretsManager, and you should use an AWS Role to limit access to the secrets. Have your code fetch any secrets it needs from AWS Secrets manager, using the name of the secret (which does not have to be kept secret, so it can be in your source repo) That way, you don’t have to put secrets in your environment, with the risk of leaking them.
xvilka 6y ago
Radare2 also supports Python bytecode of diffent versions [1]. [1] https://github.com/radareorg/radare2/tree/master/libr/asm/arch/pyc https://github.com/radareorg/radare2/tree/master/libr/asm/ar...
neurostimulant 6y ago
Another possible place to look for secrets is in public docker images. Bots are scanning github repos for secrets all the time, but what about dockerhub (and other docker images repositories)? I accidentally leaved a secret on my public docker image once and that's made me quite paranoid about it now.
- syntheticcorp 6y ago
  I scanned everything pushed to dockerhub for a few weeks but didn’t find too much interesting stuff showing up

165 points by gilad 6y ago

This would be solved if python used an (OS-specific) cache directory for its .pyc files. I have always disliked .pyc files... here's a concrete reason! Question: what does python do if it doesn't have write permission in the current working directory? Not write the cache?

eesmith 6y ago
It already (since 3.2, as the article points out) uses a cache directory. Why would an OS-specific one help? You can also set PYTHONPYCACHEPREFIX if you want to use 'a mirror directory tree at this path, instead of in __pycache__ directories within the source tree' - https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPYCACHEPREFIX https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPY... The PEP mentions your case at https://www.python.org/dev/peps/pep-3147/#case-5-read-only-file-systems https://www.python.org/dev/peps/pep-3147/#case-5-read-only-f... . But honestly, I don't really follow the answer. I think it means "ignore creation and write failures."
- deathanatos 6y ago
  > Why would an OS-specific one help? E.g., if you want to back up your home dir, and omit caches, since they can be regenerated. It's a lot easier if programs write their cache data to ~/.cache / $XDG_CACHE_HOME than if they intermix it / scatter it about.
  - deleted 6y ago
    [deleted]
  - eesmith 6y ago
    But how does it help to have one directory for all Linux-based OSes (or one directory for RHEL, one for Ubuntu, one for Debian, etc.), and one for FreeBSD and one for OpenIndiana? At least, that's what I interpret "OS-specific" to mean.
    - a_t48 6y ago
      Windows exists, too. Probably would be in %APPDATA% or %TMP%
    - Chickenosaurus 6y ago
      I suspect his thought was to use something like "%UserProfile%/pycache/" on windows and "$HOME/.pycache/" on Linux.
    - deathanatos 6y ago
      More along the line of the original comment you're replying to, if the cache was in, say, ~/.cache, then it won't get swept up in the repository's commits, since the cache data is no longer inside the repository's working directory. Then, it never gets uploaded to GitHub, and this security issue never happens. I have seen a surprising number of people — some who are engineers by profession too, and ought to know better — just git add everything, and then commit it all without looking. One should review the diff one has staged to see if it is correct, but alas…
      - eesmith 6y ago
        That's possible with 3.8's PYTHONPYCACHEPREFIX, yes? Perhaps it's worthwhile for someone to blog about this more/promote this as a best practice? Though what's missing is the hook to connected it as appropriate for the given platform. I see now that "OS-specific" was meant to be interpreted as "the OS-defined mechanism to find a cache directory", not "a cache directory which differs for each operating system". I would not have been confused by the term "platform dependent", which is what Python's tmpdir documentation uses, as in: "The default directory is chosen from a platform-dependent list" at https://docs.python.org/3/library/tempfile.html?highlight=tmpdir#tempfile.mkstemp https://docs.python.org/3/library/tempfile.html?highlight=tm... .
- globular-toast 6y ago
  That env var is new in 3.8. I've been looking forward to it to stop docker writing root owned cache files back to a bind mounted file system. I might set it to $HOME/.cache/python globally as well.
pdonis 6y ago
> what does python do if it doesn't have write permission in the current working directory? Not write the cache? In the case where the interpreter can't find any place it has write access to to write the cache, it will not write it, yes. That means it will have to re-parse the source file into bytecode (and fail to write the bytecode to cache) every time it is loaded.
jbverschoor 6y ago
It makes more sense to use a second stream or named stream.
- PeterisP 6y ago
  That sounds like a recipe for even more people accidentally storing and distributing that bytecode without wanting to do so, because it won't be immediately visible that it's there, many tools don't show or highlight the existence of those streams. I'd argue that it's more appropriate to follow the 'explicit is better than implicit' from the 'guidelines' of https://www.python.org/dev/peps/pep-0020/ https://www.python.org/dev/peps/pep-0020/ .
deleted 6y ago
[deleted]

shockinglytrue 6y ago

Highly recommend "export PYTHONDONTWRITEBYTECODE=1" in your bashrc and just forget about it. Pyc files are still an important optimization on modern machines in some circumstances (especially with huge oft-restarted apps), but the autogeneration behaviour has always been a pain in the ass The bulk of your pycs are generated during package install. What tends to remain in the usual case is a handful of files representing app code or similar.

the_jeremy 6y ago
Why do you highly recommend this? You can set git (or other VCS) settings to avoid committing .pyc files. They're not large files; I don't see the downside.
- shockinglytrue 6y ago
  Difficulty with git basics is the least of your worries. For example pycs are fundamentally racy, it is quite possible to have a .py newer than a .pyc depending on how unlucky you were with an in-progress deployment, or some tool that never updated the .py timestamp. Python continues to execute the pyc even though the code changed, since the minimal benefit of the pyc would be rendered significantly inert should Python use any kind of strong check (instead of just comparing second-granularity stat() output) to ensure the cached bytecode matches the source. In this way, without your permission, Python silently plays undesirable code execution roulette with your computer every time it starts, for as long as you have the feature enabled. I have lost count of the number of times I've seen someone lose an hour due to it. I can also count many instances of QA environments becoming inexplicably bricked by it. The correct fix for this requires opening the .py and hashing its content, at least doubling the amount of IO required to start a program. They were a great feature when parsing small files was noticeably slow, but this hasn't been true for almost 20 years. It's therefore worth turning the question around: why do you think pyc files are useful?
  - eesmith 6y ago
    Python 3.7 added support for hash-based cache files as an alternative to time-based. https://docs.python.org/3/reference/import.html#pyc-invalidation https://docs.python.org/3/reference/import.html#pyc-invalida... Verifying a hash is a bit slower than checking the timestamp but far faster than parsing and byte compiling the source file, so I don't think this option is "significantly inert".

oefrha 6y ago

Just reading the script from TFA, it attempts to find secrets.pyc and decompile it, but doesn't even check if secrets.py is also in the repo. A glance at search results (I just used GitHub's web interface, didn't bother to run the code) tells me when secrets.pyc is committed, secrets.py comes with it at least the vast majority of time. I guess the author did find cases where secrets.pyc is committed but secrets.py is not? It's hard to fathom how that could have happened (especially inside "organization" settings). Sounds like the result of absolute rookies in both Python and git following a tutorial with a step "add secrets.py to .gitignore" but unfortunately takes ignoring __pycache__ and ﹡.pyc for granted, which is too much to ask for some people. > it is very easy for an experienced programmer to accidentally commit their secrets No, it doesn't take an experienced programmer to put __pycache__ and ﹡.pyc to global ignore, or use a gitignore boilerplate at project creation, or notice random unwanted files during code review.

alexchamberlain 6y ago
I actually insist at work that most repositories don’t have a .gitignore; just setup a reasonable global one and you’re done. OSS or repos with a large number of contributors are generally exceptions here.
- dastx 6y ago
  Sure but reasonable for me is different from reasonable for you. In addition, I might miss files that are repo specific. Insisting repos don't have a .gitignore is terrible advice. It doesn't cost much to maintain it if at all.
- orf 6y ago
  That sounds like poor advice with unclear rationale behind it. There are definitely project specific ignores that you’d want to set up, and if you’re working with projects with multiple different languages, “one gitignore to rule them all” fast becomes a mess.
  - alexchamberlain 6y ago

tucnak 6y ago

I read "Finding secrets by decompiling Python bytecode in public restrooms" by accident. It never occured to me that anyone would do THAT in there.

eat_veggies 6y ago
there's bytecode sharpied on the walls

scarface74 6y ago

Public Service Announcement. While for the other secrets, I’ve got nothing, there is never a reason to have AWS secret keys in your code or in application specific configuration files. Every AWS SDK will automatically read your keys from your config file in your home directory locally. Just run aws configure When you run your code on EC2, Lambda or ECS, the same SDK’s will automatically get the keys associated with the attached role.

robbyt 6y ago
For others non AWS credentials, just use environment variables
- lysium 6y ago
  They have to be set up somewhere, though.
  - thanksforfish 6y ago
    It should be done out of band. Repo should have a `README` that says the secrets are in the team 1password account, talk to <team member> to get a 1password account, and get added to the group that has access to the vault with the creds. The repo should have a source script that will pull the credentials[1] and `export` them to your ENV. `direnv` can make that happen automatically[2], or you can run that script from your `.bashrc` or similar. You can do something similar with your favorite secrets manager. I've used a similar approach before, with good results. [1] https://support.1password.com/command-line-getting-started/ https://support.1password.com/command-line-getting-started/ [2] https://direnv.net/ https://direnv.net/
    - Someone 6y ago
      If you use AWS, “your favorite secrets manager” should be AWS SecretsManager, and you should use an AWS Role to limit access to the secrets. Have your code fetch any secrets it needs from AWS Secrets manager, using the name of the secret (which does not have to be kept secret, so it can be in your source repo) That way, you don’t have to put secrets in your environment, with the risk of leaking them.

xvilka 6y ago

Radare2 also supports Python bytecode of diffent versions [1]. [1] https://github.com/radareorg/radare2/tree/master/libr/asm/arch/pyc https://github.com/radareorg/radare2/tree/master/libr/asm/ar...

neurostimulant 6y ago

Another possible place to look for secrets is in public docker images. Bots are scanning github repos for secrets all the time, but what about dockerhub (and other docker images repositories)? I accidentally leaved a secret on my public docker image once and that's made me quite paranoid about it now.

syntheticcorp 6y ago
I scanned everything pushed to dockerhub for a few weeks but didn’t find too much interesting stuff showing up