5 ms·

Python utility for tracking third party dependencies within a library

samwillis 4y ago
This looks like a real useful tool for large projects, it can be quite possible to loose track of what a specific dependancy is used for. I also like the idea of making an import lazy so in monolithic app you could have a deployment that excludes functionality, and exclude its dependancies. When I read the title I was hoping for something else though, what I would love is a tool that logs and potentially blocks unexpected IO operations on a library basis. With the increasing common supply chain attacks we are seeing (there was a PyPI one just the other day), having a way to at least report on unexpected activity if not help prevent it would be brilliant. Has anyone ever found a tool like that? (Obviously the ultimate solution would be an outbound firewall, but it seems be that although you can easily do this in a VM or bare metal, I haven't seen any PAAS platforms have that sort of capability)
- woodruffw 4y ago
  > When I read the title I was hoping for something else though, what I would love is a tool that logs and potentially blocks unexpected IO operations on a library basis. With the increasing common supply chain attacks we are seeing (there was a PyPI one just the other day), having a way to at least report on unexpected activity if not help prevent would be brilliant. Has anyone ever found a tool like. that? You could do something close to that with Python's audit hooks, which were introduced with 3.8[1]. One massive caveat: audit hooks can be disabled by an attacker with the ability to control the interpreter, and are not perfect (there's plenty of things they don't cover.) (More generally: this kind of auditing/restriction falls under the umbrella of "capability management." OpenBSD's pledge[2] is another example.) [1]: https://peps.python.org/pep-0578/ https://peps.python.org/pep-0578/ [2]: https://man.openbsd.org/pledge.2 https://man.openbsd.org/pledge.2
- ashishbijlani 4y ago
  https://github.com/ossillate-inc/packj https://github.com/ossillate-inc/packj analyzes Python/NPM packages for risky code and metadata attributes. Uses static code analysis. We found a bunch of malicious packages on PyPI using the tool, which have now been taken down: examples https://packj.dev/malware https://packj.dev/malware [disclosure: I’m one of the developers]
simonw 4y ago
Tried this on one of my projects, it's neat. python3 -m import_tracker --name datasette --recursive | jq { "datasette": [ "aiofiles", "click", "markupsafe", "mergedeep", "pluggy", "yaml" ], "datasette.version": [], "datasette.utils.shutil_backport": [ "click", "markupsafe", "mergedeep", "yaml" ], "datasette.utils.sqlite": [ "click", "markupsafe", "mergedeep", "yaml" ], "datasette.utils": [ "click", "markupsafe", "mergedeep", "yaml" ], "datasette.utils.asgi": [ "aiofiles", "click", "markupsafe", "mergedeep", "yaml" ], "datasette.hookspecs": [ "aiofiles", "click", "markupsafe", "mergedeep", "pluggy", "yaml" ] } Related tool: pipdeptree - here's the output from that against a project that installs a lot of extra stuff: https://github.com/simonw/latest-datasette-with-all-plugins/blob/main/pipdeptree.txt https://github.com/simonw/latest-datasette-with-all-plugins/...
- blakesley 4y ago
  Is this the equivalent of Poetry's `poetry show --tree`?
deleted 4y ago
[deleted]
barefeg 4y ago
Could I use the lazy import to define a single set of dependencies of a monorepo and then load only the required subset for each project?
- beisner 4y ago
  That’s how Bazel Python works
- shankr 4y ago
  Look at pants https://www.pantsbuild.org/ https://www.pantsbuild.org/
hrpnk 4y ago
You can use Syft [1] which generates the full software bill of materials, which includes package names, licenses for a broad set of tech stack ranging from OS level (Alpine, Debian), through Go, Ruby, Python, Java, JavaScript, etc. [1] https://github.com/anchore/syft https://github.com/anchore/syft
- woodruffw 4y ago
  Since this is about Python specifically, I'll go ahead and and highlight `pip-audit`[1] as a specialized tool for generating Python SBOMs and running audits against the official PyPI vulnerability feed. FD: My company, my work. [1]: https://github.com/trailofbits/pip-audit https://github.com/trailofbits/pip-audit
lifeisstillgood 4y ago
The couple of times I have done something similar I have found an odd outcome - first was internal (to large company codebase) and second was npm imports years ago. Both times one ended up pulling in huge numbers of dependencies (900+ npm, 600 or so internal) The point was that pretty no much what starting point one used, you pulled in much the same amount. There was a common core but even so it was like a starfish - if you start at tip of one limb, you pull in that limb and the core. start on another limb same thing. but all the limbs are about the same size it's just anecdata but it has been at the back of my mind as some kind of rule.
cubes 4y ago
This looks really neat. One thing I noticed on reading the source code, it appears to actually import the modules: Quoting the docstring on the `track_module` function: """This function executes the tracking of a single module by launching a subprocess to execute this module against the target module. The implementation of thie tracking resides in the __main__ in order to carefully control the import ecosystem. Source: https://github.com/IBM/import-tracker/blob/67a1e84e5a609e52e9bab866496af60e906d97c7/import_tracker/import_tracker.py#L33-L36 https://github.com/IBM/import-tracker/blob/67a1e84e5a609e52e... Here's the actual subprocess call: https://github.com/IBM/import-tracker/blob/67a1e84e5a609e52e9bab866496af60e906d97c7/import_tracker/import_tracker.py#L96-L97 https://github.com/IBM/import-tracker/blob/67a1e84e5a609e52e... # Launch the process proc = subprocess.Popen(shlex.split(cmd), stdout=subprocess.PIPE, env=env) I think this is clever, and maybe even necessary, but feels risky to do on unaudited third-party Python libraries. Maybe I'm misunderstanding something?
- jreese 4y ago
  > I think this is clever, and maybe even necessary, but feels risky to do on unaudited third-party Python libraries. This is why my coworker built the project he called "dowsing"; it tries to understand as much as possible from the setup.py's AST, without actually executing it. https://github.com/python-packaging/dowsing https://github.com/python-packaging/dowsing
  - cubes 4y ago
    Neat, I'll take a look! I thought I was going to need to write something similar!
- gabegoodhart 4y ago
  Hi, I'm the main author of import_tracker. Thanks for taking the time to dig into it! It's a really interesting point that the subproces.Popen could itself be a security concern. The command that's being executed is executing the __main__ of the import_tracker library itself (which is not something that a user can't configure), so is your concern that import_tracker itself is untrusted and might be a concern for users running this on their machines? For context on why I'm using the suprocess here, this allows the tracking to correctly allocate dependencies that are imported more than once (think my_lib.submod1 and my_lib.submod2 both need tensorflow, but my_lib.submod3 doesn't).
throwamon 4y ago
Related: "Probably the most complete python dependency database" - https://github.com/DavHau/pypi-deps-db https://github.com/DavHau/pypi-deps-db > This data allows deterministic dependency resolution which is required by mach-nix[1] to generate reproducible python environments. [1]: https://github.com/DavHau/mach-nix/ https://github.com/DavHau/mach-nix/
DishyDev 4y ago
This tool has reminded me a few years ago I created a helper web utility that let me search Python libraries and get a tree view of their dependencies, and some license info. I had to do a lot of manual Python library compliance before we had tools like Blackduck. It accepts dependencies in requirements.txt format (e.g. Django==3.1 or tensorflow) https://pydepchecker.z33.web.core.windows.net/ https://pydepchecker.z33.web.core.windows.net/ It's got a few shortcomings. Dependency resolution in Python is pretty difficult to work out when you've got a lot of libraries with common dependencies. And the license info on Pypi isn't always correct. But it's always been a quick useful tool for me.

This looks like a real useful tool for large projects, it can be quite possible to loose track of what a specific dependancy is used for. I also like the idea of making an import lazy so in monolithic app you could have a deployment that excludes functionality, and exclude its dependancies. When I read the title I was hoping for something else though, what I would love is a tool that logs and potentially blocks unexpected IO operations on a library basis. With the increasing common supply chain attacks we are seeing (there was a PyPI one just the other day), having a way to at least report on unexpected activity if not help prevent it would be brilliant. Has anyone ever found a tool like that? (Obviously the ultimate solution would be an outbound firewall, but it seems be that although you can easily do this in a VM or bare metal, I haven't seen any PAAS platforms have that sort of capability)

woodruffw 4y ago
> When I read the title I was hoping for something else though, what I would love is a tool that logs and potentially blocks unexpected IO operations on a library basis. With the increasing common supply chain attacks we are seeing (there was a PyPI one just the other day), having a way to at least report on unexpected activity if not help prevent would be brilliant. Has anyone ever found a tool like. that? You could do something close to that with Python's audit hooks, which were introduced with 3.8[1]. One massive caveat: audit hooks can be disabled by an attacker with the ability to control the interpreter, and are not perfect (there's plenty of things they don't cover.) (More generally: this kind of auditing/restriction falls under the umbrella of "capability management." OpenBSD's pledge[2] is another example.) [1]: https://peps.python.org/pep-0578/ https://peps.python.org/pep-0578/ [2]: https://man.openbsd.org/pledge.2 https://man.openbsd.org/pledge.2
ashishbijlani 4y ago
https://github.com/ossillate-inc/packj https://github.com/ossillate-inc/packj analyzes Python/NPM packages for risky code and metadata attributes. Uses static code analysis. We found a bunch of malicious packages on PyPI using the tool, which have now been taken down: examples https://packj.dev/malware https://packj.dev/malware [disclosure: I’m one of the developers]

simonw 4y ago

Tried this on one of my projects, it's neat. python3 -m import_tracker --name datasette --recursive | jq { "datasette": [ "aiofiles", "click", "markupsafe", "mergedeep", "pluggy", "yaml" ], "datasette.version": [], "datasette.utils.shutil_backport": [ "click", "markupsafe", "mergedeep", "yaml" ], "datasette.utils.sqlite": [ "click", "markupsafe", "mergedeep", "yaml" ], "datasette.utils": [ "click", "markupsafe", "mergedeep", "yaml" ], "datasette.utils.asgi": [ "aiofiles", "click", "markupsafe", "mergedeep", "yaml" ], "datasette.hookspecs": [ "aiofiles", "click", "markupsafe", "mergedeep", "pluggy", "yaml" ] } Related tool: pipdeptree - here's the output from that against a project that installs a lot of extra stuff: https://github.com/simonw/latest-datasette-with-all-plugins/blob/main/pipdeptree.txt https://github.com/simonw/latest-datasette-with-all-plugins/...

blakesley 4y ago
Is this the equivalent of Poetry's `poetry show --tree`?

deleted 4y ago

[deleted]

barefeg 4y ago

Could I use the lazy import to define a single set of dependencies of a monorepo and then load only the required subset for each project?

beisner 4y ago
That’s how Bazel Python works
shankr 4y ago
Look at pants https://www.pantsbuild.org/ https://www.pantsbuild.org/

hrpnk 4y ago

You can use Syft [1] which generates the full software bill of materials, which includes package names, licenses for a broad set of tech stack ranging from OS level (Alpine, Debian), through Go, Ruby, Python, Java, JavaScript, etc. [1] https://github.com/anchore/syft https://github.com/anchore/syft

woodruffw 4y ago
Since this is about Python specifically, I'll go ahead and and highlight `pip-audit`[1] as a specialized tool for generating Python SBOMs and running audits against the official PyPI vulnerability feed. FD: My company, my work. [1]: https://github.com/trailofbits/pip-audit https://github.com/trailofbits/pip-audit

lifeisstillgood 4y ago

The couple of times I have done something similar I have found an odd outcome - first was internal (to large company codebase) and second was npm imports years ago. Both times one ended up pulling in huge numbers of dependencies (900+ npm, 600 or so internal) The point was that pretty no much what starting point one used, you pulled in much the same amount. There was a common core but even so it was like a starfish - if you start at tip of one limb, you pull in that limb and the core. start on another limb same thing. but all the limbs are about the same size it's just anecdata but it has been at the back of my mind as some kind of rule.

cubes 4y ago

This looks really neat. One thing I noticed on reading the source code, it appears to actually import the modules: Quoting the docstring on the `track_module` function: """This function executes the tracking of a single module by launching a subprocess to execute this module against the target module. The implementation of thie tracking resides in the __main__ in order to carefully control the import ecosystem. Source: https://github.com/IBM/import-tracker/blob/67a1e84e5a609e52e9bab866496af60e906d97c7/import_tracker/import_tracker.py#L33-L36 https://github.com/IBM/import-tracker/blob/67a1e84e5a609e52e... Here's the actual subprocess call: https://github.com/IBM/import-tracker/blob/67a1e84e5a609e52e9bab866496af60e906d97c7/import_tracker/import_tracker.py#L96-L97 https://github.com/IBM/import-tracker/blob/67a1e84e5a609e52e... # Launch the process proc = subprocess.Popen(shlex.split(cmd), stdout=subprocess.PIPE, env=env) I think this is clever, and maybe even necessary, but feels risky to do on unaudited third-party Python libraries. Maybe I'm misunderstanding something?

jreese 4y ago
> I think this is clever, and maybe even necessary, but feels risky to do on unaudited third-party Python libraries. This is why my coworker built the project he called "dowsing"; it tries to understand as much as possible from the setup.py's AST, without actually executing it. https://github.com/python-packaging/dowsing https://github.com/python-packaging/dowsing
- cubes 4y ago
  Neat, I'll take a look! I thought I was going to need to write something similar!
gabegoodhart 4y ago
Hi, I'm the main author of import_tracker. Thanks for taking the time to dig into it! It's a really interesting point that the subproces.Popen could itself be a security concern. The command that's being executed is executing the __main__ of the import_tracker library itself (which is not something that a user can't configure), so is your concern that import_tracker itself is untrusted and might be a concern for users running this on their machines? For context on why I'm using the suprocess here, this allows the tracking to correctly allocate dependencies that are imported more than once (think my_lib.submod1 and my_lib.submod2 both need tensorflow, but my_lib.submod3 doesn't).

throwamon 4y ago

Related: "Probably the most complete python dependency database" - https://github.com/DavHau/pypi-deps-db https://github.com/DavHau/pypi-deps-db > This data allows deterministic dependency resolution which is required by mach-nix[1] to generate reproducible python environments. [1]: https://github.com/DavHau/mach-nix/ https://github.com/DavHau/mach-nix/

DishyDev 4y ago

This tool has reminded me a few years ago I created a helper web utility that let me search Python libraries and get a tree view of their dependencies, and some license info. I had to do a lot of manual Python library compliance before we had tools like Blackduck. It accepts dependencies in requirements.txt format (e.g. Django==3.1 or tensorflow) https://pydepchecker.z33.web.core.windows.net/ https://pydepchecker.z33.web.core.windows.net/ It's got a few shortcomings. Dependency resolution in Python is pretty difficult to work out when you've got a lot of libraries with common dependencies. And the license info on Pypi isn't always correct. But it's always been a quick useful tool for me.