4 ms·
For Python releases, this is quite a big problem. Python packages consist of many different downloadable files - for example, the Datadog APM library has over 6
by orf 3y ago
For Python releases, this is quite a big problem. Python packages consist of many different downloadable files - for example, the Datadog APM library has over 60 different files[1] depending on the platform and Python version.
There's nothing to enforce that each of these files contains the same code - and indeed, they shouldn't: each platform-specific .whl file contains platform-specific binaries.
But for the python and other platform-agnostic code, it can and does differ if the publishing process is done manually.
One example of this is an intel project called "devcloud" - a specific file was included only in the .whl release (and not the source distribution) that contained an AWS access key[2]. The Amazon pay SDK also fell victim to this - the developer accidentally included an integration test that contained a key[3]. Someone else included a markdown file containing ~400 OpenAI keys in a similar manner, only present in a single .whl file out of several.
There are _lots_ of examples of this, so many that I created a project to attempt to quantify this, prevent accidental credential leaks and bring greater visibility to the contents of PyPI by publishing all the contents of PyPI releases to Github and indexing them: https://py-code.org/ https://py-code.org/
There's a hosted clickhouse dataset available here: https://py-code.org/datasets https://py-code.org/datasets. On my to-do list is to write a query to find differences in files within the same package version, where they should be the same. This is almost always an indication of something malicious or accidental.
"Package integrity" is a really interesting + complex area when you add a platform-specific dimension to it. For Python specifically, I wonder if the "separate whl per platform" approach is a good one. Maybe a layered approach would be better - the "source" is the source of truth, and pre-built binaries _only_ contain pre-built binaries and not the duplicated "source"?
1. https://pypi.org/project/ddtrace/#files https://pypi.org/project/ddtrace/#files
2. https://inspector.pypi.io/project/devcloud/0.0.2/packages/9f/59/739768bdffb30404afe359bea1975664e78d2d272124a08e98591ef3e8a0/devcloud-0.0.2-py3-none-any.whl/devcloud/result.py#line.16 https://inspector.pypi.io/project/devcloud/0.0.2/packages/9f...
3. https://inspector.pypi.io/project/amazon-pay/2.5.1/packages/db/20/cc6965de19ba9839823043d50de44ececffeeaa6712c77b16721224ad9c3/amazon_pay-2.5.1.tar.gz/amazon_pay-2.5.1/amazon_pay/integration_test.py#line.6 https://inspector.pypi.io/project/amazon-pay/2.5.1/packages/...