4 ms·

You know what we need? In both python and JS, and every other scripting language, we should be able to import packages from a url, but with a sha384 integrity c

You know what we need? In both python and JS, and every other scripting language, we should be able to import packages from a url, but with a sha384 integrity check like exists in HTML. Not sure why they didn't adopt this into JS or Deno. Otherwise installing random scripts is a security risk

woodruffw 2y ago
Python has fully-hashed requirements[1], which is what you'd use to assert the integrity of your dependencies. These work with both `pip` and `uv`. You can't use them to directly import the package, but that's more because "packages" aren't really part of Python's import machinery at all. (Note that hashes themselves don't make "random scripts" not a security risk, since asserting the hash of malware doesn't make it not-malware. You still need to establish a trust relationship with the hash itself, which decomposes to the basic problem of trust and identity distribution.) [1]: https://pip.pypa.io/en/stable/topics/secure-installs/ https://pip.pypa.io/en/stable/topics/secure-installs/
- gregmac 2y ago
  Good point, but it's still a very useful way to ensure it doesn't get swapped out underneath you. Transitive dependencies are still a problem though. You kind of fall back to needing a lock file or specifying everything explicitly.
- 8n4vidtmkvmk 2y ago
  Right, still a security risk, but at least if I come back to a project after a year or two I can know that even if some malicious group took over a project, they at least didn't backport a crypto-miner or worse into my script.
zahlman 2y ago
The code that you obtain for a Python "package" does not have any inherent mapping to a "package" that you import in the code. The name overload is recognized as unfortunate; the documentation writing community has been promoting the terms "distribution package" and "import package" as a result. https://packaging.python.org/en/latest/discussions/distribution-package-vs-import-package/ https://packaging.python.org/en/latest/discussions/distribut... https://zahlman.github.io/posts/2024/12/24/python-packaging-1/ https://zahlman.github.io/posts/2024/12/24/python-packaging-... While you could of course put an actual Python code file at a URL, that wouldn't solve the problem for anything involving compiled extensions in C, Fortran etc. You can't feasibly support NumPy this way, for example. That said, there are sufficient hooks in Numpy's `import` machinery that you can make `import foo` programmatically compute a URL (assuming that the name `foo` is enough information to determine the URL), download the code and create and import the necessary `module` object; and you can add this with appropriate priority to the standard set of strategies Python uses for importing modules. A full description of this process is out of scope for a HN comment, but relevant documentation: https://docs.python.org/3/library/importlib.html https://docs.python.org/3/library/importlib.html https://docs.python.org/3/library/sys.html#sys.meta_path https://docs.python.org/3/library/sys.html#sys.meta_path
HumanOstrich 2y ago
Where do you initially get the magical sha384 hash that proves the integrity of the package the first time it's imported?
- 8n4vidtmkvmk 2y ago
  Same way we do in JS-land: https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity#tools_for_generating_sri_hashes https://developer.mozilla.org/en-US/docs/Web/Security/Subres... tl;dr use `openssl` on command-line to compute the hash. Ideally, any package repositories ought to publish the hash for your convenience. This of course does nothing to prove that the package is safe to use, just that it won't change out from under your nose.
AgentME 2y ago
Deno and npm both store the hashes of all the dependencies you use in a lock file and verify them on future reinstalls.
- 8n4vidtmkvmk 2y ago
  The lockfile is good, but I'm talking about this inline dependency syntax, # dependencies = ['requests', 'beautifulsoup4'] And likewise, Deno can import by URL. Neither include an integrity hash. For JS, I'd suggest import * as goodlib from 'https://verysecure.com/notmalicious.mjs' with { integrity="sha384-xxx" } which mirrors https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity#subresource_integrity_with_the_script_element https://developer.mozilla.org/en-US/docs/Web/Security/Subres... and https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/import/with https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe... The Python/UV thing will have to come up with some syntax, I don't know what. Not sure if there's a precedent for attributes.
  - deleted 2y ago
    [deleted]

woodruffw 2y ago

Python has fully-hashed requirements[1], which is what you'd use to assert the integrity of your dependencies. These work with both `pip` and `uv`. You can't use them to directly import the package, but that's more because "packages" aren't really part of Python's import machinery at all. (Note that hashes themselves don't make "random scripts" not a security risk, since asserting the hash of malware doesn't make it not-malware. You still need to establish a trust relationship with the hash itself, which decomposes to the basic problem of trust and identity distribution.) [1]: https://pip.pypa.io/en/stable/topics/secure-installs/ https://pip.pypa.io/en/stable/topics/secure-installs/

gregmac 2y ago
Good point, but it's still a very useful way to ensure it doesn't get swapped out underneath you. Transitive dependencies are still a problem though. You kind of fall back to needing a lock file or specifying everything explicitly.
8n4vidtmkvmk 2y ago
Right, still a security risk, but at least if I come back to a project after a year or two I can know that even if some malicious group took over a project, they at least didn't backport a crypto-miner or worse into my script.

zahlman 2y ago

The code that you obtain for a Python "package" does not have any inherent mapping to a "package" that you import in the code. The name overload is recognized as unfortunate; the documentation writing community has been promoting the terms "distribution package" and "import package" as a result. https://packaging.python.org/en/latest/discussions/distribution-package-vs-import-package/ https://packaging.python.org/en/latest/discussions/distribut... https://zahlman.github.io/posts/2024/12/24/python-packaging-1/ https://zahlman.github.io/posts/2024/12/24/python-packaging-... While you could of course put an actual Python code file at a URL, that wouldn't solve the problem for anything involving compiled extensions in C, Fortran etc. You can't feasibly support NumPy this way, for example. That said, there are sufficient hooks in Numpy's `import` machinery that you can make `import foo` programmatically compute a URL (assuming that the name `foo` is enough information to determine the URL), download the code and create and import the necessary `module` object; and you can add this with appropriate priority to the standard set of strategies Python uses for importing modules. A full description of this process is out of scope for a HN comment, but relevant documentation: https://docs.python.org/3/library/importlib.html https://docs.python.org/3/library/importlib.html https://docs.python.org/3/library/sys.html#sys.meta_path https://docs.python.org/3/library/sys.html#sys.meta_path

HumanOstrich 2y ago

Where do you initially get the magical sha384 hash that proves the integrity of the package the first time it's imported?

8n4vidtmkvmk 2y ago
Same way we do in JS-land: https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity#tools_for_generating_sri_hashes https://developer.mozilla.org/en-US/docs/Web/Security/Subres... tl;dr use `openssl` on command-line to compute the hash. Ideally, any package repositories ought to publish the hash for your convenience. This of course does nothing to prove that the package is safe to use, just that it won't change out from under your nose.

AgentME 2y ago

Deno and npm both store the hashes of all the dependencies you use in a lock file and verify them on future reinstalls.

8n4vidtmkvmk 2y ago
The lockfile is good, but I'm talking about this inline dependency syntax, # dependencies = ['requests', 'beautifulsoup4'] And likewise, Deno can import by URL. Neither include an integrity hash. For JS, I'd suggest import * as goodlib from 'https://verysecure.com/notmalicious.mjs' with { integrity="sha384-xxx" } which mirrors https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity#subresource_integrity_with_the_script_element https://developer.mozilla.org/en-US/docs/Web/Security/Subres... and https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/import/with https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe... The Python/UV thing will have to come up with some syntax, I don't know what. Not sure if there's a precedent for attributes.
- deleted 2y ago
  [deleted]