6 ms·

Title is a little bit hyperbolic. I get that it's a real issue that needs fixing, but a GitHub URL is not a secure package identifier, it was never designed th

Title is a little bit hyperbolic. I get that it's a real issue that needs fixing, but a GitHub URL is not a secure package identifier, it was never designed this way, and it's an unfortunate hack that it's become a defacto standard. What we really need is for language-level package managers to support signed immutable packages, and for that to start being taken seriously above the OS-level. Edit: author changed the title, it used to say something about a "GitHub's security facepalm"

tomphoolery 9y ago
> but a GitHub URL is not a secure package identifier, it was never designed this way, and it's an unfortunate hack that it's become a defacto standard. Thank you. GitHub shouldn't be the only line of defense between a malicious coder and your software.
Demiurge 9y ago
I tried to debug Go project once, and the moment I saw that, I was completely shocked that it was importing from github urls, without even tags, or versions as far as I could tell. It seems dangerous to rely on a for-profit host and individual users. If 'network' dependencies are part of the language, it should at least be non-profit, or better Go project managed host. There are many examples, like PyPi, or PECL.
- dewey 9y ago
  That's what vendoring and dependency tools are for.
- crdoconnor 9y ago
  That was the first wtf I hit when learning go that made me think that it possibly wasn't all it was cracked up to be.
- ovao 9y ago
  Go doesn't specifically rely on GitHub, but it can pull in dependencies with a simplified URL that points to a git repository hosted on GitHub. This is possible because GitHub renders an HTML meta tag on project pages that the go tool can reference to fetch information about the repository. Anyone can actually 'reflect' dependencies hosted with various VCS with this method, and gopkg.in does essentially this (while adding version pinning).
  - strkek 9y ago
    The only drawback I see to this approach is that adding a new method is not trivial, and requires recompiling the `go` executable. Makes it more difficult to implement a handler for the BitTorrent protocol to `go get` from a DHT, just for the sake of mad science.
    - Klathmon 9y ago
      That sounds like an incredible idea! Has anyone tried to do something like this before? I would absolutely run a torrent server to serve up my own and others' open source packages.
      - ovao 9y ago
        To my knowledge it hasn’t been tried. It shouldn’t be too difficult to author a tool to do so, which could itself be made go-gettable. With this you’d avoid the mess of shipping an alternate build of the go tool and all that entails.
        Klathmon 9y ago
        Yeah, the more I think about it the easier it seems. You could probably write "plugins" or shims/wrappers for most package managers out there pretty easily. And a great MVP would just be the ability to install from a magnet link.
        comex 9y ago
        This project was fairly close to that idea: https://github.com/cjb/GitTorrent https://github.com/cjb/GitTorrent I think it’s abandoned though.
      - strkek 9y ago
        I read somewhere that big companies use BitTorrent internally to update their codebase. But yeah, I'd really like for someone to develop a VCS like that (even more if it's in Go, with no Cgo, and under BSD, MIT or Apache license). It'd make distribution, immutability and mirroring so much easier.
- sannee 9y ago
  > It seems dangerous to rely on a for-profit host Unfortunately, "non-profit" hosts are somewhat in a short supply.
whack 9y ago
> What we really need is for language-level package managers to support signed immutable packages +1 Emphasis on immutable. Can someone explain why popular package managers like npm don't offer this? During the leftpad fiasco, it boggled my mind that someone could just "liberate" their packages one morning, and transitively break the build for thousands of projects. Immutable packages would have prevented this entire problem.
- AgentME 9y ago
  NPM fixed that situation by no longer allowing packages to be unpublished after 24 hours. NPM does not allow packages to republish new content under a previously-used version number. NPM by default generates a package-lock.json file (separate from package.json) which pins the versions of all dependencies (including transitive ones) and includes the hashes of the dependencies, which would protect users if npm's servers ever became malicious.
  - oneweekwonder 9y ago
    > NPM by default generates a package-lock.json file Thanks for the tidbit, seen it been added May last year to npm5[0]. Hard to keep up if you don't use the tooling activity and only depend on it for other tooling. [0]: http://blog.npmjs.org/post/161081169345/v500 http://blog.npmjs.org/post/161081169345/v500
t0mk 9y ago
I thought the same with the exaggerating. The author of the article can hardly blame github. But he makes a good point, I never realized how much I rely on github username policy in Go code :). If golang allows to import from Git URLs over HTTP or SSH, then it is the same problem with DNS as with the github usernames, isn't it? If the domain name, from which you import, will disappear, then someone else can register it and serve malicious code. This is also pretty hyperbolic. IMO It's simply solvable by refering to a commit hash, like govendor does it (or the gemfile in ruby for that matter). I am actually not sure why it's not possible to add commit hash to a golang import, I thought that $GOPATH/src contains only git repos (but I'm not sure about this).
- donatj 9y ago
  > It's simply solvable by refering to a commit hash Do you never update your dependencies?
  - bmon 9y ago
    If you don't pin your deps to a commit, what's the difference between the author deleting their account (and being replaced), and the author merging an evil commit?
    - macrael 9y ago
      There is no difference technically, but the point is that as is the author has built up a reputation and trust, and that someone taking their name inherits the trust without the reputation.
      - avar 9y ago
        It's still a bad idea not to pin your dependencies even if you trust the author. Say you want to check out some older version of the code for bisecting, and it doesn't even build anymore because it worked with some version of the dependencies that was the latest years ago, good luck figuring out what commit they were all on at the time. It's trivial to just update your own project to point to the latest upstream SHA-1s and commit that, this is why git's own facility to do this (submodules) pins you at specific upstream commits.
Zamicol 9y ago
I really feel like git having a secure package identifier built in is the right answer. I haven't heard any news of git moving to SHA3-256. Why shouldn't this be built in? Here's an old thread from 2017-03-04: https://public-inbox.org/git/CA+dhYEViN4-boZLN+5QJyE7RtX+q6a92p0C2O6TA53==BZfTrQ@mail.gmail.com/T/ https://public-inbox.org/git/CA+dhYEViN4-boZLN+5QJyE7RtX+q6a...
- craftyguy 9y ago
  > I really feel like git having a secure package identifier built in is the right answer. git commits can be gpg signed: https://git-scm.com/book/id/v2/Git-Tools-Signing-Your-Work https://git-scm.com/book/id/v2/Git-Tools-Signing-Your-Work
  - Zamicol 9y ago
    I would never use that. It uses SHA1. The problem is the hash, not the signing. A secure package identifier doesn't necessarily need to be signed. Conversely, you must have a decent hash algorithm. Your mentioned method signs the git commit hash. A signature is only as good as the hash you use, and SHA1 is not suitable. https://blog.thoughtram.io/git/2014/11/18/the-anatomy-of-a-git-commit.html https://blog.thoughtram.io/git/2014/11/18/the-anatomy-of-a-g... Also see https://github.com/git/git/commit/721cc4314cb593e799213ad5f926a1e9fc5779b0 https://github.com/git/git/commit/721cc4314cb593e799213ad5f9...
    - Shoothe 9y ago
      What do you mean by "secure package identifier"? Do you mean commit hash using strong hashing function or something else?
beached_whale 9y ago
It is git, so everything is signed. Just not authenticated. If you don't use head but a release or some tag you should be fine. Also, github supports permanent URL's

tomphoolery 9y ago

> but a GitHub URL is not a secure package identifier, it was never designed this way, and it's an unfortunate hack that it's become a defacto standard. Thank you. GitHub shouldn't be the only line of defense between a malicious coder and your software.

Demiurge 9y ago

I tried to debug Go project once, and the moment I saw that, I was completely shocked that it was importing from github urls, without even tags, or versions as far as I could tell. It seems dangerous to rely on a for-profit host and individual users. If 'network' dependencies are part of the language, it should at least be non-profit, or better Go project managed host. There are many examples, like PyPi, or PECL.

dewey 9y ago
That's what vendoring and dependency tools are for.
crdoconnor 9y ago
That was the first wtf I hit when learning go that made me think that it possibly wasn't all it was cracked up to be.
ovao 9y ago
Go doesn't specifically rely on GitHub, but it can pull in dependencies with a simplified URL that points to a git repository hosted on GitHub. This is possible because GitHub renders an HTML meta tag on project pages that the go tool can reference to fetch information about the repository. Anyone can actually 'reflect' dependencies hosted with various VCS with this method, and gopkg.in does essentially this (while adding version pinning).
- strkek 9y ago
  The only drawback I see to this approach is that adding a new method is not trivial, and requires recompiling the `go` executable. Makes it more difficult to implement a handler for the BitTorrent protocol to `go get` from a DHT, just for the sake of mad science.
  - Klathmon 9y ago
    That sounds like an incredible idea! Has anyone tried to do something like this before? I would absolutely run a torrent server to serve up my own and others' open source packages.
    - ovao 9y ago
      To my knowledge it hasn’t been tried. It shouldn’t be too difficult to author a tool to do so, which could itself be made go-gettable. With this you’d avoid the mess of shipping an alternate build of the go tool and all that entails.
      - Klathmon 9y ago
        Yeah, the more I think about it the easier it seems. You could probably write "plugins" or shims/wrappers for most package managers out there pretty easily. And a great MVP would just be the ability to install from a magnet link.
      - comex 9y ago
        This project was fairly close to that idea: https://github.com/cjb/GitTorrent https://github.com/cjb/GitTorrent I think it’s abandoned though.
    - strkek 9y ago
      I read somewhere that big companies use BitTorrent internally to update their codebase. But yeah, I'd really like for someone to develop a VCS like that (even more if it's in Go, with no Cgo, and under BSD, MIT or Apache license). It'd make distribution, immutability and mirroring so much easier.
sannee 9y ago
> It seems dangerous to rely on a for-profit host Unfortunately, "non-profit" hosts are somewhat in a short supply.

whack 9y ago

> What we really need is for language-level package managers to support signed immutable packages +1 Emphasis on immutable. Can someone explain why popular package managers like npm don't offer this? During the leftpad fiasco, it boggled my mind that someone could just "liberate" their packages one morning, and transitively break the build for thousands of projects. Immutable packages would have prevented this entire problem.

AgentME 9y ago
NPM fixed that situation by no longer allowing packages to be unpublished after 24 hours. NPM does not allow packages to republish new content under a previously-used version number. NPM by default generates a package-lock.json file (separate from package.json) which pins the versions of all dependencies (including transitive ones) and includes the hashes of the dependencies, which would protect users if npm's servers ever became malicious.
- oneweekwonder 9y ago
  > NPM by default generates a package-lock.json file Thanks for the tidbit, seen it been added May last year to npm5[0]. Hard to keep up if you don't use the tooling activity and only depend on it for other tooling. [0]: http://blog.npmjs.org/post/161081169345/v500 http://blog.npmjs.org/post/161081169345/v500

t0mk 9y ago

I thought the same with the exaggerating. The author of the article can hardly blame github. But he makes a good point, I never realized how much I rely on github username policy in Go code :). If golang allows to import from Git URLs over HTTP or SSH, then it is the same problem with DNS as with the github usernames, isn't it? If the domain name, from which you import, will disappear, then someone else can register it and serve malicious code. This is also pretty hyperbolic. IMO It's simply solvable by refering to a commit hash, like govendor does it (or the gemfile in ruby for that matter). I am actually not sure why it's not possible to add commit hash to a golang import, I thought that $GOPATH/src contains only git repos (but I'm not sure about this).

donatj 9y ago
> It's simply solvable by refering to a commit hash Do you never update your dependencies?
- bmon 9y ago
  If you don't pin your deps to a commit, what's the difference between the author deleting their account (and being replaced), and the author merging an evil commit?
  - macrael 9y ago
    There is no difference technically, but the point is that as is the author has built up a reputation and trust, and that someone taking their name inherits the trust without the reputation.
    - avar 9y ago
      It's still a bad idea not to pin your dependencies even if you trust the author. Say you want to check out some older version of the code for bisecting, and it doesn't even build anymore because it worked with some version of the dependencies that was the latest years ago, good luck figuring out what commit they were all on at the time. It's trivial to just update your own project to point to the latest upstream SHA-1s and commit that, this is why git's own facility to do this (submodules) pins you at specific upstream commits.

Zamicol 9y ago

I really feel like git having a secure package identifier built in is the right answer. I haven't heard any news of git moving to SHA3-256. Why shouldn't this be built in? Here's an old thread from 2017-03-04: https://public-inbox.org/git/CA+dhYEViN4-boZLN+5QJyE7RtX+q6a92p0C2O6TA53==BZfTrQ@mail.gmail.com/T/ https://public-inbox.org/git/CA+dhYEViN4-boZLN+5QJyE7RtX+q6a...

craftyguy 9y ago
> I really feel like git having a secure package identifier built in is the right answer. git commits can be gpg signed: https://git-scm.com/book/id/v2/Git-Tools-Signing-Your-Work https://git-scm.com/book/id/v2/Git-Tools-Signing-Your-Work
- Zamicol 9y ago
  I would never use that. It uses SHA1. The problem is the hash, not the signing. A secure package identifier doesn't necessarily need to be signed. Conversely, you must have a decent hash algorithm. Your mentioned method signs the git commit hash. A signature is only as good as the hash you use, and SHA1 is not suitable. https://blog.thoughtram.io/git/2014/11/18/the-anatomy-of-a-git-commit.html https://blog.thoughtram.io/git/2014/11/18/the-anatomy-of-a-g... Also see https://github.com/git/git/commit/721cc4314cb593e799213ad5f926a1e9fc5779b0 https://github.com/git/git/commit/721cc4314cb593e799213ad5f9...
  - Shoothe 9y ago
    What do you mean by "secure package identifier"? Do you mean commit hash using strong hashing function or something else?

beached_whale 9y ago

It is git, so everything is signed. Just not authenticated. If you don't use head but a release or some tag you should be fine. Also, github supports permanent URL's