5 ms·

A couple of years ago I wrote a Go library that wraps the xz C code and allows you to do xz compression in Go: https://github.com/jamespfennell/xz https://githu

A couple of years ago I wrote a Go library that wraps the xz C code and allows you to do xz compression in Go: https://github.com/jamespfennell/xz https://github.com/jamespfennell/xz About a week ago I received the first PR on that repo, to upgrade to 5.6.1. I thought it was odd to get such a random PR...it's not the same GitHub account as upstream though.

pinko 3y ago
> it's not the same GitHub account as upstream This is valuable information, and a sign that this may be the tip of an iceberg.
Bromeo 3y ago
I don't want to read too much into it, but the person (supposedly) submitting the PR seems to work at 1Password since December last year, as per his Linkedin. (And his Linkedin page has a link to the Github profile that made the PR).
- returningfory2 3y ago
  Yeah the GitHub account looks really really legitimate. Maybe it was compromised though?
  - jethro_tell 3y ago
    What looks legit about a gmail address and some stock art for a profile?
    - gpm 3y ago
      [Deleted per below]
      - TeMPOraL 3y ago
        Can you stay in that org after leaving Google?
        bananapub 3y ago
        whoever is in charge of removing people from the Google github org has the itchiest trigger finger in the whole exiting-the-company process tree.
        fooker 3y ago
        No
      - Jyaif 3y ago
        You are not looking at the right profile. This is the profile that people are talking about: https://github.com/jaredallard https://github.com/jaredallard
        gpm 3y ago
        Oops, you're absolutely correct. Deleted (via edit) my comment above. Thanks.
      - ncr100 3y ago
        He was just (50 minutes ago) removed from the oss fuzz repo. I hope this also (at least temporarily until verification of 'bad/good') remove him from the org?
    - buildbot 3y ago
      Plus the README.md that is just a rickroll
  - ncr100 3y ago
    The 2 GMail accounts are 85% / mainly associated with XZ work, since 2021, per searching for them explicitly via Google.
  - computerfriend 3y ago
    The PR's two commits are signed by a key that was also used to sign previous commits belonging to that author.
    - dralley 3y ago
      Hold up, are you saying that https://github.com/jaredallard https://github.com/jaredallard and the accounts affiliated with this XZ backdoor share a PGP key? Or something else?
      - computerfriend 3y ago
        No, this account made a PR and their commits were signed [1]. Take a look at their other repositories, e.g. they did AoC 2023 in Rust and published it, the commits in that repository are signed by the same key. So this is not (just) a GitHub account compromise. I find this aspect to be an outlier, the other attacker accounts were cutouts. So this doesn't quite make sense to me. [1] https://github.com/jamespfennell/xz/pull/2/commits https://github.com/jamespfennell/xz/pull/2/commits
- bombcar 3y ago
  If I were trying to compromise supply chains, getting into someplace like 1Password would be high up on the list. Poor guy, he's probably going to get the third degree now.
- switch007 3y ago
  As a 1Password user, I just got rather nervous.
  - bombcar 3y ago
    Yubikeys starting to look kinda yummy.
    - wiml 3y ago
      Hardware gets backdoored too, remember Crypto AG?
- lelandbatey 3y ago
  They're definitely a real person. I know cause that "1Password employee since December" is a person I know IRL and worked with for years at their prior employer. They're not a no-name person or a fake identity just FYI. Please don't be witch hunting; this genuinely looks like an unfortunate case where Jared was merely proactively doing their job by trying to get an externally maintained golang bindings of XZ to the latest version of XZ. Jared's pretty fantastic to work with and is definitely the type of person to be filing PRs on external tools to get them to update dependencies. I think the timing is comically bad, but I can vouch for Jared. https://github.com/jamespfennell/xz/pull/2 https://github.com/jamespfennell/xz/pull/2
  - greatjack613 3y ago
    [flagged]
    - lelandbatey 3y ago
      Here's a PR on an employer-owned public Github repository where I made a change and Jared approved it. Please, let's not witch hunt. https://github.com/getoutreach/stencil-golang/pull/404 https://github.com/getoutreach/stencil-golang/pull/404
deleted 3y ago
[deleted]
arp242 3y ago
As a bit of an aside, I would never accept a PR like this, and would always update $large_vendored_dependency myself. This is unreviewable, and trivial to insert any backdoor (unless you go through the motions of updating it yourself and diffing, at which point the PR becomes superfluous). I'd be wary even from a well-known author unless I knew them personally on some level (real-life or via internet). Not that I wouldn't trust them, but people's machines or accounts can get compromised, people can have psychotic episodes, things like that. At the very least I'd like to have some out-of-band "is this really you?" signal. This is how I once inserted a joke in one of our (private) repos that would randomly send cryptic messages to our chat channel. This was pretty harmless and just a joke (there's some context that made it funny), but it took them years to find it – and that was only because I told them after I quit. That said, looking at the GitHub account I'd be surprised if there's anything nefarious going on here. Probably just someone using your repo, seeing it's outdated, and updating it.
- LVB 3y ago
  The (most?) popular SQLite driver for Go often gets PRs to update the SQLite C amalgamation, which the owner politely declines (and I appreciate him for that stance, and for taking on the maintenance burden it brings). e.g., https://github.com/mattn/go-sqlite3/pull/1042#issuecomment-1125574121 https://github.com/mattn/go-sqlite3/pull/1042#issuecomment-1...
  - astrange 3y ago
    Meanwhile SQLite itself doesn't accept any patches for anything; if you show the author one he will at best rewrite it.
- deleted 3y ago
  [deleted]
- creatonez 3y ago
  In this case, the project is using Git submodules for its vendored dependencies, so you can trivially cryptographically verify that they have vendored the correct dependency just by checking the commit hash. It looks really crazy on Github but in most git clients it will just display the commit hash change.
5kg 3y ago
The backdoor (test binary blob and autoconf) is not part of the pull request.
deleted 3y ago
[deleted]
cbmuser 3y ago
There was also a bug report in Debian which requested updating xz-utils to 5.6.1: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1067708 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1067708
- roflmaostc 3y ago
  That's the same Hans Jansen mentioned here: https://boehs.org/node/everything-i-know-about-the-xz-backdoor https://boehs.org/node/everything-i-know-about-the-xz-backdo...
icambron 3y ago
IMO your prior on this should be that it's most likely just someone innocently updating a dependency.
jaredallard2 3y ago
Hey all, I’m the author of that PR. Just posted to Github with additional context: https://github.com/jamespfennell/xz/pull/2#issuecomment-2027836356 https://github.com/jamespfennell/xz/pull/2#issuecomment-2027...
- blueflow 3y ago
  That sucks to have people write mails to your employer...
  - jaredallard2 3y ago
    To be honest, I probably wouldn't have noticed the comments on the PR if it wasn't for that since my Github notifications are an absolute mess. Thankfully, my employer has been super supportive throughout this :D
- SheinhardtWigCo 3y ago
  I appreciated your detailed update!
- ikekkdcjkfke 3y ago
  The dopamine hits from updating stuff should come to an end, it should be thought of as adding potentially new bugs or exploits, unless the update fixes a CVE. Also Github needs to remove the green colors and checkmarks in PR's to prevent these dopamine traps from overriding any critical thinking
  - btown 3y ago
    Counterpoint: if you wait to keep things up to date until there's a CVE, there's a higher likelihood that things will break doing such a massive upgrade, and this may slow down a very time-sensitive CVE response. Allowing people to feel rewarded for keeping things up to date is not inherently a bad thing. As with all things, the balance point will vary from project to project!
    - 3y ago
deleted 3y ago
[deleted]
squigz 3y ago
Internet detectives at work in this thread!
baxtr 3y ago
Suddenly anything like that becomes super suspicious. I wonder how this will affect the OS community in general.
- account42 2y ago
  More caution might not be a bad thing.

pinko 3y ago

> it's not the same GitHub account as upstream This is valuable information, and a sign that this may be the tip of an iceberg.

Bromeo 3y ago

I don't want to read too much into it, but the person (supposedly) submitting the PR seems to work at 1Password since December last year, as per his Linkedin. (And his Linkedin page has a link to the Github profile that made the PR).

returningfory2 3y ago
Yeah the GitHub account looks really really legitimate. Maybe it was compromised though?
- jethro_tell 3y ago
  What looks legit about a gmail address and some stock art for a profile?
  - gpm 3y ago
    [Deleted per below]
    - TeMPOraL 3y ago
      Can you stay in that org after leaving Google?
      - bananapub 3y ago
        whoever is in charge of removing people from the Google github org has the itchiest trigger finger in the whole exiting-the-company process tree.
      - fooker 3y ago
        No
    - Jyaif 3y ago
      You are not looking at the right profile. This is the profile that people are talking about: https://github.com/jaredallard https://github.com/jaredallard
      - gpm 3y ago
        Oops, you're absolutely correct. Deleted (via edit) my comment above. Thanks.
    - ncr100 3y ago
      He was just (50 minutes ago) removed from the oss fuzz repo. I hope this also (at least temporarily until verification of 'bad/good') remove him from the org?
  - buildbot 3y ago
    Plus the README.md that is just a rickroll
- ncr100 3y ago
  The 2 GMail accounts are 85% / mainly associated with XZ work, since 2021, per searching for them explicitly via Google.
- computerfriend 3y ago
  The PR's two commits are signed by a key that was also used to sign previous commits belonging to that author.
  - dralley 3y ago
    Hold up, are you saying that https://github.com/jaredallard https://github.com/jaredallard and the accounts affiliated with this XZ backdoor share a PGP key? Or something else?
    - computerfriend 3y ago
      No, this account made a PR and their commits were signed [1]. Take a look at their other repositories, e.g. they did AoC 2023 in Rust and published it, the commits in that repository are signed by the same key. So this is not (just) a GitHub account compromise. I find this aspect to be an outlier, the other attacker accounts were cutouts. So this doesn't quite make sense to me. [1] https://github.com/jamespfennell/xz/pull/2/commits https://github.com/jamespfennell/xz/pull/2/commits
bombcar 3y ago
If I were trying to compromise supply chains, getting into someplace like 1Password would be high up on the list. Poor guy, he's probably going to get the third degree now.
switch007 3y ago
As a 1Password user, I just got rather nervous.
- bombcar 3y ago
  Yubikeys starting to look kinda yummy.
  - wiml 3y ago
    Hardware gets backdoored too, remember Crypto AG?
lelandbatey 3y ago
They're definitely a real person. I know cause that "1Password employee since December" is a person I know IRL and worked with for years at their prior employer. They're not a no-name person or a fake identity just FYI. Please don't be witch hunting; this genuinely looks like an unfortunate case where Jared was merely proactively doing their job by trying to get an externally maintained golang bindings of XZ to the latest version of XZ. Jared's pretty fantastic to work with and is definitely the type of person to be filing PRs on external tools to get them to update dependencies. I think the timing is comically bad, but I can vouch for Jared. https://github.com/jamespfennell/xz/pull/2 https://github.com/jamespfennell/xz/pull/2
- greatjack613 3y ago
  [flagged]
  - lelandbatey 3y ago
    Here's a PR on an employer-owned public Github repository where I made a change and Jared approved it. Please, let's not witch hunt. https://github.com/getoutreach/stencil-golang/pull/404 https://github.com/getoutreach/stencil-golang/pull/404

deleted 3y ago

[deleted]

arp242 3y ago

As a bit of an aside, I would never accept a PR like this, and would always update $large_vendored_dependency myself. This is unreviewable, and trivial to insert any backdoor (unless you go through the motions of updating it yourself and diffing, at which point the PR becomes superfluous). I'd be wary even from a well-known author unless I knew them personally on some level (real-life or via internet). Not that I wouldn't trust them, but people's machines or accounts can get compromised, people can have psychotic episodes, things like that. At the very least I'd like to have some out-of-band "is this really you?" signal. This is how I once inserted a joke in one of our (private) repos that would randomly send cryptic messages to our chat channel. This was pretty harmless and just a joke (there's some context that made it funny), but it took them years to find it – and that was only because I told them after I quit. That said, looking at the GitHub account I'd be surprised if there's anything nefarious going on here. Probably just someone using your repo, seeing it's outdated, and updating it.

LVB 3y ago
The (most?) popular SQLite driver for Go often gets PRs to update the SQLite C amalgamation, which the owner politely declines (and I appreciate him for that stance, and for taking on the maintenance burden it brings). e.g., https://github.com/mattn/go-sqlite3/pull/1042#issuecomment-1125574121 https://github.com/mattn/go-sqlite3/pull/1042#issuecomment-1...
- astrange 3y ago
  Meanwhile SQLite itself doesn't accept any patches for anything; if you show the author one he will at best rewrite it.
deleted 3y ago
[deleted]
creatonez 3y ago
In this case, the project is using Git submodules for its vendored dependencies, so you can trivially cryptographically verify that they have vendored the correct dependency just by checking the commit hash. It looks really crazy on Github but in most git clients it will just display the commit hash change.

5kg 3y ago

The backdoor (test binary blob and autoconf) is not part of the pull request.

deleted 3y ago

[deleted]

cbmuser 3y ago

There was also a bug report in Debian which requested updating xz-utils to 5.6.1: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1067708 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1067708

roflmaostc 3y ago
That's the same Hans Jansen mentioned here: https://boehs.org/node/everything-i-know-about-the-xz-backdoor https://boehs.org/node/everything-i-know-about-the-xz-backdo...

icambron 3y ago

IMO your prior on this should be that it's most likely just someone innocently updating a dependency.

jaredallard2 3y ago

Hey all, I’m the author of that PR. Just posted to Github with additional context: https://github.com/jamespfennell/xz/pull/2#issuecomment-2027836356 https://github.com/jamespfennell/xz/pull/2#issuecomment-2027...

blueflow 3y ago
That sucks to have people write mails to your employer...
- jaredallard2 3y ago
  To be honest, I probably wouldn't have noticed the comments on the PR if it wasn't for that since my Github notifications are an absolute mess. Thankfully, my employer has been super supportive throughout this :D
SheinhardtWigCo 3y ago
I appreciated your detailed update!
ikekkdcjkfke 3y ago
The dopamine hits from updating stuff should come to an end, it should be thought of as adding potentially new bugs or exploits, unless the update fixes a CVE. Also Github needs to remove the green colors and checkmarks in PR's to prevent these dopamine traps from overriding any critical thinking
- btown 3y ago
  Counterpoint: if you wait to keep things up to date until there's a CVE, there's a higher likelihood that things will break doing such a massive upgrade, and this may slow down a very time-sensitive CVE response. Allowing people to feel rewarded for keeping things up to date is not inherently a bad thing. As with all things, the balance point will vary from project to project!
  - 3y ago

deleted 3y ago

[deleted]

squigz 3y ago

Internet detectives at work in this thread!

baxtr 3y ago

Suddenly anything like that becomes super suspicious. I wonder how this will affect the OS community in general.

account42 2y ago
More caution might not be a bad thing.