4 ms·

I wonder, once the attacker gained commit permissions, were they able to rewrite and force push existing commits? In that case rolling back to older commits may

I wonder, once the attacker gained commit permissions, were they able to rewrite and force push existing commits? In that case rolling back to older commits may not be a solution. If my speculation is correct then the the exact date on which access was granted must then first be known, after that a trusted backup of the repo from before that date is needed. Ideally Lasse Collin would have a daily backup of the repo. Although perhaps the entire repo may have to be completely audited at this point.

ajross 3y ago
Force pushes tend to be noticed easily. All it takes is for one external developer to try to pull to see the failure. And it's actually hard to do because you need to comb through the tree to update all the tags that point to the old commits. On top of that it obviously breaks any external references to the commit IDs (e.g. in distro or build configurations), all the way up to cryptographic signatures that might have been made on releases. I think it's a pretty reasonable assumption that this didn't happen, though it would be nice to see a testimony to that effect from someone trustworthy (e.g. "I restored a xz checkout from a backup taken before 5.6.0 and all the commit IDs match").
- fl7305 3y ago
  > And it's actually hard to do because you need to comb through the tree to update all the tags that point to the old commits. Isn't this part just a few pages of code, if that? I agree that it will be blindingly obvious for the reasons you list.
- skykooler 3y ago
  Of course, that's only since the attack was caught early. Since it affected the machines that build packages, a next step could have been, say, to add code to Git to make it ignore certain commit IDs or mask them in some way. Maybe even modify Firefox, Chromium etc to not show those changes on github. With a backdoor like this and enough time undetected, you could control basically everything.
sloowm 3y ago
The way the hack works is incredibly sophisticated and has specifically sought out how to get past all normal checks. If messing with some commits would be possible this entire rube goldberg hack would not have been set up.
Denvercoder9 3y ago
There are trusted copies of historic releases from third-party sources (at least Linux distributions, but there's probably other sources as well), it's pretty easy to check whether the tags in the git repository match those. (This can be done as the tarballs are a superset of the files in the git repository, the other way around doesn't work).

ajross 3y ago

Force pushes tend to be noticed easily. All it takes is for one external developer to try to pull to see the failure. And it's actually hard to do because you need to comb through the tree to update all the tags that point to the old commits. On top of that it obviously breaks any external references to the commit IDs (e.g. in distro or build configurations), all the way up to cryptographic signatures that might have been made on releases. I think it's a pretty reasonable assumption that this didn't happen, though it would be nice to see a testimony to that effect from someone trustworthy (e.g. "I restored a xz checkout from a backup taken before 5.6.0 and all the commit IDs match").

fl7305 3y ago
> And it's actually hard to do because you need to comb through the tree to update all the tags that point to the old commits. Isn't this part just a few pages of code, if that? I agree that it will be blindingly obvious for the reasons you list.
skykooler 3y ago
Of course, that's only since the attack was caught early. Since it affected the machines that build packages, a next step could have been, say, to add code to Git to make it ignore certain commit IDs or mask them in some way. Maybe even modify Firefox, Chromium etc to not show those changes on github. With a backdoor like this and enough time undetected, you could control basically everything.

sloowm 3y ago

The way the hack works is incredibly sophisticated and has specifically sought out how to get past all normal checks. If messing with some commits would be possible this entire rube goldberg hack would not have been set up.

Denvercoder9 3y ago

There are trusted copies of historic releases from third-party sources (at least Linux distributions, but there's probably other sources as well), it's pretty easy to check whether the tags in the git repository match those. (This can be done as the tarballs are a superset of the files in the git repository, the other way around doesn't work).