4 ms·

This is IMO one of the coolest tech stories to ever happen, seriously amazing spycraft & hacking skills, but I haven't been keeping up with new developments fro

by coldpie 8mo ago

This is IMO one of the coolest tech stories to ever happen, seriously amazing spycraft & hacking skills, but I haven't been keeping up with new developments from this story since it broke. Last I heard, the best guess at what happened was some state-sponsored actor worked very hard to get this merged, and it was caught luckily at the last minute. But no one had any smoking gun as to who did it or why or who they were targeting. Any new developments since then? Are we still just totally in the dark about what was going on here?

nerevarthelame 8mo ago
Still no smoking gun, but possibly Russia. From the video https://youtu.be/aoag03mSuXQ?t=2883 https://youtu.be/aoag03mSuXQ?t=2883: > A lot of the aliases, like Jia Tan, they sound like Asian names, and the published changes are all timestamped in UTC+8, Beijing time. So the signs point to China. And that's why it's probably not China. I mean, why would they make it that obvious? Every other part of the operation has been so meticulous, so cautious. > And they also worked on Chinese New Year, but not on Christmas. And over the years, there were nine changes that fall outside of the Beijing time into UTC+2, which is a time zone that includes Israel and parts of Western Russia. That's why some experts have speculated that this could be the work of APT29, a Russian-state-backed hacker group also known as Cozy Bear. But again, do we know? No, of course we don't know who it is, and we likely will never know.
- gosub100 8mo ago
  Russians don't celebrate Christmas on the 25th.
  - dijit 8mo ago
    That was also what I took away when watching the video. Russians don't celebrate Christmas on the 25th (they Celebrate on January 7th), but even more than that: Russians don't celebrate Christmas the same way we do in the west. Their "Christmas" family celebrations are on New Years Eve. So if you're drawing conclusions from them not working on the 25th (which is a literal normal day in eastern europe) then signs point elsewhere unfortunately.
    - siddbudd 7mo ago
      ah, Eastern Europe is a homogeneous slop, good to know. The 25th is only celebrated as a national holiday in like 90% of Eastern European countries. Russia and Serbia celebrate it on January 7th instead, and even that is only due to their use of the old Julian calendar. But sure, normal day in a very narrow and obscure definition of Eastern Europe.
      - dijit 7mo ago
        Calm your tits, it’s pretty clear I meant slavic nations, and yes: even the January 7th stuff is not celebrated like we do in the west. I’m trying to convey to people why its materially a red-herring that “russians” wouldn’t work on the 25th: since its a normal day.
- mc32 8mo ago
  Those anecdotes don’t mean anything. If I were China and wanted plausible deniability I would work on CNY and take off on foreign holidays. Of course that leaves Beijing time as a weird oversight though it’s always Beijing time anywhere in China.
  - ranger_danger 8mo ago
    One can also make commits with any arbitrary date/time(zone) values they want that don't reflect reality.
- ginko 8mo ago
  >And that's why it's probably not China. I mean, why would they make it that obvious? That's just what they want you to think!
- lrasinen 8mo ago
  UTC+2 isn't very convincing as an argument for Russia. Only the Kaliningrad exclave uses that timezone, and if I were in a state-backed group, I'd live in one of the big cities. Also quick search suggested UTC+3 was seen during the summer, and Russia doesn't do DST either. Edit: some of the UTC+2/3 times are attributable to being differences in git committer and author dates (e.g. email patches)
  - chatmasta 8mo ago
    I’ve always found this an amusing method of attribution considering top tier hackers are unlikely to be writing code only during office hours.
  - lrasinen 8mo ago
    I couldn't let this be, so I went through the commits and as far as I can tell, that's the case. The committer/author names and timestamps are consistent with using --author on a commit (... or in a few cases, --amend --author). Except one: commit 3d1fdddf9 has Jia Tan as both author and committer but the author timestamp is in +0300 while the commit timestamp is +0800.
leonidasv 8mo ago
Stuxnet is also another mindblowing case. Wired write-up on it is a recommended reading: https://web.archive.org/web/20141028182107/http://www.wired.com/2011/07/how-digital-detectives-deciphered-stuxnet/all/ https://web.archive.org/web/20141028182107/http://www.wired....
tokyobreakfast 8mo ago
> and it was caught luckily at the last minute This isn't correct at all. The changes were merged into xz and made it into testing branches of major Linux distros. It was caught at T plus a few minutes only because a neurotic Microsoft employee performing debugging noticed an obscure performance issue. You can literally say Microsoft saved Linux that day. Imagine thinking this 25 years ago. It's the difference between something really bad which happened, and something really, really, really, really bad: a malicious actor having RCE credentials to every new Debian and Red Hat box on planet Earth.
- ApolloFortyNine 8mo ago
  Redhat actually stumbled on the bug separately with valgrind errors triggering, so it's days were likely numbered regardless. Probably saved them a lot of debugging but the writing was on the wall.
  - tokyobreakfast 8mo ago
    A lot of people fail to fully grasp how bad this could have been on the off chance the authors were slightly less sloppy.
    - ApolloFortyNine 8mo ago
      Imo it just proves there's a 99% chance a standard distro has a current zero day in it. If a state actor (it almost has to be a state actor at the time frame they were operating under) could put in this much effort once, they clearly could afford to do it X times. And when you look through the history of communications from the author, it just reads like 'another day at the office'.
  - dralley 8mo ago
    Red Hat noticed that something was off, but there was a new version published by "Jia Tan" that fixed the warnings and the performance issue, so it's not really clear that the original version would have still gotten as deep of an investigation as would have been needed to find the issue. It's possible though. The noise around it did at least put Freund on alert and we should be very glad both that "Jia Tan" made the mistakes they made originally and that Freund followed up on their gut feeling

by coldpie 8mo ago

nerevarthelame 8mo ago

Still no smoking gun, but possibly Russia. From the video https://youtu.be/aoag03mSuXQ?t=2883 https://youtu.be/aoag03mSuXQ?t=2883: > A lot of the aliases, like Jia Tan, they sound like Asian names, and the published changes are all timestamped in UTC+8, Beijing time. So the signs point to China. And that's why it's probably not China. I mean, why would they make it that obvious? Every other part of the operation has been so meticulous, so cautious. > And they also worked on Chinese New Year, but not on Christmas. And over the years, there were nine changes that fall outside of the Beijing time into UTC+2, which is a time zone that includes Israel and parts of Western Russia. That's why some experts have speculated that this could be the work of APT29, a Russian-state-backed hacker group also known as Cozy Bear. But again, do we know? No, of course we don't know who it is, and we likely will never know.

gosub100 8mo ago
Russians don't celebrate Christmas on the 25th.
- dijit 8mo ago
  That was also what I took away when watching the video. Russians don't celebrate Christmas on the 25th (they Celebrate on January 7th), but even more than that: Russians don't celebrate Christmas the same way we do in the west. Their "Christmas" family celebrations are on New Years Eve. So if you're drawing conclusions from them not working on the 25th (which is a literal normal day in eastern europe) then signs point elsewhere unfortunately.
  - siddbudd 7mo ago
    ah, Eastern Europe is a homogeneous slop, good to know. The 25th is only celebrated as a national holiday in like 90% of Eastern European countries. Russia and Serbia celebrate it on January 7th instead, and even that is only due to their use of the old Julian calendar. But sure, normal day in a very narrow and obscure definition of Eastern Europe.
    - dijit 7mo ago
      Calm your tits, it’s pretty clear I meant slavic nations, and yes: even the January 7th stuff is not celebrated like we do in the west. I’m trying to convey to people why its materially a red-herring that “russians” wouldn’t work on the 25th: since its a normal day.
mc32 8mo ago
Those anecdotes don’t mean anything. If I were China and wanted plausible deniability I would work on CNY and take off on foreign holidays. Of course that leaves Beijing time as a weird oversight though it’s always Beijing time anywhere in China.
- ranger_danger 8mo ago
  One can also make commits with any arbitrary date/time(zone) values they want that don't reflect reality.
ginko 8mo ago
>And that's why it's probably not China. I mean, why would they make it that obvious? That's just what they want you to think!
lrasinen 8mo ago
UTC+2 isn't very convincing as an argument for Russia. Only the Kaliningrad exclave uses that timezone, and if I were in a state-backed group, I'd live in one of the big cities. Also quick search suggested UTC+3 was seen during the summer, and Russia doesn't do DST either. Edit: some of the UTC+2/3 times are attributable to being differences in git committer and author dates (e.g. email patches)
- chatmasta 8mo ago
  I’ve always found this an amusing method of attribution considering top tier hackers are unlikely to be writing code only during office hours.
- lrasinen 8mo ago
  I couldn't let this be, so I went through the commits and as far as I can tell, that's the case. The committer/author names and timestamps are consistent with using --author on a commit (... or in a few cases, --amend --author). Except one: commit 3d1fdddf9 has Jia Tan as both author and committer but the author timestamp is in +0300 while the commit timestamp is +0800.

leonidasv 8mo ago

Stuxnet is also another mindblowing case. Wired write-up on it is a recommended reading: https://web.archive.org/web/20141028182107/http://www.wired.com/2011/07/how-digital-detectives-deciphered-stuxnet/all/ https://web.archive.org/web/20141028182107/http://www.wired....

tokyobreakfast 8mo ago

> and it was caught luckily at the last minute This isn't correct at all. The changes were merged into xz and made it into testing branches of major Linux distros. It was caught at T plus a few minutes only because a neurotic Microsoft employee performing debugging noticed an obscure performance issue. You can literally say Microsoft saved Linux that day. Imagine thinking this 25 years ago. It's the difference between something really bad which happened, and something really, really, really, really bad: a malicious actor having RCE credentials to every new Debian and Red Hat box on planet Earth.

ApolloFortyNine 8mo ago
Redhat actually stumbled on the bug separately with valgrind errors triggering, so it's days were likely numbered regardless. Probably saved them a lot of debugging but the writing was on the wall.
- tokyobreakfast 8mo ago
  A lot of people fail to fully grasp how bad this could have been on the off chance the authors were slightly less sloppy.
  - ApolloFortyNine 8mo ago
    Imo it just proves there's a 99% chance a standard distro has a current zero day in it. If a state actor (it almost has to be a state actor at the time frame they were operating under) could put in this much effort once, they clearly could afford to do it X times. And when you look through the history of communications from the author, it just reads like 'another day at the office'.
- dralley 8mo ago
  Red Hat noticed that something was off, but there was a new version published by "Jia Tan" that fixed the warnings and the performance issue, so it's not really clear that the original version would have still gotten as deep of an investigation as would have been needed to find the issue. It's possible though. The noise around it did at least put Freund on alert and we should be very glad both that "Jia Tan" made the mistakes they made originally and that Freund followed up on their gut feeling