5 ms·

The sophistication here is really interesting. And it all got caught because of a fairly obvious perf regression. It reminds of a quote I heard in one of those

The sophistication here is really interesting. And it all got caught because of a fairly obvious perf regression. It reminds of a quote I heard in one of those "real crime" shows: "There's a million ways to get caught for murder, and if you can think of half of them, you're a genius."

dboreham 3y ago
> And it all got caught because of a fairly obvious perf regression Always possible that was "parallel construction" evidence. Someone at a TLA discovered the attack by some other means, had a quiet Signal chat with a former colleague who works at MS...
- paleotrope 3y ago
  Interesting possibilty but it seems like the "discovery" story is too complex and unbelievable.
- eli 3y ago
  There doesn’t seem to be any evidence to support this whatsoever yet it’s nearly impossible to disprove. Classic conspiracy theory.
  - account42 3y ago
    There also isn't really a reason for some contrived parallel construction here - whoever found the issue could just point to it without explaining how it was detected. They could even do that anonymously. > Classic conspiracy theory. I would not be too quick to shit on "conspiracy theories" however as there are plenty of proven cases of people conspiring against the interests of the public.
- GrantMoyer 3y ago
  It seems like a much more suitable parallel construction story to invent in this instance would be something like "there were valgrind issues reported, but I couldn't reproduce them, so I sanity checked the tarball was the same as the git source. It wasn't."
- rtpg 3y ago
  Wouldn't it have been easier to just have someone drive-by comment on the changes in the source tree in the comment? Like "what's up with this?" Though I guess you end up with some other questions if it's totally anonymous. But I often will do a quick look over commits of things that I upgrade (more for backwards compat questions than anything but)
xorvoid 3y ago
Maybe I’m just being naive or too trusting, but this is sort of what I think when folks are getting worried about other backdoors like this in the wild. Is it that they just got unlucky to get caught, or is this type of attack just too hard to pull off in practice? I’d like to think the later. But, we really don’t know.
- ordu 3y ago
  I feel the same way. It is too much complexity in one place, it couldn't work without hiccups.
- lyu07282 3y ago
  One measure might be that we never really found that many backdoors. Over time there is quite a large accumulation of hackers looking at the most mundane technical details. This may be confirmed by regular vulnerabilities that are found in sometimes many decades old software, since vulnerabilities are much harder to find than backdoors. For example shellshock was 30 year old code, PwnKit 12 and log4j was ~10 ish. So if backdoors were commonplace, we probably would've found more by now. Perhaps that's changing now, the xz backdoor will for sure attract many copycats.
  - sjs382 3y ago
    > Over time there is quite a large accumulation of hackers looking at the most mundane technical details. Are there though? Even if true, there are probably enough places with very few eyes on them.
    - almostnormal 3y ago
      Maybe something could be built to put more eyeballs on things. A kind of online-tool that collects the sources to build some relevant distributions, a web front-end to show a random piece of code (filtered by language, probability to show inreasing by less-recently/frequently/qualified viewed) to a volunteering visitor to review. The reviewer leaves a self assesment about their own skills (feed back into selection probability) and any potential findings. Tool-staff double-checks findings (so that the tool does not create too much noise) and forwards to the original authors (bugs) or elsewhere (backdoors). A bit like wikipedias show random page.
anamax 3y ago
> "There's a million ways to get caught for murder, and if you can think of half of them, you're a genius." Does "think of half" apply to the folks trying to solve murders?
- atomicnumber3 3y ago
  Nah, it applies to the person trying to get away with the murder. People will do really, really intricate jobs of trying to cover up, then slip up because like, they leave a receipt in their car that accidentally breaks their alibi.
  - deleted 3y ago
    [deleted]
  - graemep 3y ago
    A clever murderer will disguise the murder as an accident, suicide or natural death. It will not even show in the stats as unsolved. I got the idea from fiction (specifically Dorothy Sayers), but the number of murders Harold Shipman committed before anyone even noticed makes it plausible that people with relevant expertise (doctors, pharmacists, cops, etc.) could easily get away with murder. If Shipman had stopped after the first 100 or so he would have.
  - hackeraccount 3y ago
    My favorite get away with murder stories are the imperfect frame up type stories. So commit a crime and lay a trail of bread crumbs to a false path that will be picked up by the investigators and then later on easily refuted by yourself - because you did it but not in the way you're accused of.
- ball_of_lint 3y ago
  Even if you can think of 10 relatively uncorrelated reasons, that lets you catch the genius murderer 1-(1/2^10) of the time, which is quite good.
- beautifulfreak 3y ago
  That's from Body Heat, said by Mickey Rourke to William Hurt. "...you got fifty ways you're gonna fuck up. If you think of twenty-five of them, then you're a genius - and you ain't no genius." (But a million sounds closer to the truth.)
rdtsc 3y ago
I can believe it’s because it was a team behind the account. Someone developed the feature and another more careless or less experienced one integrated it. Another one possibly managing sock puppets and interacting in comments and PRs.
- akira2501 3y ago
  I wonder what the web admin control panel for the "fake human" looks like, or if it even rises to that level of sophistication yet.
  - cookiengineer 3y ago
    It's called AIMS (Advanced Impact Media Solutions) and is used by several state-level actors these days, both pro- and contra-NATO. Well, at least that one is the most sophisticated one on the market (as of now) and Team Jorge is probably making shitloads of money with it while not giving a damn about who uses their software in the end.
TheDudeMan 3y ago
Yet most murders go unsolved.
- ssl-3 3y ago
  Then most murderers are geniuses. Or most murder investigations are (by definition) incompetent. Or (more likely): The old idiom quoted above is stupid and useless. (That it presumes that murdering and getting away with it is somehow a noble or esteemed deed should be damning enough.)
  - graphe 3y ago
    Wrong. There’s no money or benefits in solving crimes. It could be done easily in many cases but nobody cares about certain people like gang members. Lots of cases where the murderer tells everyone but nobody cares.
    - ssl-3 3y ago
      Wrong? Which part is wrong? Only 2/3 of these to choices can be wrong. The remaining one must be correct.
      - withinboredom 3y ago
        Technically, all 3 could be wrong and an unknown 4th option could be correct. That seems to be what they are proposing here. In both cases, the premise is unclear so good luck!
        ssl-3 3y ago
        Eh, good call I guess. I didn't see that aspect. The 4th option they may appear to propose suggests that murder investigators don't get paid -- neither in money, nor in benefits. So, to that end: As far as I know, that's not usually the case with government employees, and it is always actionable when it does happen to be the case.
- 3y ago
INTPenis 3y ago
Given enough time and local testing they could have gotten away with it. I'm positive their deadline changed due to @teknoraver's patch in libsystemd.
ufo 3y ago
IIRC, the perf regression only happened if the code was compiled with -fno-omit-frame-pointer, which was not the default. https://mastodon.social/@AndresFreundTec/112187000944648334 https://mastodon.social/@AndresFreundTec/112187000944648334
- mananaysiempre 3y ago
  Fedora and Ubuntu both enable frame pointers / disable -fomit-frame-pointer by default now[1]. That’s quite recent news in comparison to the backdoor’s history, admittedly. [1] https://www.brendangregg.com/blog/2024-03-17/the-return-of-the-frame-pointers.html https://www.brendangregg.com/blog/2024-03-17/the-return-of-t...
account42 3y ago
Yet plenty of murders go unsolved forever. Sometimes the secret ingredient is a big pile of luck I guess.

dboreham 3y ago

> And it all got caught because of a fairly obvious perf regression Always possible that was "parallel construction" evidence. Someone at a TLA discovered the attack by some other means, had a quiet Signal chat with a former colleague who works at MS...

paleotrope 3y ago
Interesting possibilty but it seems like the "discovery" story is too complex and unbelievable.
eli 3y ago
There doesn’t seem to be any evidence to support this whatsoever yet it’s nearly impossible to disprove. Classic conspiracy theory.
- account42 3y ago
  There also isn't really a reason for some contrived parallel construction here - whoever found the issue could just point to it without explaining how it was detected. They could even do that anonymously. > Classic conspiracy theory. I would not be too quick to shit on "conspiracy theories" however as there are plenty of proven cases of people conspiring against the interests of the public.
GrantMoyer 3y ago
It seems like a much more suitable parallel construction story to invent in this instance would be something like "there were valgrind issues reported, but I couldn't reproduce them, so I sanity checked the tarball was the same as the git source. It wasn't."
rtpg 3y ago
Wouldn't it have been easier to just have someone drive-by comment on the changes in the source tree in the comment? Like "what's up with this?" Though I guess you end up with some other questions if it's totally anonymous. But I often will do a quick look over commits of things that I upgrade (more for backwards compat questions than anything but)

xorvoid 3y ago

Maybe I’m just being naive or too trusting, but this is sort of what I think when folks are getting worried about other backdoors like this in the wild. Is it that they just got unlucky to get caught, or is this type of attack just too hard to pull off in practice? I’d like to think the later. But, we really don’t know.

ordu 3y ago
I feel the same way. It is too much complexity in one place, it couldn't work without hiccups.
lyu07282 3y ago
One measure might be that we never really found that many backdoors. Over time there is quite a large accumulation of hackers looking at the most mundane technical details. This may be confirmed by regular vulnerabilities that are found in sometimes many decades old software, since vulnerabilities are much harder to find than backdoors. For example shellshock was 30 year old code, PwnKit 12 and log4j was ~10 ish. So if backdoors were commonplace, we probably would've found more by now. Perhaps that's changing now, the xz backdoor will for sure attract many copycats.
- sjs382 3y ago
  > Over time there is quite a large accumulation of hackers looking at the most mundane technical details. Are there though? Even if true, there are probably enough places with very few eyes on them.
  - almostnormal 3y ago
    Maybe something could be built to put more eyeballs on things. A kind of online-tool that collects the sources to build some relevant distributions, a web front-end to show a random piece of code (filtered by language, probability to show inreasing by less-recently/frequently/qualified viewed) to a volunteering visitor to review. The reviewer leaves a self assesment about their own skills (feed back into selection probability) and any potential findings. Tool-staff double-checks findings (so that the tool does not create too much noise) and forwards to the original authors (bugs) or elsewhere (backdoors). A bit like wikipedias show random page.

anamax 3y ago

> "There's a million ways to get caught for murder, and if you can think of half of them, you're a genius." Does "think of half" apply to the folks trying to solve murders?

atomicnumber3 3y ago
Nah, it applies to the person trying to get away with the murder. People will do really, really intricate jobs of trying to cover up, then slip up because like, they leave a receipt in their car that accidentally breaks their alibi.
- deleted 3y ago
  [deleted]
- graemep 3y ago
  A clever murderer will disguise the murder as an accident, suicide or natural death. It will not even show in the stats as unsolved. I got the idea from fiction (specifically Dorothy Sayers), but the number of murders Harold Shipman committed before anyone even noticed makes it plausible that people with relevant expertise (doctors, pharmacists, cops, etc.) could easily get away with murder. If Shipman had stopped after the first 100 or so he would have.
- hackeraccount 3y ago
  My favorite get away with murder stories are the imperfect frame up type stories. So commit a crime and lay a trail of bread crumbs to a false path that will be picked up by the investigators and then later on easily refuted by yourself - because you did it but not in the way you're accused of.
ball_of_lint 3y ago
Even if you can think of 10 relatively uncorrelated reasons, that lets you catch the genius murderer 1-(1/2^10) of the time, which is quite good.
beautifulfreak 3y ago
That's from Body Heat, said by Mickey Rourke to William Hurt. "...you got fifty ways you're gonna fuck up. If you think of twenty-five of them, then you're a genius - and you ain't no genius." (But a million sounds closer to the truth.)

rdtsc 3y ago

I can believe it’s because it was a team behind the account. Someone developed the feature and another more careless or less experienced one integrated it. Another one possibly managing sock puppets and interacting in comments and PRs.

akira2501 3y ago
I wonder what the web admin control panel for the "fake human" looks like, or if it even rises to that level of sophistication yet.
- cookiengineer 3y ago
  It's called AIMS (Advanced Impact Media Solutions) and is used by several state-level actors these days, both pro- and contra-NATO. Well, at least that one is the most sophisticated one on the market (as of now) and Team Jorge is probably making shitloads of money with it while not giving a damn about who uses their software in the end.

TheDudeMan 3y ago

Yet most murders go unsolved.

ssl-3 3y ago
Then most murderers are geniuses. Or most murder investigations are (by definition) incompetent. Or (more likely): The old idiom quoted above is stupid and useless. (That it presumes that murdering and getting away with it is somehow a noble or esteemed deed should be damning enough.)
- graphe 3y ago
  Wrong. There’s no money or benefits in solving crimes. It could be done easily in many cases but nobody cares about certain people like gang members. Lots of cases where the murderer tells everyone but nobody cares.
  - ssl-3 3y ago
    Wrong? Which part is wrong? Only 2/3 of these to choices can be wrong. The remaining one must be correct.
    - withinboredom 3y ago
      Technically, all 3 could be wrong and an unknown 4th option could be correct. That seems to be what they are proposing here. In both cases, the premise is unclear so good luck!
      - ssl-3 3y ago
        Eh, good call I guess. I didn't see that aspect. The 4th option they may appear to propose suggests that murder investigators don't get paid -- neither in money, nor in benefits. So, to that end: As far as I know, that's not usually the case with government employees, and it is always actionable when it does happen to be the case.
3y ago

INTPenis 3y ago

Given enough time and local testing they could have gotten away with it. I'm positive their deadline changed due to @teknoraver's patch in libsystemd.

ufo 3y ago

IIRC, the perf regression only happened if the code was compiled with -fno-omit-frame-pointer, which was not the default. https://mastodon.social/@AndresFreundTec/112187000944648334 https://mastodon.social/@AndresFreundTec/112187000944648334

mananaysiempre 3y ago
Fedora and Ubuntu both enable frame pointers / disable -fomit-frame-pointer by default now[1]. That’s quite recent news in comparison to the backdoor’s history, admittedly. [1] https://www.brendangregg.com/blog/2024-03-17/the-return-of-the-frame-pointers.html https://www.brendangregg.com/blog/2024-03-17/the-return-of-t...

account42 3y ago

Yet plenty of murders go unsolved forever. Sometimes the secret ingredient is a big pile of luck I guess.