5 ms·

What's more worrying is whether the access can realistically be revoked. As a general rule, when a security even rises to the level of root access to internal

What's more worrying is whether the access can realistically be revoked. As a general rule, when a security even rises to the level of root access to internal systems, you don't even try to remove them - you just rebuild the affected VMs from scratch because it's the only way to be sure the attacker didn't leave anything behind. For the systems we're talking about, payment processing stuff at Treasury and Social Security and so forth, one wonders if they can even be rebuilt on a reasonable timeframe?

nicce 2y ago
The bigger question is whether there is currently any personnel physically even allowed to do such thing.
MisoRamen 2y ago
How does someone even clean up this mess? One of the DOGE kids may have just cloned every repo and then connect the machine to public internet because they need to fed it to an AI to figure out how things work. We can only assume the worst and that foreign adversaries may already be combing the code line by line. What will happen when PIIs of every individual with dealings with the Treasury gets leaked? Then there is going to be thousands of hours of meetings to review various processes...
- nicce 2y ago
  > One of the DOGE kids may have just cloned every repo and then connect the machine to public internet because they need to fed it to an AI to figure out how things work. May... https://www.washingtonpost.com/nation/2025/02/06/elon-musk-doge-ai-department-education/ https://www.washingtonpost.com/nation/2025/02/06/elon-musk-d...
  - nelsonic 2y ago
    This is _exactly_ what overpriced management consultants would have done … the only question is which AI tools are they feeding the data to?
- SV_BubbleTime 2y ago
  I like how you are 100% convinced that this “a mess” and “we can only assume the worst” and that PII is compromised, etc. What actual facts do you have for anything? I understand why the media is mad, why NGOs, and why liberal politicians are mad, I get why foreign countries are mad. I’m interested in fraud and abuse, regardless of who does it. So if Musk’s team finds it, great, if they get caught committing it and have to deal with that, also great. But right now we know there is something broken. Instead of being mad about that, you are angry about hypotheticals that have not happened. Why is that?
  - dkjaudyeqooe 2y ago
    How does Musk find "fraud and abuse" if he doesn't have access to the whole stack of decision making? It's not up to him to make those decisions, it's up to congress. Musk is just making up bullshit (I'm surprised he didn't say he was rooting out pedophiles) to justify his jihad against the public service.
    - genewitch 2y ago
      Lol congress that just had to have the Judiciary tell them they actually have to do their jobs and not let the agencies run roughshod? OK!
  - WaxProlix 2y ago
    If you read the article, you might get some insight into the comments.
    - an_guy 2y ago
      > might... So you haven't found any insights regarding this?
  - Fnoord 2y ago
    So, say in a few weeks, a massive fraud gets caught. Musk announces it, releases the documents. Who's to say these crooks didn't manipulate it? Same goes for achives of *.gov but those are public. So we can compare hashes independently from each other. With these private repos, we can't. Originals are getting burned. Hopefully there's solid offsite backups untouched.
- jodrellblank 2y ago
  > "What will happen when PIIs of every individual with dealings with the Treasury gets leaked?" We see what happens when big companies leak personal data - almost nothing. Maybe they give you 3 months of 'credit monitoring' or 'identity theft monitoring' service, maybe they write an apology press release. We've seen how Trump presents things, he quite realistically could say either "it was Democrats weak security, we're fixing it" or "hackers must have got it, we'll clean it up" or even "fake news" and then ... do nothing, we never hear about it again and the affected people deal with it as best they can. Why would you expect more than that to happen - a newspaper writes a damning article, lawsuits are filed, the news moves on in 24 hours. We saw how he reacted to COVID, it wasn't a world class good reaction.
  - genewitch 2y ago
    Boy I sure would love to hear how you'd deal with a novel virus in a country as large as the US with as many people in it. How'd China do, since they're an authoritarian regime? When you reply, don't use the word "bleach" or "UV"
    - jodrellblank 2y ago
      I don't want to engage in your bad-faith trolling. Make a substantial comment. Compare Trump's response with other world leaders, other countries, and medical advisors' recommendations, and what you'd hope an ideal leader would do, leave me out of it, I'm not a part of a government's COVID response.
      - genewitch 2y ago
        there's only a couple of coutries that have the area/population demographics of the US. it isn't bad faith. what you're doing is conflating "The US" as just "any other country" when it isn't. we're the third or 4th largest area nation in the world. is it because you wanted to say bleach and UV? edit: "look what UK and denmark did!" is so completely irrelevant that it's ... not "good" faith to suggest it.
bigiain 2y ago
> you just rebuild the affected VMs from scratch These people have administrative access, and at least in some cases network and physical access. Once you determine they are untrustworthy and potentially malicious, you can't just rebuild the VMs, since you can no longer trust the hypervisor or even the hardware. If they were Chinese or Mossad agents, you'd start from scratch in a different DC on supply chain audited new compute, storage, and networking hardware. And you'd compile everything from audited source. And I have NFI how you'd deal with potential malicious changes to your data and backups.
- leptons 2y ago
  > And I have NFI how you'd deal with potential malicious changes to your data and backups. The backups should be stored on WORM tape. They can't be altered (easily or at all?). Of course they're probably wiping their asses with the backups like they are the constitution.
  - modderation 2y ago
    WORM prevents after-the-fact modification, but it isn't very helpful in the case of persistent threats. The concern is that the tampering has already been committed to the backups. When was the "Break Glass" password last rotated? Is it protected by one or more Yubikeys that were manufactured before they fixed that nasty exploit? What other attack vectors are baked in through malfeasance or human error?
    - leptons 2y ago
      My comment was not in reply to passwords, "yubikeys" or anything else you mentioned, so your techsplaining about those things was a bit misplaced. MY point was that if the backups are on WORM tapes, and we still have those backups, then there's nothing to fear being compromised from those backups. Everything other than WORM tapes you wrote about is outside the scope of my comment.
analog31 2y ago
This is going to be like coming home from a vacation and discovering that squatters have been living in your house for a month, going through your stuff. You'll probably end up bulldozing the place and starting over.
A4ET8a8uTh0_v2 2y ago
I will admit I did not think of that aspect of it. I think the reason I didn't is because, supposedly - as it was presented to me at the time, those systems run on some ancient hardware/software. In other words, even if something was left behind, it shouldn't be that hard to locate. If anyone with real experience in that area could chime in. Until now I was under impression COBOL ran it all:P
- testbjjl 2y ago
  Worked in IT a long time ago for a branch. There was some Java, a lot of Perl and SVN. We got releases from DC to run on local servers. Folks with experience with SDLC were prevalent and that was a prerequisite for doing anything meaningful. Never saw Cobol, doesn’t mean it wasn’t there.
_kb 2y ago
You don’t just rebuild the VMs. You burn the DC and start again from scratch. That’s exactly what Cloudflare had to do: https://blog.cloudflare.com/thanksgiving-2023-security-incident/ https://blog.cloudflare.com/thanksgiving-2023-security-incid...
- cozzyd 2y ago
  The double entendre of DC is amusing here
PaulDavisThe1st 2y ago
The VMs? You're imagining that the Federal payment systems run in VMs? This is not some web stack thing. It's not some Linux system shenanigans.

nicce 2y ago

The bigger question is whether there is currently any personnel physically even allowed to do such thing.

MisoRamen 2y ago

How does someone even clean up this mess? One of the DOGE kids may have just cloned every repo and then connect the machine to public internet because they need to fed it to an AI to figure out how things work. We can only assume the worst and that foreign adversaries may already be combing the code line by line. What will happen when PIIs of every individual with dealings with the Treasury gets leaked? Then there is going to be thousands of hours of meetings to review various processes...

nicce 2y ago
> One of the DOGE kids may have just cloned every repo and then connect the machine to public internet because they need to fed it to an AI to figure out how things work. May... https://www.washingtonpost.com/nation/2025/02/06/elon-musk-doge-ai-department-education/ https://www.washingtonpost.com/nation/2025/02/06/elon-musk-d...
- nelsonic 2y ago
  This is _exactly_ what overpriced management consultants would have done … the only question is which AI tools are they feeding the data to?
SV_BubbleTime 2y ago
I like how you are 100% convinced that this “a mess” and “we can only assume the worst” and that PII is compromised, etc. What actual facts do you have for anything? I understand why the media is mad, why NGOs, and why liberal politicians are mad, I get why foreign countries are mad. I’m interested in fraud and abuse, regardless of who does it. So if Musk’s team finds it, great, if they get caught committing it and have to deal with that, also great. But right now we know there is something broken. Instead of being mad about that, you are angry about hypotheticals that have not happened. Why is that?
- dkjaudyeqooe 2y ago
  How does Musk find "fraud and abuse" if he doesn't have access to the whole stack of decision making? It's not up to him to make those decisions, it's up to congress. Musk is just making up bullshit (I'm surprised he didn't say he was rooting out pedophiles) to justify his jihad against the public service.
  - genewitch 2y ago
    Lol congress that just had to have the Judiciary tell them they actually have to do their jobs and not let the agencies run roughshod? OK!
- WaxProlix 2y ago
  If you read the article, you might get some insight into the comments.
  - an_guy 2y ago
    > might... So you haven't found any insights regarding this?
- Fnoord 2y ago
  So, say in a few weeks, a massive fraud gets caught. Musk announces it, releases the documents. Who's to say these crooks didn't manipulate it? Same goes for achives of *.gov but those are public. So we can compare hashes independently from each other. With these private repos, we can't. Originals are getting burned. Hopefully there's solid offsite backups untouched.
jodrellblank 2y ago
> "What will happen when PIIs of every individual with dealings with the Treasury gets leaked?" We see what happens when big companies leak personal data - almost nothing. Maybe they give you 3 months of 'credit monitoring' or 'identity theft monitoring' service, maybe they write an apology press release. We've seen how Trump presents things, he quite realistically could say either "it was Democrats weak security, we're fixing it" or "hackers must have got it, we'll clean it up" or even "fake news" and then ... do nothing, we never hear about it again and the affected people deal with it as best they can. Why would you expect more than that to happen - a newspaper writes a damning article, lawsuits are filed, the news moves on in 24 hours. We saw how he reacted to COVID, it wasn't a world class good reaction.
- genewitch 2y ago
  Boy I sure would love to hear how you'd deal with a novel virus in a country as large as the US with as many people in it. How'd China do, since they're an authoritarian regime? When you reply, don't use the word "bleach" or "UV"
  - jodrellblank 2y ago
    I don't want to engage in your bad-faith trolling. Make a substantial comment. Compare Trump's response with other world leaders, other countries, and medical advisors' recommendations, and what you'd hope an ideal leader would do, leave me out of it, I'm not a part of a government's COVID response.
    - genewitch 2y ago
      there's only a couple of coutries that have the area/population demographics of the US. it isn't bad faith. what you're doing is conflating "The US" as just "any other country" when it isn't. we're the third or 4th largest area nation in the world. is it because you wanted to say bleach and UV? edit: "look what UK and denmark did!" is so completely irrelevant that it's ... not "good" faith to suggest it.

bigiain 2y ago

> you just rebuild the affected VMs from scratch These people have administrative access, and at least in some cases network and physical access. Once you determine they are untrustworthy and potentially malicious, you can't just rebuild the VMs, since you can no longer trust the hypervisor or even the hardware. If they were Chinese or Mossad agents, you'd start from scratch in a different DC on supply chain audited new compute, storage, and networking hardware. And you'd compile everything from audited source. And I have NFI how you'd deal with potential malicious changes to your data and backups.

leptons 2y ago
> And I have NFI how you'd deal with potential malicious changes to your data and backups. The backups should be stored on WORM tape. They can't be altered (easily or at all?). Of course they're probably wiping their asses with the backups like they are the constitution.
- modderation 2y ago
  WORM prevents after-the-fact modification, but it isn't very helpful in the case of persistent threats. The concern is that the tampering has already been committed to the backups. When was the "Break Glass" password last rotated? Is it protected by one or more Yubikeys that were manufactured before they fixed that nasty exploit? What other attack vectors are baked in through malfeasance or human error?
  - leptons 2y ago
    My comment was not in reply to passwords, "yubikeys" or anything else you mentioned, so your techsplaining about those things was a bit misplaced. MY point was that if the backups are on WORM tapes, and we still have those backups, then there's nothing to fear being compromised from those backups. Everything other than WORM tapes you wrote about is outside the scope of my comment.

analog31 2y ago

This is going to be like coming home from a vacation and discovering that squatters have been living in your house for a month, going through your stuff. You'll probably end up bulldozing the place and starting over.

A4ET8a8uTh0_v2 2y ago

I will admit I did not think of that aspect of it. I think the reason I didn't is because, supposedly - as it was presented to me at the time, those systems run on some ancient hardware/software. In other words, even if something was left behind, it shouldn't be that hard to locate. If anyone with real experience in that area could chime in. Until now I was under impression COBOL ran it all:P

testbjjl 2y ago
Worked in IT a long time ago for a branch. There was some Java, a lot of Perl and SVN. We got releases from DC to run on local servers. Folks with experience with SDLC were prevalent and that was a prerequisite for doing anything meaningful. Never saw Cobol, doesn’t mean it wasn’t there.

_kb 2y ago

You don’t just rebuild the VMs. You burn the DC and start again from scratch. That’s exactly what Cloudflare had to do: https://blog.cloudflare.com/thanksgiving-2023-security-incident/ https://blog.cloudflare.com/thanksgiving-2023-security-incid...

cozzyd 2y ago
The double entendre of DC is amusing here

PaulDavisThe1st 2y ago

The VMs? You're imagining that the Federal payment systems run in VMs? This is not some web stack thing. It's not some Linux system shenanigans.