7 ms·

CISA broke into a US federal agency, and no one noticed for a full 5 months

191 points by rntn 2y ago

sybercecurity 2y ago
The only reason they noticed is because they were told, so it really it should have been "..and no one noticed."
jmclnx 2y ago
Nice they are doing their job and glad they exist. But how to fix ? Most US Gov agencies are underfunded, it is either beef up security or provide services. Really a tough choice, and the outlook looks like they may lose even more funding.
- akira2501 2y ago
  Lesson learned: The assessed organization had insufficient controls to prevent and detect malicious activity. Lesson learned: The organization did not effectively or efficiently collect, retain, and analyze logs. Lesson learned: Bureaucratic processes and decentralized teams hindered the organization’s network defenders. Lesson learned: A “known-bad” detection approach hampered detection of alternate TTPs. https://www.cisa.gov/news-events/cybersecurity-advisories/aa24-193a https://www.cisa.gov/news-events/cybersecurity-advisories/aa...
  - doe_eyes 2y ago
    The problem with such "lessons learned" is that they are usually not lessons learned - I bet there were numerous people within the organization who knew about these issues before the test. The actual issue is probably that these people are (a) ineffectual at communicating and prioritizing concerns clearly; or (b) good at communicating, but are not being listened to; or (c) they are listened to, but the organization has no practical means to fix this - no money, unable to recruit talent, etc. Most techies often assume (b), but (a) is at least as common. The last issue - (c) - might be superficially true, although it's usually not correct in a deeper sense: there is plenty of discretionary and wasteful spending in any sufficiently large bureaucracy. Central resource allocation is just a hard problem. Anyway, my point is that the problems that need fixing are almost never just technical. Recommendations such as "implement sufficient controls to detect malicious activity" seldom get to the root cause. They are still useful in temporarily overcoming organizational obstacles, but it usually doesn't last.
- deleted 2y ago
  [deleted]
- Rinzler89 2y ago
  >Most US Gov agencies are underfunded Doesn't the US outspend in terms of dollars almost every single developed country on the planet at absolutely everything, even in per capita statistics, from military, police to education and healthcare? How could it be underfunded?
  - lazide 2y ago
    The first thing you learn in government is that you’re always underfunded. If you aren’t, your budget will go to someone who is.
  - throwway120385 2y ago
    The US is really really big. People way underestimate how huge it is in terms of land area. You could fit Europe inside the US. So the reason Germany or France underspend the US has more to do with the area they have to cover and the number of people they have in their borders.
    - dmix 2y ago
      > even in per capita statistics
      - tossandthrow 2y ago
        The argument is density. Things tend to be cheaper per capita, when density is high. I don't buy it thought. I think the reason why the spend is less in Europe is due to higher salary equality - good people take job in government because the salary is only 50% higher in private sector (for tech, even less for other areas).
        vidarh 2y ago
        There are only a couple of US states less dense than e.g. Norway, and even in those states the vast majority of the population live in small, higher density areas. The areas where the vast majority of Americans actually live are fairly high density. Some cost might come down to density, but not much.
- treflop 2y ago
  Private companies get breached often too (see AT&T). Pretty much everyone gets breached. The only ones I don't think get breached deep are the really big software engineering companies where most of the company are also software engineers... like Google. Software is too complex to be secure without a massive team IMO.
  - lazide 2y ago
    Eh, a massive team makes its own major issues. See the many Google leaks over the years. Assuming software can be secure (and hence not doing proper defense in depth, limiting the types and nature of information processed, etc). is the bigger issue IMO.
  - tjohns 2y ago
    > Software is too complex to be secure without a massive team IMO. We could do better as an industry though. Modern operating system design makes it far too easy to shoot yourself in the foot. Imagine a world where all we all use memory-safe/null-safe/type-safe languages, applications and data are strictly sandboxed, access to data is only granted using capabilities-based security, application-level security patches are automatically applied by the OS, data was always encrypted while at rest and while in transit, and passwords are completely replaced with passkeys / smartcards (for users) and X.509 certificates (for servers). While this isn't a panacea, it would solve a great number of the most common security vulnerabilities. Each of these pieces exist individually. There's no reason why we can't have all of these things today, other than support for legacy applications and retraining engineers. However, it's nearly impossible to get away from legacy software needs. But if you want low hanging fruit... stop writing C/C++, and get rid of passwords. These are the biggest flaws in the stack.
    - infamouscow 2y ago
      There is no incentive for the industry to do better. As for government penalties being an incentive: If it was popular amongst the voters to hold corporations seriously responsible, you would see politicians campaign on it and win. It's not nearly as popular as virtually anything else based on empirical data. Another way to say this is if 80% of all voters, regardless of party prioritized this as the #1, #2, and #3 issue, politicians would pass laws. It makes the politician look good and solidifies their reelection. Likewise, politicians that vote against those laws would almost certainly not be reelected.
- banish-m4 2y ago
  Standardization by centralized administration, reduction of rogue external and DIY options, concentrating efforts on delivering managed services based on a common and repeatable platform, strict change control, clear training enumerating what is and what is not allowed, and a security team that actually digs in to make cross-cutting concerns more audited automatically, practical recommendations, least privilege continuously ratcheted down, and advise on new features and new projects.
nvr219 2y ago
[flagged]
- pjmlp 2y ago
  One of the best UNIXes, and one of the few OSes with hardware memory tagging in production.
- deleted 2y ago
  [deleted]
breck 2y ago
I loved security training at Microsoft. I remember one time Satya said the red teams reported to him which Microsoft services they were currently in. He would then ask the heads of those services if they had detected any breaches. Sometimes there would be services that had been breached for years, undetected. Must have been hard for Satya to keep a straight face. One phrase that struck with me from their security training: "Assume Breach".
- selimthegrim 2y ago
  This sounds like China Miéville.
- Thorrez 2y ago
  The red team would beach them then wait years before telling them? The goal the the red team should be to increase security. Waiting years before telling them means there's a years-long delay before security can be improved. That goes against the goal.
- Aaronstotle 2y ago
  Maybe they should have warned them after some time, because Microsoft has gotten breached a few times in the last few years.
- renewiltord 2y ago
  > Be me: security researcher on Death Star > hired to find flaws in space station > find that exhaust port has flaw > single blast to it would lead to reactor > would blow whole station > tell my boss > he asks engineers if there's flaw > they say no > he snickers behind their back > "we know better, don't we anon?" > doesn't tell engineers > mfw blown up by Luke Skywalker
  - swozey 2y ago
    > fired for poisoned chalice position and it all blamed on you
  - swyx 2y ago
    i liked that Solo told the backstory to this that made sense. iirc it was an intentional vuln built in as a protest by the architect who was forced to work on the death star
asynchronous 2y ago
I really really want to root for CISA, but just a few months ago they leaked a trove of critical infrastructure documents that they had collected from partners, that if they hadn’t collected wouldn’t be in the wrong hands currently.
academia_hack 2y ago
Until the US federal government pays civilian tech talent competitively, this is always going to be an issue. Your typical hands-on-keyboard blue team engineer in federal government is a GS-12 getting paid around $68,000 per year (or $99k in very high cost of living areas like DC). They have expensive health benefits, 13 days of PTO a year, put a huge chunk of their paycheck (almost 5%) into a mandatory pension plan that consistently underperforms the market, and can literally go to jail for making mistakes at work depending on the statutory context they work in. The best people in these jobs burn out fast and quit or they end up having to abandon IC work for GS-14/15 jobs (max pay is around $190 for those) in order to keep up with cost-of-living and justify their careers. As a result, you have almost zero genuinely capable principal/senior engineers in government who have the authority to architect complex IT systems for security. Instead you get contractors who charge the taxpayers enormous overhead costs and cut corners wherever possible. If there's one letter to write your congress person to improve government - my vote would be for civil service reform to attract and retain actual top tech talent. They've done it for doctors and lawyers (both of whom can get paid well above the $190k GS pay ceiling), but engineering is still not treated as a comparably skilled professional trade.
- 2OEH8eoCRo0 2y ago
  Yes, because civilian tech companies are never hacked. I do largely agree with your post but I'm also suspicious that stratospheric civilian tech compensation is a bubble.
  - academia_hack 2y ago
    Totally. I think comp is a necessary but not sufficient precondition for fixing government technology. The actual solutions (good authentication and least privilege systems, robust monitoring, rapid intrusion detection and response, secure by default system architectures) all take talented people to execute and the government doesn't have enough of those in-house. Instead most systems are built with a 7-figure contract to Booz Allen and friends and then maintenance and sustainment is left as an exercise to the reader.
  - ImPostingOnHN 2y ago
mrpippy 2y ago
The exploited vulnerability (CVE-2022-21587) is some Oracle E-Business web thing, nothing Solaris-specific like it sounded from the article.
clwg 2y ago
I'm not a huge fan of how red teaming is generally conducted. It's sometimes necessary, but the CISA report seems to indicate that the organization wasn't responding to their requests the way they wanted, leading to a communication breakdown right from the start. The vulnerability was patched and the red team's initial compromise was contained, so they targeted them with phishing, owned their domain controllers, then maintained access for months while pivoting to partner organizations, then published a public report. It's hard to establish constructive dialogue after that, allot of bad feelings and burnt bridges - and sometimes HR. It's tough because there's generally allot of dynamics at play, but I'm sure the impact of this testing was felt by people within the targeted org.
- Jtsummers 2y ago
  > The vulnerability was patched and the red team's initial compromise was contained, so they targeted them with phishing, owned their domain controllers, then maintained access for months while pivoting to partner organizations, then published a public report. You're missing a few steps between the pivot to partner organizations and the report. The public report was made on 11 July 2024. They revealed the breach to the target last year, June 2023, and then began a collaboration effort at that point, running through September 2023. They also don't name and shame in this public report, we don't know what the target organization was.
  - clwg 2y ago
    My next sentence is really a comment on that step. I generally find that working incrementally alongside the teams provides better outcomes. This is not to say you can't use black-box testing or perform unscoped testing, but collaborating throughout the process gives you additional visibility that can save a lot of time, especially in large-scale and diverse environments. People generally respond in a more positive and collaborative manner as well.
treflop 2y ago
I don’t see how this is that newsworthy. There are many federal agencies. One of them will fuck up. Same with private companies. If you have 100 people doing the same thing, at least one of them is going to fuck it up.
cafard 2y ago
Long ago I worked on a government contract at a civil agency, which ran WordPerfect Office on DG minis. The main contractor won a contract with another division in that agency, setting up a slightly spiffier version. Somebody at the COTR's office at the other division encouraged or perhaps dared us to break in. It took about two hours. We let them know at once, but I think that with a bit of discretion we could have maintained our presence for a long time.
snickerbockers 2y ago
Who on earth is still using solaris in this 2024th year of our Lord Jesus Christ!?!?!? I legitimately thought it had been EOL'd at least half a decade ago, guess I was wrong about that.
- harisec 2y ago
  The article is wrong, CVE-2022-21587 is related to Oracle not Solaris
  - vanchor3 2y ago
    Unfortunately it is both... the CVE is for Oracle software but they're running it on Solaris servers.
shalmanese 2y ago
"It said real adversaries may have instead used prolonged password-praying attacks rather than phishing at this stage" Did the author mistype Password-Spraying or is there a seperate type of attack known as Password-Praying? Googling doesn't reveal any other hits on this term.
BrandoElFollito 2y ago
I work with a security officer who joined our company after a career in the DoD and other similar places, always in computer security. He is so incompetent that I initially thought that we have communication problems (I am French). But no - he simply has absolutely no idea about cybersecurity and the teams he "oversees" from that perspective are losing their minds. I had no idea that you could work for years in such sensitive US environments and have completely no knowledge. He is good at saying generalities, though, with complicated words.
dante4151 2y ago
[flagged]

191 points by rntn 2y ago

The only reason they noticed is because they were told, so it really it should have been "..and no one noticed."

Nice they are doing their job and glad they exist. But how to fix ? Most US Gov agencies are underfunded, it is either beef up security or provide services. Really a tough choice, and the outlook looks like they may lose even more funding.

akira2501 2y ago
Lesson learned: The assessed organization had insufficient controls to prevent and detect malicious activity. Lesson learned: The organization did not effectively or efficiently collect, retain, and analyze logs. Lesson learned: Bureaucratic processes and decentralized teams hindered the organization’s network defenders. Lesson learned: A “known-bad” detection approach hampered detection of alternate TTPs. https://www.cisa.gov/news-events/cybersecurity-advisories/aa24-193a https://www.cisa.gov/news-events/cybersecurity-advisories/aa...
- doe_eyes 2y ago
  The problem with such "lessons learned" is that they are usually not lessons learned - I bet there were numerous people within the organization who knew about these issues before the test. The actual issue is probably that these people are (a) ineffectual at communicating and prioritizing concerns clearly; or (b) good at communicating, but are not being listened to; or (c) they are listened to, but the organization has no practical means to fix this - no money, unable to recruit talent, etc. Most techies often assume (b), but (a) is at least as common. The last issue - (c) - might be superficially true, although it's usually not correct in a deeper sense: there is plenty of discretionary and wasteful spending in any sufficiently large bureaucracy. Central resource allocation is just a hard problem. Anyway, my point is that the problems that need fixing are almost never just technical. Recommendations such as "implement sufficient controls to detect malicious activity" seldom get to the root cause. They are still useful in temporarily overcoming organizational obstacles, but it usually doesn't last.
deleted 2y ago
[deleted]
Rinzler89 2y ago
>Most US Gov agencies are underfunded Doesn't the US outspend in terms of dollars almost every single developed country on the planet at absolutely everything, even in per capita statistics, from military, police to education and healthcare? How could it be underfunded?
- lazide 2y ago
  The first thing you learn in government is that you’re always underfunded. If you aren’t, your budget will go to someone who is.
- throwway120385 2y ago
  The US is really really big. People way underestimate how huge it is in terms of land area. You could fit Europe inside the US. So the reason Germany or France underspend the US has more to do with the area they have to cover and the number of people they have in their borders.
  - dmix 2y ago
    > even in per capita statistics
    - tossandthrow 2y ago
      The argument is density. Things tend to be cheaper per capita, when density is high. I don't buy it thought. I think the reason why the spend is less in Europe is due to higher salary equality - good people take job in government because the salary is only 50% higher in private sector (for tech, even less for other areas).
      - vidarh 2y ago
        There are only a couple of US states less dense than e.g. Norway, and even in those states the vast majority of the population live in small, higher density areas. The areas where the vast majority of Americans actually live are fairly high density. Some cost might come down to density, but not much.
treflop 2y ago
Private companies get breached often too (see AT&T). Pretty much everyone gets breached. The only ones I don't think get breached deep are the really big software engineering companies where most of the company are also software engineers... like Google. Software is too complex to be secure without a massive team IMO.
- lazide 2y ago
  Eh, a massive team makes its own major issues. See the many Google leaks over the years. Assuming software can be secure (and hence not doing proper defense in depth, limiting the types and nature of information processed, etc). is the bigger issue IMO.
- tjohns 2y ago
  > Software is too complex to be secure without a massive team IMO. We could do better as an industry though. Modern operating system design makes it far too easy to shoot yourself in the foot. Imagine a world where all we all use memory-safe/null-safe/type-safe languages, applications and data are strictly sandboxed, access to data is only granted using capabilities-based security, application-level security patches are automatically applied by the OS, data was always encrypted while at rest and while in transit, and passwords are completely replaced with passkeys / smartcards (for users) and X.509 certificates (for servers). While this isn't a panacea, it would solve a great number of the most common security vulnerabilities. Each of these pieces exist individually. There's no reason why we can't have all of these things today, other than support for legacy applications and retraining engineers. However, it's nearly impossible to get away from legacy software needs. But if you want low hanging fruit... stop writing C/C++, and get rid of passwords. These are the biggest flaws in the stack.
  - infamouscow 2y ago
    There is no incentive for the industry to do better. As for government penalties being an incentive: If it was popular amongst the voters to hold corporations seriously responsible, you would see politicians campaign on it and win. It's not nearly as popular as virtually anything else based on empirical data. Another way to say this is if 80% of all voters, regardless of party prioritized this as the #1, #2, and #3 issue, politicians would pass laws. It makes the politician look good and solidifies their reelection. Likewise, politicians that vote against those laws would almost certainly not be reelected.
banish-m4 2y ago
Standardization by centralized administration, reduction of rogue external and DIY options, concentrating efforts on delivering managed services based on a common and repeatable platform, strict change control, clear training enumerating what is and what is not allowed, and a security team that actually digs in to make cross-cutting concerns more audited automatically, practical recommendations, least privilege continuously ratcheted down, and advise on new features and new projects.

nvr219 2y ago

[flagged]

pjmlp 2y ago
One of the best UNIXes, and one of the few OSes with hardware memory tagging in production.
deleted 2y ago
[deleted]

breck 2y ago

I loved security training at Microsoft. I remember one time Satya said the red teams reported to him which Microsoft services they were currently in. He would then ask the heads of those services if they had detected any breaches. Sometimes there would be services that had been breached for years, undetected. Must have been hard for Satya to keep a straight face. One phrase that struck with me from their security training: "Assume Breach".

selimthegrim 2y ago
This sounds like China Miéville.
Thorrez 2y ago
The red team would beach them then wait years before telling them? The goal the the red team should be to increase security. Waiting years before telling them means there's a years-long delay before security can be improved. That goes against the goal.
Aaronstotle 2y ago
Maybe they should have warned them after some time, because Microsoft has gotten breached a few times in the last few years.
renewiltord 2y ago
> Be me: security researcher on Death Star > hired to find flaws in space station > find that exhaust port has flaw > single blast to it would lead to reactor > would blow whole station > tell my boss > he asks engineers if there's flaw > they say no > he snickers behind their back > "we know better, don't we anon?" > doesn't tell engineers > mfw blown up by Luke Skywalker
- swozey 2y ago
  > fired for poisoned chalice position and it all blamed on you
- swyx 2y ago
  i liked that Solo told the backstory to this that made sense. iirc it was an intentional vuln built in as a protest by the architect who was forced to work on the death star

asynchronous 2y ago

I really really want to root for CISA, but just a few months ago they leaked a trove of critical infrastructure documents that they had collected from partners, that if they hadn’t collected wouldn’t be in the wrong hands currently.

academia_hack 2y ago

Until the US federal government pays civilian tech talent competitively, this is always going to be an issue. Your typical hands-on-keyboard blue team engineer in federal government is a GS-12 getting paid around $68,000 per year (or $99k in very high cost of living areas like DC). They have expensive health benefits, 13 days of PTO a year, put a huge chunk of their paycheck (almost 5%) into a mandatory pension plan that consistently underperforms the market, and can literally go to jail for making mistakes at work depending on the statutory context they work in. The best people in these jobs burn out fast and quit or they end up having to abandon IC work for GS-14/15 jobs (max pay is around $190 for those) in order to keep up with cost-of-living and justify their careers. As a result, you have almost zero genuinely capable principal/senior engineers in government who have the authority to architect complex IT systems for security. Instead you get contractors who charge the taxpayers enormous overhead costs and cut corners wherever possible. If there's one letter to write your congress person to improve government - my vote would be for civil service reform to attract and retain actual top tech talent. They've done it for doctors and lawyers (both of whom can get paid well above the $190k GS pay ceiling), but engineering is still not treated as a comparably skilled professional trade.

2OEH8eoCRo0 2y ago
Yes, because civilian tech companies are never hacked. I do largely agree with your post but I'm also suspicious that stratospheric civilian tech compensation is a bubble.
- academia_hack 2y ago
  Totally. I think comp is a necessary but not sufficient precondition for fixing government technology. The actual solutions (good authentication and least privilege systems, robust monitoring, rapid intrusion detection and response, secure by default system architectures) all take talented people to execute and the government doesn't have enough of those in-house. Instead most systems are built with a 7-figure contract to Booz Allen and friends and then maintenance and sustainment is left as an exercise to the reader.
- ImPostingOnHN 2y ago

mrpippy 2y ago

The exploited vulnerability (CVE-2022-21587) is some Oracle E-Business web thing, nothing Solaris-specific like it sounded from the article.

clwg 2y ago

I'm not a huge fan of how red teaming is generally conducted. It's sometimes necessary, but the CISA report seems to indicate that the organization wasn't responding to their requests the way they wanted, leading to a communication breakdown right from the start. The vulnerability was patched and the red team's initial compromise was contained, so they targeted them with phishing, owned their domain controllers, then maintained access for months while pivoting to partner organizations, then published a public report. It's hard to establish constructive dialogue after that, allot of bad feelings and burnt bridges - and sometimes HR. It's tough because there's generally allot of dynamics at play, but I'm sure the impact of this testing was felt by people within the targeted org.

Jtsummers 2y ago
> The vulnerability was patched and the red team's initial compromise was contained, so they targeted them with phishing, owned their domain controllers, then maintained access for months while pivoting to partner organizations, then published a public report. You're missing a few steps between the pivot to partner organizations and the report. The public report was made on 11 July 2024. They revealed the breach to the target last year, June 2023, and then began a collaboration effort at that point, running through September 2023. They also don't name and shame in this public report, we don't know what the target organization was.
- clwg 2y ago
  My next sentence is really a comment on that step. I generally find that working incrementally alongside the teams provides better outcomes. This is not to say you can't use black-box testing or perform unscoped testing, but collaborating throughout the process gives you additional visibility that can save a lot of time, especially in large-scale and diverse environments. People generally respond in a more positive and collaborative manner as well.

treflop 2y ago

I don’t see how this is that newsworthy. There are many federal agencies. One of them will fuck up. Same with private companies. If you have 100 people doing the same thing, at least one of them is going to fuck it up.

cafard 2y ago

Long ago I worked on a government contract at a civil agency, which ran WordPerfect Office on DG minis. The main contractor won a contract with another division in that agency, setting up a slightly spiffier version. Somebody at the COTR's office at the other division encouraged or perhaps dared us to break in. It took about two hours. We let them know at once, but I think that with a bit of discretion we could have maintained our presence for a long time.

snickerbockers 2y ago

Who on earth is still using solaris in this 2024th year of our Lord Jesus Christ!?!?!? I legitimately thought it had been EOL'd at least half a decade ago, guess I was wrong about that.

harisec 2y ago
The article is wrong, CVE-2022-21587 is related to Oracle not Solaris
- vanchor3 2y ago
  Unfortunately it is both... the CVE is for Oracle software but they're running it on Solaris servers.

shalmanese 2y ago

"It said real adversaries may have instead used prolonged password-praying attacks rather than phishing at this stage" Did the author mistype Password-Spraying or is there a seperate type of attack known as Password-Praying? Googling doesn't reveal any other hits on this term.

BrandoElFollito 2y ago

I work with a security officer who joined our company after a career in the DoD and other similar places, always in computer security. He is so incompetent that I initially thought that we have communication problems (I am French). But no - he simply has absolutely no idea about cybersecurity and the teams he "oversees" from that perspective are losing their minds. I had no idea that you could work for years in such sensitive US environments and have completely no knowledge. He is good at saying generalities, though, with complicated words.

dante4151 2y ago

[flagged]