12 ms·

SolarWinds hasn't bothered to revoke their certs or remove the package https://twitter.com/KyleHanslovan/status/1338360093767823362 https://twitter.com/KyleHan

SolarWinds hasn't bothered to revoke their certs or remove the package https://twitter.com/KyleHanslovan/status/1338360093767823362 https://twitter.com/KyleHanslovan/status/1338360093767823362 Back in 2019 apparently their FTP server credentials were exposed on GitHub, allowing automated updates being pushed https://twitter.com/vinodsparrow/status/1338431183588188160/photo/1 https://twitter.com/vinodsparrow/status/1338431183588188160/... Edit: If updates failed due to signature not matching, SolarWinds recommended downloading the package and installing it manually, LOL https://twitter.com/KyleHanslovan/status/1338419999665508354/photo/1 https://twitter.com/KyleHanslovan/status/1338419999665508354...

Nightshaxx 6y ago
LOL that last one. Why bother having the checksum at all in this case....
Merman_Mike 6y ago
Am I understanding the last one correctly? 1. Customers complain that they can't install latest version because it's checksum doesn't match what SolarWinds posted 2. The checksum doesn't match because malware has been inserted into the package during build/delivery 3. SolarWinds tells customers to ignore this and install it manually Did no one think to check why the checksum didn't match?
- jessaustin 6y ago
  One suspects they've given this advice for a long time... because their shit has been hacked for a long time.
  - gitweb 6y ago
    I don't understand why anyone would pay for SW in the first place. It has been garbage software for a long time. If government clients are paying for this and installing in on their servers, we have bigger worries.
- deleted 6y ago
  [deleted]
- RobRivera 6y ago
  Solarwinds is def. Used by acrive duty cyber units at Lackland afb...and they wonder why we tell them they can't just install what they feel like.
  - dabockster 6y ago
    And you posted this US military vulnerability on a publicly searchable internet site? head desk
    - rootsudo 6y ago
      TO be fair, it isn't really secret, if you look at any job posting for lackland, you'll see it mentioned over and over.. https://careers-salientcrgt.icims.com/jobs/11200/network-systems-engineer-2/job+&cd=4&hl=en&ct=clnk&gl=us https://careers-salientcrgt.icims.com/jobs/11200/network-sys...' https://i.imgur.com/d8KbSZp.png https://i.imgur.com/d8KbSZp.png But, wow, imagine that's a job, just walk in, look at two programs and swap out parts as needed.
      - Aperocky 6y ago
        The qualifications reads 'Someone from HR came up with this'
        astrea 6y ago
        Isn't that true of most postings?
    - RobRivera 6y ago
      Facebook query Find people who work for US Air Force. Vulnerabilities publicly available are numerous, and I gave no such details to anyone that would give them an easier time finding said compromises. Its like saying windows 10 bug found --> HEY THE MILITARY USES WINDOWS 10.
    - willis936 6y ago
      My employer has a knowledgebase on the public internet that is littered with lists of softwares and practices. There are thousands of employees. Name dropping software should be a risky thing to do, but that isn’t the world we live in.
- eli 6y ago
  #2 is speculation. Seems possible that there's an unrelated bug causing checksum errors. In any event, it's not a good look right now.
  - wyldfire 6y ago
    Regardless of the motivation, cause, mechanism of #2 - #3 is not the appropriate way to handle the problem. Attack is indistinguishable from unintentional corruption. And #3 trains customers to do the wrong thing when they encounter an attack.
    - eli 6y ago
      The malicious file was signed with the right certificate. So yeah you should ideally be more careful with checksums but there already was a much more robust and secure authentication mechanism and it was defeated.
      - heleninboodler 6y ago
        Yes, these are two orthogonal egregious security problems.
- rayvd 6y ago
  This seems an unfair leap. The most common cause of a checksum mis-match is going to be a partial download or something similar. It's also not relevant to the current attack since the code was legitimately included in the official release and, as such, baked into the valid checksum results.
  - deleted 6y ago
    [deleted]
  - tstrimple 6y ago
    Is the proper response to tell a customer to install the package anyway because it's just a partial download or something similar? Regardless, it seems irresponsible.
  - deleted 6y ago
    [deleted]
35fbe7d3d5b9 6y ago
> SolarWinds hasn't bothered to revoke their certs or remove the package Amazing. While I'm sure the attackers have already shut up shop and the threat no longer exists, this feels insanely tone-deaf from SolarWinds.
- bluedino 6y ago
  Maybe they were just bribed?
  - 35fbe7d3d5b9 6y ago
    An employee, possibly. The whole company, unlikely. And either way, even if someone was bribed to introduce the attack there's zero reason to allow the hacked software to be downloaded now. I work at a large and highly regulated (HIPAA) company and we have the equivalent of Electric Dylan/Pete Seeger with the axe: if someone at the VP+ level declares a major incident, our infosec team has a script that will lock down all inbound/outbound traffic, snapshot all our running machines for later forensics, lock our AWS IAM access down to a single incident response account, and move DNS for our web properties to a "we've been hacked" page. (OK, it obviously doesn't say that, but something similar that has been heavily vetted by legal and marketing ;-)). We've drilled and timed it out and can stop the ship in ~5 minutes. Either SolarWinds doesn't have a major security incident response plan, or they don't have the stomach to pull the trigger. Neither is promising.
    - strogonoff 6y ago
      Sounds like a solid information security incident response mechanism! The only missing piece is making sure that VP+ level folks are not incentivized in any way to suppress incidents. However, that’s beyond infosec—in that treacherous area between information security, shareholder interests and organizational politics. I wish business continuity planning (which would include infosec procedures but has a much wider overall scope) was paid more attention and more widely scrutinized.
      - 6y ago
coldcode 6y ago
Clearly whoever is the CIO/CISO could care less? I find it hilarious that people get these positions without seemingly a care in the world. Or maybe they do care and the CEO didn't? Hardly anyone ever gets fired in these circumstances.
- ilikeerp 6y ago
  Surely he COULDN'T care less?
  - bobthepanda 6y ago
    This is one of those colloquialisms that has become so commonplace that it actually becomes a part of the language rather than being incorrect. https://www.oxfordlearnersdictionaries.com/us/definition/english/care_2?q=could+care+less https://www.oxfordlearnersdictionaries.com/us/definition/eng... lists it as a usage in North American English.
    - xbar 6y ago
      Those of us who know better bristle and retain pedantry. Consider semi-annual and biannual. Grr.
    - ilikeerp 6y ago
      Yes, when enough people do / say something wrong over and over again, it sometimes becomes normalised.
      - bobthepanda 6y ago
        Wrong would imply a right, but most interpretations of what is "right" are fairly arbitrary and mostly have to do with social and cultural hierarchy.
        ilikeerp 6y ago
        Logically it's wrong. If you COULD care less then that means you do care to some level whereas most people mean you COULD NOT care less. Simple...
swiley 6y ago
I guess if you can be as successful as SolarWinds with that level of incompetence I should stop worrying so much about myself.
- spondyl 6y ago
  You'd be surprised honestly
- Aperocky 6y ago
  You'll be surprised at how technically illiterate most corporations are and how marketing and not engineering are responsible for the success for some of the software companies.
1vuio0pswjnm7 6y ago
Is it possible that there could be SolarWinds customers who are not vulnerable because, for whatever reason, they did not enable/install updates. Were updates to the Orion software necessary for the original software to continue to function or were they optional.
- zimpenfish 6y ago
  They've said that 33k customers were potentially exposed but only 18k actually downloaded that update. https://www.zdnet.com/article/sec-filings-solarwinds-says-18000-customers-are-impacted-by-recent-hack/ https://www.zdnet.com/article/sec-filings-solarwinds-says-18...
- jdironman 6y ago
  They were optional. I know a company still using a version from 2015/2016.
oldkn 6y ago
To be fair, the number of times this type of error occurs due to a state-backed actor who's quietly hijacked the build system with an undetectable backdoor is low compared to the number of customers who see the error for some other reasons (bad download for example) - so it's not entirely surprising they'd give this advice.

Nightshaxx 6y ago

LOL that last one. Why bother having the checksum at all in this case....

Merman_Mike 6y ago

Am I understanding the last one correctly? 1. Customers complain that they can't install latest version because it's checksum doesn't match what SolarWinds posted 2. The checksum doesn't match because malware has been inserted into the package during build/delivery 3. SolarWinds tells customers to ignore this and install it manually Did no one think to check why the checksum didn't match?

jessaustin 6y ago
One suspects they've given this advice for a long time... because their shit has been hacked for a long time.
- gitweb 6y ago
  I don't understand why anyone would pay for SW in the first place. It has been garbage software for a long time. If government clients are paying for this and installing in on their servers, we have bigger worries.
deleted 6y ago
[deleted]
RobRivera 6y ago
Solarwinds is def. Used by acrive duty cyber units at Lackland afb...and they wonder why we tell them they can't just install what they feel like.
- dabockster 6y ago
  And you posted this US military vulnerability on a publicly searchable internet site? head desk
  - rootsudo 6y ago
    TO be fair, it isn't really secret, if you look at any job posting for lackland, you'll see it mentioned over and over.. https://careers-salientcrgt.icims.com/jobs/11200/network-systems-engineer-2/job+&cd=4&hl=en&ct=clnk&gl=us https://careers-salientcrgt.icims.com/jobs/11200/network-sys...' https://i.imgur.com/d8KbSZp.png https://i.imgur.com/d8KbSZp.png But, wow, imagine that's a job, just walk in, look at two programs and swap out parts as needed.
    - Aperocky 6y ago
      The qualifications reads 'Someone from HR came up with this'
      - astrea 6y ago
        Isn't that true of most postings?
  - RobRivera 6y ago
    Facebook query Find people who work for US Air Force. Vulnerabilities publicly available are numerous, and I gave no such details to anyone that would give them an easier time finding said compromises. Its like saying windows 10 bug found --> HEY THE MILITARY USES WINDOWS 10.
  - willis936 6y ago
    My employer has a knowledgebase on the public internet that is littered with lists of softwares and practices. There are thousands of employees. Name dropping software should be a risky thing to do, but that isn’t the world we live in.
eli 6y ago
#2 is speculation. Seems possible that there's an unrelated bug causing checksum errors. In any event, it's not a good look right now.
- wyldfire 6y ago
  Regardless of the motivation, cause, mechanism of #2 - #3 is not the appropriate way to handle the problem. Attack is indistinguishable from unintentional corruption. And #3 trains customers to do the wrong thing when they encounter an attack.
  - eli 6y ago
    The malicious file was signed with the right certificate. So yeah you should ideally be more careful with checksums but there already was a much more robust and secure authentication mechanism and it was defeated.
    - heleninboodler 6y ago
      Yes, these are two orthogonal egregious security problems.
rayvd 6y ago
This seems an unfair leap. The most common cause of a checksum mis-match is going to be a partial download or something similar. It's also not relevant to the current attack since the code was legitimately included in the official release and, as such, baked into the valid checksum results.
- deleted 6y ago
  [deleted]
- tstrimple 6y ago
  Is the proper response to tell a customer to install the package anyway because it's just a partial download or something similar? Regardless, it seems irresponsible.
- deleted 6y ago
  [deleted]

35fbe7d3d5b9 6y ago

> SolarWinds hasn't bothered to revoke their certs or remove the package Amazing. While I'm sure the attackers have already shut up shop and the threat no longer exists, this feels insanely tone-deaf from SolarWinds.

bluedino 6y ago
Maybe they were just bribed?
- 35fbe7d3d5b9 6y ago
  An employee, possibly. The whole company, unlikely. And either way, even if someone was bribed to introduce the attack there's zero reason to allow the hacked software to be downloaded now. I work at a large and highly regulated (HIPAA) company and we have the equivalent of Electric Dylan/Pete Seeger with the axe: if someone at the VP+ level declares a major incident, our infosec team has a script that will lock down all inbound/outbound traffic, snapshot all our running machines for later forensics, lock our AWS IAM access down to a single incident response account, and move DNS for our web properties to a "we've been hacked" page. (OK, it obviously doesn't say that, but something similar that has been heavily vetted by legal and marketing ;-)). We've drilled and timed it out and can stop the ship in ~5 minutes. Either SolarWinds doesn't have a major security incident response plan, or they don't have the stomach to pull the trigger. Neither is promising.
  - strogonoff 6y ago
    Sounds like a solid information security incident response mechanism! The only missing piece is making sure that VP+ level folks are not incentivized in any way to suppress incidents. However, that’s beyond infosec—in that treacherous area between information security, shareholder interests and organizational politics. I wish business continuity planning (which would include infosec procedures but has a much wider overall scope) was paid more attention and more widely scrutinized.
    - 6y ago

coldcode 6y ago

Clearly whoever is the CIO/CISO could care less? I find it hilarious that people get these positions without seemingly a care in the world. Or maybe they do care and the CEO didn't? Hardly anyone ever gets fired in these circumstances.

ilikeerp 6y ago
Surely he COULDN'T care less?
- bobthepanda 6y ago
  This is one of those colloquialisms that has become so commonplace that it actually becomes a part of the language rather than being incorrect. https://www.oxfordlearnersdictionaries.com/us/definition/english/care_2?q=could+care+less https://www.oxfordlearnersdictionaries.com/us/definition/eng... lists it as a usage in North American English.
  - xbar 6y ago
    Those of us who know better bristle and retain pedantry. Consider semi-annual and biannual. Grr.
  - ilikeerp 6y ago
    Yes, when enough people do / say something wrong over and over again, it sometimes becomes normalised.
    - bobthepanda 6y ago
      Wrong would imply a right, but most interpretations of what is "right" are fairly arbitrary and mostly have to do with social and cultural hierarchy.
      - ilikeerp 6y ago
        Logically it's wrong. If you COULD care less then that means you do care to some level whereas most people mean you COULD NOT care less. Simple...

swiley 6y ago

I guess if you can be as successful as SolarWinds with that level of incompetence I should stop worrying so much about myself.

spondyl 6y ago
You'd be surprised honestly
Aperocky 6y ago
You'll be surprised at how technically illiterate most corporations are and how marketing and not engineering are responsible for the success for some of the software companies.

1vuio0pswjnm7 6y ago

Is it possible that there could be SolarWinds customers who are not vulnerable because, for whatever reason, they did not enable/install updates. Were updates to the Orion software necessary for the original software to continue to function or were they optional.

zimpenfish 6y ago
They've said that 33k customers were potentially exposed but only 18k actually downloaded that update. https://www.zdnet.com/article/sec-filings-solarwinds-says-18000-customers-are-impacted-by-recent-hack/ https://www.zdnet.com/article/sec-filings-solarwinds-says-18...
jdironman 6y ago
They were optional. I know a company still using a version from 2015/2016.

oldkn 6y ago

To be fair, the number of times this type of error occurs due to a state-backed actor who's quietly hijacked the build system with an undetectable backdoor is low compared to the number of customers who see the error for some other reasons (bad download for example) - so it's not entirely surprising they'd give this advice.