5 ms·

Absolutely agree. Solarwinds focuses a disproportionate amount of effort in ensuring it shows up favorably in Gartner magazine reviews and trade publications.

Absolutely agree. Solarwinds focuses a disproportionate amount of effort in ensuring it shows up favorably in Gartner magazine reviews and trade publications. As a monitoring platform its a monolithic, expensive, slow and rather dated monitoring solution. Agile does not come to mind, and you certainly wouldnt use it for anything approaching "observability." But the concerted marketing effort pays dividends. Solarwinds is almost the only choice for government. For potential attackers its a big red arrow. Exploit a rent-seeking company that writes mediocre software and exists mostly to cash in on "best practice" lock-in with government contracts. as far as i know nobodys really addressed the elephant in the room. Solarwinds is still a preferred government purchase for monitoring. every company that was affected by it still uses it (or at least hasnt publically refuted it) and no government official has come forward to admit they will discontinue it. Solarwinds hasnt offered any remediation or major changes in their development, leadership or code. just patch, rinse, and repeat and try not to pay too much attention to the issue.

downwithdisease 6y ago
>> "Solarwinds hasnt offered any remediation or major changes in their development, leadership or code." Not accurate re: leadership. A new CEO just started in January, and their CTO was either terminated or he left just after the hack news.
- spoonjim 6y ago
  LOL the whole “fall guy” thing is how companies keep doing what they’re doing while making some symbolic penance. For a software company a fuckup of this magnitude should be: all customers leave, company dies, execs never work again. Anything less is an insufficient incentive to work extremely hard to prevent this from happening.
  - notimetorelax 6y ago
    Sorry, I can’t agree with you here. What you’re saying is basically company’s dissolution of it is hacked. If this indeed becomes the case the companies will indeed kill to keep their secrets.
    - sofixa 6y ago
      First, with GDPR and co, they can't keep their secrets if they get hacked. Second, with such negligence? They deserve to be shut down. This isn't a highly sophisticated zero day exploit, there were multiple huge failure at Solarwinds that allowed the attack to happen.
  - mrmonkeyman 6y ago
    And if you ever ignore a red light, pay your taxes late or do anything other than your absolute best to be an upstanding citizen you should be put into prison for the rest of your life.
    - spoonjim 6y ago
      If you ignore a red light and kill 10,000 people? Maybe. That’s what this kind of software hack is.
  - bzzzt 6y ago
    Years ago, the company I worked at had a very big production issue which resulted in a customer's database being deleted. Of course, the customer was furious and called the CEO asking for the person responsible to be fired. Calmly, our CEO said: 'No. If there's anyone in this company who will never make that mistake again it's him.'
    - treis 6y ago
      I've seen this sort of Zen of CEO type story a lot but it doesn't match up with reality. Someone that deletes a production DB by being careless or reckless is likely to do something similar again. Perhaps not in the exact same way, but there's infinite ways to break things. Those that stumble upon one are likely to stumble upon another.
      - bzzzt 6y ago
        In that case everyone making 1 mistake is 'out'. Seems "cancel culture" is leaking into ops... I was in the room when the CEO told that (one of the nice things about small companies) and have had contact with the person responsible for a few years until he got another job. True story ;) There were changes though: the 'two pair of eyes' principle was enforced a lot stricter from then on.
  - WalterBright 6y ago
    Your method sounds good, but it makes things less safe in practice because it provides every possible incentive to hide, cover up, deny, etc., any problems. A far better system is "no fault" where the company has incentive to be open about problems and finding solutions.
unixhero 6y ago
Anectodally; I have been a one man army CTO for a period. I could not have managed my Windows servers without Solarwinds. It was an excellent product which did everything I could ever want.
- elygre 6y ago
  ...and something you would not ever want.
  - unixhero 6y ago
    Sure. It paid hansomely for a time. Then I got the hell out. :)
flower-giraffe 6y ago
> nobodys really addressed the elephant in the room. I think the elephant in the room is actually a more general issue that is cross platform and independent of the product implementation. It’s 2021 and we are still ignoring the fundamental “best practice” that we’ve known about for at least 20 years. Systems should be isolated from each other unless there is an overwhelming need for them to be connected and everything should run with the least privilege. Centralised credential vaults and binary artefact repositories are an obvious example.
- WalterBright 6y ago
  It's also important to have diversity. For example, on the Boeing 757 there are two computers that control the stab trim, that do the same thing. They must agree or both computers are automatically locked out. The two computers are developed by two independent teams who are not allowed to talk to each other. Two different CPUs, two different algorithms, two different programming languages. The idea, of course, is a design problem with one does not propagate to the other. If one program is used in all your infrastructure, all your infrastructure is compromised when that program has a bug.
  - aasasd 6y ago
    > They must agree or both computers are automatically locked out As a marine saying goes, never take two chronometers with you to the sea. Take either one or three.
    - WalterBright 6y ago
      I must remember to use "chronometer" rather than "clock".
    - hackeraccount 6y ago
      It's funny because it's true.
  - KajMagnus 6y ago
    I wonder if there's a way to bring that approach, into the world of 1) software build servers, 2) third party dependencies, and 3) code review. 1) Build servers: Maybe by keeping it possible to build one's software on a laptop — then, when it's time to release something, one can build the software at the build server plus some people's laptops and compare hashes. 2) ??? 3) Different people who don't talk with each other, do security code review, independently of each other? Hmm does that make sense
joncrane 6y ago
They release earnings for Q3 2020 on Feb 25th. The hack was announced Dec 12th so it will be difficult to gauge how many orders they lost as a result. On April 28th 2021 they will announce Q1 2020 results. THAT's when we'll know for sure how bad this hurt them. They may release less than sanguine guidance on Feb 25th, we'll see. The reason why I know this stuff I will leave as an exercise to the reader. :)
heresie-dabord 6y ago
The attacker in this case new the exact moves to insert a backdoor into .NET software. It wasn't hard to do, requiring no science. But Microsoft's "system of trust" was undermined. The attacker was even inside Microsoft and Azure. No anti-virus, no "defender", no amount of basic or advanced telemetry caught this. FireEye alone caught it... By accident. The real "elephant in the room" is that software security continues to be marketing theatre. Closed-source software should not be trusted and consumers should not believe that any due diligence has been done to protect them.
- jnwatson 6y ago
  I wouldn't say they caught it "by accident". They caught it the way that most organizations detect a compromise: they saw some suspicious network traffic and they investigated. They just happen to have among the most sophisticated investigation capability in the world.
  - heresie-dabord 6y ago
    From what I have read, the attacker attempted a second vector of infiltration, by e-mail. FireEye caught the second attempt, then noticed the SolarWinds activity.
- jorblumesea 6y ago
  > is that software security continues to be marketing theatre Overall I tend to agree with your general points, but, this statement. Would you say that seatbelts are "safety theatre" because some people die in car accidents while wearing a seatbelt? I really don't think it's fair to attribute all software security as BS, especially given the resources state actors throw at breaking software. It's like arguing your seatbelts don't work because your 4 door sedan was hit by a bus.
  - heresie-dabord 6y ago
    The standard proviso about metaphors being given, I see your point. Seatbelts constitute an effort, seatbelts save lives, testing with dummies and survival results show it. Seatbelts are real security measures. Still, more than 60K people die every year in motor accidents in North America and in the EU. Seatbelts cannot save life in all cases, such as outrageous speeds, but they are a sane measure for most -- if not all -- use-cases of driving. In this compromise, the major purveyor of OS and security products itself was compromised. No company, no $agency did due diligence. Some of these victims are supposed to be in the business of high-assurance and due diligence. None of the systems that we are conditioned to believe to be effective actually worked. And it was a standard use-case for all the victims.

downwithdisease 6y ago

>> "Solarwinds hasnt offered any remediation or major changes in their development, leadership or code." Not accurate re: leadership. A new CEO just started in January, and their CTO was either terminated or he left just after the hack news.

spoonjim 6y ago
LOL the whole “fall guy” thing is how companies keep doing what they’re doing while making some symbolic penance. For a software company a fuckup of this magnitude should be: all customers leave, company dies, execs never work again. Anything less is an insufficient incentive to work extremely hard to prevent this from happening.
- notimetorelax 6y ago
  Sorry, I can’t agree with you here. What you’re saying is basically company’s dissolution of it is hacked. If this indeed becomes the case the companies will indeed kill to keep their secrets.
  - sofixa 6y ago
    First, with GDPR and co, they can't keep their secrets if they get hacked. Second, with such negligence? They deserve to be shut down. This isn't a highly sophisticated zero day exploit, there were multiple huge failure at Solarwinds that allowed the attack to happen.
- mrmonkeyman 6y ago
  And if you ever ignore a red light, pay your taxes late or do anything other than your absolute best to be an upstanding citizen you should be put into prison for the rest of your life.
  - spoonjim 6y ago
    If you ignore a red light and kill 10,000 people? Maybe. That’s what this kind of software hack is.
- bzzzt 6y ago
  Years ago, the company I worked at had a very big production issue which resulted in a customer's database being deleted. Of course, the customer was furious and called the CEO asking for the person responsible to be fired. Calmly, our CEO said: 'No. If there's anyone in this company who will never make that mistake again it's him.'
  - treis 6y ago
    I've seen this sort of Zen of CEO type story a lot but it doesn't match up with reality. Someone that deletes a production DB by being careless or reckless is likely to do something similar again. Perhaps not in the exact same way, but there's infinite ways to break things. Those that stumble upon one are likely to stumble upon another.
    - bzzzt 6y ago
      In that case everyone making 1 mistake is 'out'. Seems "cancel culture" is leaking into ops... I was in the room when the CEO told that (one of the nice things about small companies) and have had contact with the person responsible for a few years until he got another job. True story ;) There were changes though: the 'two pair of eyes' principle was enforced a lot stricter from then on.
- WalterBright 6y ago
  Your method sounds good, but it makes things less safe in practice because it provides every possible incentive to hide, cover up, deny, etc., any problems. A far better system is "no fault" where the company has incentive to be open about problems and finding solutions.

unixhero 6y ago

Anectodally; I have been a one man army CTO for a period. I could not have managed my Windows servers without Solarwinds. It was an excellent product which did everything I could ever want.

elygre 6y ago
...and something you would not ever want.
- unixhero 6y ago
  Sure. It paid hansomely for a time. Then I got the hell out. :)

flower-giraffe 6y ago

> nobodys really addressed the elephant in the room. I think the elephant in the room is actually a more general issue that is cross platform and independent of the product implementation. It’s 2021 and we are still ignoring the fundamental “best practice” that we’ve known about for at least 20 years. Systems should be isolated from each other unless there is an overwhelming need for them to be connected and everything should run with the least privilege. Centralised credential vaults and binary artefact repositories are an obvious example.

WalterBright 6y ago
It's also important to have diversity. For example, on the Boeing 757 there are two computers that control the stab trim, that do the same thing. They must agree or both computers are automatically locked out. The two computers are developed by two independent teams who are not allowed to talk to each other. Two different CPUs, two different algorithms, two different programming languages. The idea, of course, is a design problem with one does not propagate to the other. If one program is used in all your infrastructure, all your infrastructure is compromised when that program has a bug.
- aasasd 6y ago
  > They must agree or both computers are automatically locked out As a marine saying goes, never take two chronometers with you to the sea. Take either one or three.
  - WalterBright 6y ago
    I must remember to use "chronometer" rather than "clock".
  - hackeraccount 6y ago
    It's funny because it's true.
- KajMagnus 6y ago
  I wonder if there's a way to bring that approach, into the world of 1) software build servers, 2) third party dependencies, and 3) code review. 1) Build servers: Maybe by keeping it possible to build one's software on a laptop — then, when it's time to release something, one can build the software at the build server plus some people's laptops and compare hashes. 2) ??? 3) Different people who don't talk with each other, do security code review, independently of each other? Hmm does that make sense

joncrane 6y ago

They release earnings for Q3 2020 on Feb 25th. The hack was announced Dec 12th so it will be difficult to gauge how many orders they lost as a result. On April 28th 2021 they will announce Q1 2020 results. THAT's when we'll know for sure how bad this hurt them. They may release less than sanguine guidance on Feb 25th, we'll see. The reason why I know this stuff I will leave as an exercise to the reader. :)

heresie-dabord 6y ago

The attacker in this case new the exact moves to insert a backdoor into .NET software. It wasn't hard to do, requiring no science. But Microsoft's "system of trust" was undermined. The attacker was even inside Microsoft and Azure. No anti-virus, no "defender", no amount of basic or advanced telemetry caught this. FireEye alone caught it... By accident. The real "elephant in the room" is that software security continues to be marketing theatre. Closed-source software should not be trusted and consumers should not believe that any due diligence has been done to protect them.

jnwatson 6y ago
I wouldn't say they caught it "by accident". They caught it the way that most organizations detect a compromise: they saw some suspicious network traffic and they investigated. They just happen to have among the most sophisticated investigation capability in the world.
- heresie-dabord 6y ago
  From what I have read, the attacker attempted a second vector of infiltration, by e-mail. FireEye caught the second attempt, then noticed the SolarWinds activity.
jorblumesea 6y ago
> is that software security continues to be marketing theatre Overall I tend to agree with your general points, but, this statement. Would you say that seatbelts are "safety theatre" because some people die in car accidents while wearing a seatbelt? I really don't think it's fair to attribute all software security as BS, especially given the resources state actors throw at breaking software. It's like arguing your seatbelts don't work because your 4 door sedan was hit by a bus.
- heresie-dabord 6y ago
  The standard proviso about metaphors being given, I see your point. Seatbelts constitute an effort, seatbelts save lives, testing with dummies and survival results show it. Seatbelts are real security measures. Still, more than 60K people die every year in motor accidents in North America and in the EU. Seatbelts cannot save life in all cases, such as outrageous speeds, but they are a sane measure for most -- if not all -- use-cases of driving. In this compromise, the major purveyor of OS and security products itself was compromised. No company, no $agency did due diligence. Some of these victims are supposed to be in the business of high-assurance and due diligence. None of the systems that we are conditioned to believe to be effective actually worked. And it was a standard use-case for all the victims.