12 ms·

Yes, but... Having worked at SolarWinds they're especially susceptible to demands from sales and marketing. "Go faster, ignore tech best practices, etc." It's

Yes, but... Having worked at SolarWinds they're especially susceptible to demands from sales and marketing. "Go faster, ignore tech best practices, etc." It's not unique, but their culture is not a dev-first, or security-first, culture to say the least. Many product managers answer to marketing first, and don't have earnest tech backgrounds that would let them know right from wrong past sales numbers. The culture changed significantly when they went public the first time; it went from a place where devs built good tools... to a place looking to buy products / competitor products so they could charge more for their services. Look at how long it took them to get into cloud tools -- great example of how marketing and sales missed the boat because they were only focused on things they had sold before and not focused on systemic changes to the industry -- because technologists weren't driving. Anyway, like I've worked a lot places with better security built into the culture, better tech best practices built into the culture... that's all I'm trying to say. Knowing that attacks like this are out there... and it was just a matter of time before it happened, SolarWinds did next to nothing to avoid it happening to them.

josho 6y ago
> SolarWinds did next to nothing to avoid it happening to them And how does one stop this from happening to them? Assume 'standard' security practices are in place, e.g. firewalls. Clearly the attacker was able to get their payload to be executed by a developer in SolarWinds. This payload clearly didn't trigger antivirus software, and hid itself from the developer and the build systems. How does a typical software dev. team guard against this?
- josephg 6y ago
  As others have mentioned, building on secured CI/CD infrastructure (+ ideally deterministic builds) would have helped here. But I'm not sure if thats good enough. Presumably if CI/CD was set up, some of the engineers would have had security credentials to configure any build servers anyway. And if the hackers could leverage that, they might have been able to just infect the build servers too. As annoying as it is, I can't help but think we need to bring parts of ios / android's security model to our desktop machines. Its ridiculous that a single rogue npm module can access any file on my computer, including SSH credentials and (probably) secure cookies in my browser's trust store. Our desktop security model is obsolete, and should do more to protect us from threats like this.
- dbg31415 6y ago
  When I worked there, about a decade ago, they were aiming for 40-50% year over year growth metrics. Anything that threatened that was drug out and shot. If you were a dev who was like, "Hey, we need to test our stuff... do proper code reviews... not outsource quite so much..." you were not treated kindly. QA cycles were definitely rushed to hit launch dates; if Dev was delayed, rather than push the launch the QA window just got shorter. And the QA team was outsourced -- which further reduced their ability to give any push back. I just remember it was the sort of place where -- again not uncommon -- but every password was like "$olarwinds" (making that up, but password reuse was rampant) and passwords were shared in plain text / written on whiteboards, etc. Also a lot of tool / account re-use, and a lot shared passwords there too -- everyone using the same Google Analytics account sort of thing, and huge walls up around getting more licenses for tools you needed so you'd have to share sign ins with other people. I remember a ticket to "sync passwords" across a bunch of test environments with production... just intentional bad practices left and right; convenience > security at every turn. At times it felt a bit like a frat house. The sales guys had a big gong they'd ring, the CEO was a cheerleader who would run down the halls yelling at the top of his lungs, "Woo! Woo! Another sale!" type stuff. There was no security training in place... for anyone. No password managers. No security audits as far as I could tell. No 2FA as far as I could tell. Generally speaking any badge worked to open any door. And you'd be met by anger if you ever tried to flag anything like, "Hey, I think the QA team just closed all my tickets without actually testing them..." Been a while, and like I said, not unique... but they were just sloppy. But back to the question: what can you do? There's not like one magic bullet here. But they need to make security a priority. Do whatever it takes to make it so everyone in the company stops and thinks for 30 seconds about if what they're doing is a good idea from a security standpoint or not. Here's a good starter pack for you to help your company get their foot in the door for security best practices. https://sudo.pagerduty.com/for_everyone/ https://sudo.pagerduty.com/for_everyone/

josho 6y ago

> SolarWinds did next to nothing to avoid it happening to them And how does one stop this from happening to them? Assume 'standard' security practices are in place, e.g. firewalls. Clearly the attacker was able to get their payload to be executed by a developer in SolarWinds. This payload clearly didn't trigger antivirus software, and hid itself from the developer and the build systems. How does a typical software dev. team guard against this?

josephg 6y ago
As others have mentioned, building on secured CI/CD infrastructure (+ ideally deterministic builds) would have helped here. But I'm not sure if thats good enough. Presumably if CI/CD was set up, some of the engineers would have had security credentials to configure any build servers anyway. And if the hackers could leverage that, they might have been able to just infect the build servers too. As annoying as it is, I can't help but think we need to bring parts of ios / android's security model to our desktop machines. Its ridiculous that a single rogue npm module can access any file on my computer, including SSH credentials and (probably) secure cookies in my browser's trust store. Our desktop security model is obsolete, and should do more to protect us from threats like this.
dbg31415 6y ago
When I worked there, about a decade ago, they were aiming for 40-50% year over year growth metrics. Anything that threatened that was drug out and shot. If you were a dev who was like, "Hey, we need to test our stuff... do proper code reviews... not outsource quite so much..." you were not treated kindly. QA cycles were definitely rushed to hit launch dates; if Dev was delayed, rather than push the launch the QA window just got shorter. And the QA team was outsourced -- which further reduced their ability to give any push back. I just remember it was the sort of place where -- again not uncommon -- but every password was like "$olarwinds" (making that up, but password reuse was rampant) and passwords were shared in plain text / written on whiteboards, etc. Also a lot of tool / account re-use, and a lot shared passwords there too -- everyone using the same Google Analytics account sort of thing, and huge walls up around getting more licenses for tools you needed so you'd have to share sign ins with other people. I remember a ticket to "sync passwords" across a bunch of test environments with production... just intentional bad practices left and right; convenience > security at every turn. At times it felt a bit like a frat house. The sales guys had a big gong they'd ring, the CEO was a cheerleader who would run down the halls yelling at the top of his lungs, "Woo! Woo! Another sale!" type stuff. There was no security training in place... for anyone. No password managers. No security audits as far as I could tell. No 2FA as far as I could tell. Generally speaking any badge worked to open any door. And you'd be met by anger if you ever tried to flag anything like, "Hey, I think the QA team just closed all my tickets without actually testing them..." Been a while, and like I said, not unique... but they were just sloppy. But back to the question: what can you do? There's not like one magic bullet here. But they need to make security a priority. Do whatever it takes to make it so everyone in the company stops and thinks for 30 seconds about if what they're doing is a good idea from a security standpoint or not. Here's a good starter pack for you to help your company get their foot in the door for security best practices. https://sudo.pagerduty.com/for_everyone/ https://sudo.pagerduty.com/for_everyone/