16 ms·

SolarWinds CEO blames intern for password leak

305 points by tnolet 6y ago

rvz 6y ago
Should have used an encrypted complex password and a password manager. It is the fault of both of you. ¯\_(シ)_/¯
- gavingmiller 6y ago
  In no way is this an interns fault. If your entire infrastructure relies on the secure password of ... checks notes ... a single intern! then you're doing it wrong.
  - rvz 6y ago
    > "secure password" Whatever that means. This would never have happened in the first place had they used an encrypted complex password and a simple password manager. The whole company takes the hit with blunders like this. It's everyone's fault responsible for the infrastructure allowing this to happen in the first place. Those who pass the blame on others very quickly are equally to blame which means the CEO is just as to blame as the 'intern'. Clearly the whole company doesn't train their interns.
    - frombody 6y ago
      The password was coded into a file that was checked in to git. The git repository just happened to be public. It's entirely reasonable to think that the person in question possibly didn't even stop to think that Solarwinds123 was an actual secret that needed to be kept, as it is the equivalent of common passwords that are published publicly in manufacturer documentation.
    - shoelessone 6y ago
      I think the point is at no point is it acceptable to be in a position to be able to do something deeply damaging to a company with something as simple as a intern leaking a password. The intern should never been put in a position where this was even possible. I’d say in all the ways that matters this was basically everybody BUT the interns fault.
- mhh__ 6y ago
  You can't really blame someone who is explicitly there to basically do what they're told and learn as much as they can for a short period of time.
kbd 6y ago
Good companies: “this was a failure of process, no individual is responsible”. SolarWinds: “the intern revealed our ‘solarwinds123’ password.”
- jacquesm 6y ago
  Yes, an individual was responsible: the CEO. Ultimately that job comes with the responsibility for everything happening inside the company and as a CEO you ensure that you have the proper culture and control mechanisms in place that that sort of thing does not happen, and that if it should happen that it gets detected.
  - lawnchair_larry 6y ago
    By your standards, zero companies have a proper “culture” then. That’s just not how things work anywhere. You might think or even assume they should, but they don’t.
harryf 6y ago
The story line may be being twisted by CNN but the idea of a CEO blaming his staff is fundamentally wrong. The message should be “I created a culture which meant an intern could put the whole company at risk. This was my mistake”
- loceng 6y ago
  Ted Cruz claimed it was his children's fault for pressuring him into their trip, e.g. there is no lacking in incompetence and integrity in many systems, organizations, companies, communities, families. It starts with the individual which requires someone caring about how they look, about accountability, chain of command/responsibility. How do we solve this though? Who is the CEO responsible to - who do they care, if anyone, about how they appear to? How do we shift this behaviour for the examples of CEOs and politicians who maybe haven't been outed yet by such a security breach but are vulnerable to such lacking of a rigid structure of command/responsibility? Edit to add: I'm beginning to think HN is full of really lazy people.
  - rootusrootus 6y ago
    > Edit to add: I'm beginning to think HN is full of really lazy people. Because of downvotes? That's probably just because you brought partisan politics into the discussion.
    - reagank 6y ago
      > Because of downvotes? That's probably just because you brought partisan politics into the discussion. Wut? Mentioning a politician’s name isn’t partisan politics. He said nothing about the politics of Cruz, he compared the Senator shifting blame onto his kids with this CEO blaming an intern. Don’t be like that. (Edited to correct typo)
      - loceng 6y ago
        Thank you for pointing this out - it's unfortunate that people can be immediately triggered to react after simply seeing a name that prevents/blocks them from continuing on to understand or critically think to understand what's actually said; it's definitely a sign of the times and state of our health and our thinking ability as a society.
switchstance 6y ago
Ultimately, the CEO should take responsibility for everything. Finger pointing is not a leadership skill.
- duckmysick 6y ago
  Somebody appointed the CEO, no?
  - kowlo 6y ago
    Ah, yes. To blame someone, we must first invent the universe.
  - tialaramex 6y ago
    Sure, SolarWinds is a corporation, it will have a Board of Directors representing the interests of its shareholders, it has at times been a private company (so its shareholders were some handful of private equity investment companies) and public (so they will have included funds and the great unwashed) but in either case the Directors are responsible. So in terms of specific people that's Bill Bock, the chairman of that board at the time. They did indeed replace their CEO. I would assume after agreeing to pay off the old one as this is the usual practice and, unlike some no-name intern, a handsomely compensated CEO can afford expensive lawyers if you try to kick them out uncompensated for incompetence based merely on the evidence that their inadequate oversight cost you billions of dollars.
lousken 6y ago
Intern, yea sure, the cheapest excuse ever
alfiedotwtf 6y ago
This is the equivalent of stock crashes being blamed on what the media has dubbed “fat fingers” of a trader
octetta 6y ago
What a good look... one of the people at the highest tier of a company blaming someone at the lowest tier. Are we all this blind?
- loceng 6y ago
  No, we're all mostly just distracted by so many problems and failures throughout our society. Someone commented the other day mentioning how it doesn't take a genius to see the West is degenerating. To me it's obvious it's the industrial complexes involved in regulatory capture - fuelled by bad actors, greed, targeting the richest nation in history - leading to bad policy for people, individuals, for too many decades now; including policy allowing the duopoly to exist, thrive, and the tightening and expanding conglomerates of a handful of mainstream media companies also then perpetuating the two main narratives of the duopoly, along with whatever other for-profit interests decide to influence/manipulate us through shallow, cheap advertising. Thank Goodness for the internet, technology, to allow anyone with a voice a better chance at gaining a following and a communication channel - so we can bypass the control of those systems to educate people, and share different narratives - truths.
carlisle_ 6y ago
Did these guys really think people would buy it when their excuse is a literal cliché? What’s next? Did a dog eat their homework?
mhh__ 6y ago
Always the honourable thing to do!
nfjrbrnnffk 6y ago
I worked at two big, well known software companies which had the same default password - company123, albeit for low security stuff, where you wanted wide access but a password was required for some reason.
- chapium 6y ago
  I find it interesting that some companies don't secure their test domains, but do rely on merging from test to prod.
  - YarickR2 6y ago
    Well, a lot of companies are doing it in reverse , by not properly securing test sites, and bringing live sensitive data from prod to test . I don't know which is worse
llarsson 6y ago
The point of being a CEO is that you are ultimately responsible for what decisions are made. Even if decisions are delegated. Because guess what, you as CEO are responsible for delegating correctly. Failure to realize this is shameful.
- iou 6y ago
  +1
- krapp 6y ago
  There's a reason "shit rolls downhill" is a common idiom.
  - sonotathrowaway 6y ago
    “The buck stops here” is also a well known saying. President Truman had a sign on his desk with the saying.
    - srswtf123 6y ago
      Alas, thats a relic of a bygone era. More recently, our politicians simply “don’t recall”, or worse directly lie to us. For C-level folks, its simpler. Take no responsibility ever, unless forced to by the courts. Even then, taking actual responsibility is so rare that I have no examples.
    - pkulak 6y ago
      Sadly, now the people with the most power also want the least responsibility (and the most money).
      - deleted 6y ago
        [deleted]
      - toss1 6y ago
        Yes, this describes nicely what sociopaths seek. That this is common in most organizations shows that most larger organizations end up being selection systems for filtering sociopathic to the top. It is not because their are deliberately designed this way, but because this is what sociopaths seek, and the organization fails to actively filter against it.
akadruid1 6y ago
I think they know this is a pretty desperate story. Some of the other coverage [1] suggest Solarwinds had so many different overlapping security failures they may never be able to attribute to a single cause. Still there's some interesting things that could help a (much smaller, less critical) software vendor decide where to focus their security efforts. Perhaps near the top of the list should be: 1. Who in your organisation has access to your build and distribution toolchain, and how secure are their credentials? 2. How good is your record keeping? Are all your builds traceable back to a specific revision in your source control, and are you keeping build logs somewhere they can't be tampered with? [1] https://www.bloomberg.com/opinion/articles/2021-02-26/deepening-mystery-and-fear-over-the-solarwinds-hack https://www.bloomberg.com/opinion/articles/2021-02-26/deepen...
ChrisMarshallNY 6y ago
Waitaminute...The password was “solarwinds123,” and the CEO is blaming an intern?
- slickrick216 6y ago
  Yeah because everyone knows you go all the way to 9 like you are slapping the piano keyboard in movie Big.
mkl95 6y ago
As an intern, you usually work for free or for subpar money, and expect to be taught the basics of the job and some good fundamentals. The fact this intern lacked the security knowledge any mom and pop company employee should have suggests the company simply wanted free labour from them.
TruthWillHurt 6y ago
And the same culture runs through Microsoft- Rep. Katie Porter: "You and your company were supposed to be preventing the Russians from reading DoD emails!" Microsoft President Brad Smith: "There is no indication, to my knowledge, that the DoD was attacked" That's accountability! dilligence! Microsoft at its best.
higeorge13 6y ago
This is so bad. A proper ceo, cto, executive, manager, leader does not point any fingers.
jorblumesea 6y ago
Would you blame the intern if they ruined the production DB after being given write access? I wouldn't. I'd blame senior technical leadership for putting processes into place that allow failures like that. Most especially, the CEO.
dmingod666 6y ago
If your companies existence relies on interns not making mistakes. Security breach is the smallest of your issues.. Company board is probably like, fingers crossed, hope the other interns are good... The last time an intern made such a disproportionate impact was in 1996.
jacquesm 6y ago
So, who is responsible for giving an intern - likely not even strictly under an employment contact with the company - access at that level. And who, in turn is responsible for failing in their oversight duty of this situation? And who in turn is responsible for a the department that this fell under where such oversight duty failures are possible? And so on. Before long, you're back at the CEO, but that time the question will stick. As a CEO you've got nobody to blame after your first 90 days of employment.
- DoubleGlazing 6y ago
  I don't think I have ever been more than one week in to a job before I was given some form of admin level rights. In small companies I can understand that. But I've had it also happen in government jobs, in big companies and startups that collect a lot of sensitive personal data. In one case I was with a startup that had names, addresses, DOBs, phone numbers and debit/card details in their DB. When they hired me they didn't ask for references or ID. I was given full admin rights to their Azure account on day one. If I were in charge I would at the very least want to validate a new employees ID before giving them any form of access to IT systems, and only elevate access privileges once they had an established track record within the company.
  - jacquesm 6y ago
    As an intern? I would find it somewhat worrisome if an intern at some random company would be given access to mission critical systems and data within a week. That would definitely qualify as an oversight failure, and 'small' is a pretty relative affair, if the company has four people then I could see your point - maybe - but if it had 30 or more than it really isn't ok.
boh 6y ago
The fun thing about being in charge is that everything is your fault. Blaming the intern, the person least able to defend themselves, is ridiculous. If an intern can destroy your business, you're doing it wrong.
- ed25519FUUU 6y ago
  What’s more damaging is that a simple password is the only thing keeping your business from destruction, not to mention exposing all Americans with a security risk.
- vardaro 6y ago
  And its at the expense of a young professional just starting their career...
mensetmanusman 6y ago
That is a symptom of an organization that improperly allows interns to access important information.
hibbelig 6y ago
According to the article, SolarWinds doesn't seem to think that the password itself is a problem, only that it was leaked. And they "took it down", that sounds as if they deleted the leaked document. It doesn't sound as if they changed the password. Amazing.
- reagank 6y ago
  Seriously. Blaming the intern is a bad look, but here it reinforces the idea that they don’t even understand what was wrong. The fact that it leaked is only relevant is that it shows 1) how bad their password is and 2) how deficient their process is for dealing with (let’s face it, inevitable) leaks.
jpmattia 6y ago
> Current and former top executives at SolarWinds are blaming a company intern for a critical lapse in password security that apparently went undiagnosed for years. The SolarWinds execs have provided us with keen insight for the root cause of the SolarWinds attack, but the insight they conveyed is probably not the insight they intended to convey.
AdmiralAsshat 6y ago
> "They violated our password policies and they posted that password on an internal, on their own private Github account," Thompson said. So let's analyze the various scenarios under which the "intern" might have been responsible, and why each one is bullshit. 1) The intern exposed a company password on their own account. Counter: What kind of "password policies" allowed such a weak password in the first place? 2) The intern came up with the weak password themselves, thus violating "password policies" not just for secrecy but strength/security. This password was then used for several critical, production applications. Counter: Why was an intern in charge of deciding a password used for anything critical? 3) The intern came up with the weak password and exposed it, but it was only used for the intern's own corporate accounts (e.g. their Windows workstation). Counter: Why did an intern have a level of access such that their account being breached could lead to this level of compromise/exfiltration? Conclusion: There is no conceivable scenario under which this makes sense.
- deleted 6y ago
  [deleted]
- laurent92 6y ago
  Worst scenario: The intern is malevolent. Counter: Credentials should be given progressively as trust is built (and as lessons are learnt), and an intern can’t have access to production. I have an employee who I thought I could give more responsibilities, but he keeps not locking his computer when he walks away. He has very limited access to everything and it would impede his career if he didn’t also have the same attitude about other issues. (My question is, how do I make him diligent — It’s real potential wasted).
  - ethanwillis 6y ago
    Have you explained this to him?
    - marcus_holmes 6y ago
      Further, have you really listened to his explanation of why he isn't bothered about locking his computer? It might be that in your organisation this is a cargo-cult security practice that he's not bothered by because he knows it's not an effective practice. Or it could be that he knows he doesn't have enough permissions to do any damage, so he doesn't bother locking his computer. Trusting him with a little responsibility might change that.
jmfldn 6y ago
It shouldn't be possible for an intern to leak a password, or if it is, the blast radius should be limited. In other words, it's the job of senior people to make this sort of thing hard or at least not super damaging when it occurs.
plandis 6y ago
Blaming the intern instead of root causing it as a systemic failure just leads me to wonder if they will actually take the correct steps to prevent this from happening in the future. Fwiw this is a good archive of mostly well done post mortems: https://github.com/danluu/post-mortems https://github.com/danluu/post-mortems
dartharva 6y ago
The article says the "former CEO" said that, not the current CEO.
dccoolgai 6y ago
Pretty terrible when you think of all the govt money this company took to provide these services. Likely charged at massively inflated rates for the benefit of their "expertise". Guarantee you that intern was probably billed out at 10-20x what it cost SolarWinds to employ them. Doesn't even pass a level 1 smell test.
toss1 6y ago
A chain is only as strong as its weakest link. This means that the CEO, CTO, and all management down the chain failed to create any kind of robustly secure system. Their entire system was never more than a single password leak away from complete failure for the security of the entire USG. With literally thousands of opportunities for such leaks per day, it is inevitable. So, a system must be designed such that when the inevitable occurs, the consequences are minimal. The most basic of fail-safe designs. These people, from the CEO on down, utterly failed to do this. Yet they made millions from false claims to have succeeded in creating a secure system, when they created a highway for global espionage. They literally could not have provided our adversaries a better avenue for espionage if they tried. The free world would literally be better off if all of that company and it's management had never existed. Meanwhile, as a small manufacturer who has some govt-related work, the blizzard of new security certification requirements for Controlled Militarily Critical Technical Data and the like coming down the pike is like nothing I've ever seen before. Sure, some of it is due to increased threats, but much of it is definitely because of a*$holes like those CxOs who so utterly and deliberately failed in their most basic duties, and should never work in the industry again.
citizenpaul 6y ago
Cool so its totally not his fault then. If only there was someone in charge that could handle this situation.
hedora 6y ago
So, if you want to compromise the security of 100,000’s of IT departments, get an internship at SolarWinds?
- loveistheanswer 6y ago
  This is an interesting way of looking at it. Undoubtedly there must be nation state actors using such attack vectors
sjreese 6y ago
The password is invalid on its face Eg - Maximum number of days a password may be used. Minimum number of days allowed between password changes. Number of days warning given before a password expires. Says that you have to use any password within 0+x days - otherwise it would have expired - the older the posting the more unlikely it would be valid. Would you risk detection with a password without mix case and consecutive numbers. Most would avoid traps or suspect a trap
- bawolff 6y ago
  Password expiries are not really considered good practise anymore.
ineedasername 6y ago
That really doesn't make them look any better. If a single intern's password mishap can breach a security company's systems on this level, they've lost the fight long before this incident.
zepto 6y ago
This just means that the CEO doesn’t understand security risks and must be fired. Any company that continues to rely on Solarwinds after this, if the CEO is not fired, is accepting that their security is only as good as a SolarWinds intern.
- itsdrewmiller 6y ago
  He already resigned - the actual CNN headline is "Former" CEO.
  - deleted 6y ago
    [deleted]
womitt 6y ago
This is the weakest leadership reaction possible in a giant fuckup scenario like this
SecurityLagoon 6y ago
Wow, way to remove your product from consideration ever again. Security breaches happen - fix up the issues and show people your reform. Blaming the intern just makes you look like an ass whos company I never want to do business with again.
stanrivers 6y ago
If you have a system setup where a single intern can destroy your company, you are doing something wrong. Let's even say the intern was malicious and trying to do harm... that is still your fault. One person in the lowest position in the company can break everything? Again, you are setup incorrectly.
aklemm 6y ago
Somebody hasn’t read Extreme Ownership.
pjmorris 6y ago
"A loss of X dollars is always the responsibility of an executive whose financial responsibility exceeds X dollars." - Gerald Weinberg's 'First Principle of Financial Management' and 'Second Rule of Failure Prevention' [1] [1] 'First-Order Measurement', Quality Software Management, Volume 2, Gerald Weinberg, Dorset House Publishing, 1993
vinay_ys 6y ago
I prefer to remember only one long passphrase and enter it in only one place – my password manager. And then I trust my password manager to generate, store and submit the right password to the right portal and only upon my consent. This is vastly better than I remembering and entering the passwords myself. But it is still not perfect. I would rather all websites and apps universally moved to a standardized machine friendly authentication API and then I have a authentication user-agent (my password manager) do the actual authentication.
- gizmo385 6y ago
  Wouldn't that be introducing a globalized single point of failure?
  - vinay_ys 6y ago
    The password managers can have many different strategies for keeping their seed secret – it could be local hardware backed or cloud based but locked to a few of the devices you own etc.
- jtsiskin 6y ago
  This exists - webauthn!
ldbooth 6y ago
Someone didn't see Spaceballs. "12345? That's amazing I've got the same combination on my luggage!"
quallzone 6y ago
There is a stark contrast in the former SolarWinds CEO response versus Berkshire Hathaway CEO Warren Buffet's attitude. I noticed an interesting statement in Buffet's 2020 earnings report: "The final component in our GAAP figure – that ugly $11 billion write-down – is almost entirely the quantification of a mistake I made in 2016. That year, Berkshire purchased Precision Castparts (“PCC”), and I paid too much for the company. No one misled me in any way – I was simply too optimistic about PCC’s normalized profit potential. Last year, my miscalculation was laid bare by adverse developments throughout the aerospace industry, PCC’s most important source of customers." . . . "I was wrong, however, in judging the average amount of future earnings and, consequently, wrong in my calculation of the proper price to pay for the business. PCC is far from my first error of that sort. But it’s a big one." WOW. I rarely see that level of accountability at senior organizational levels.
- jtsiskin 6y ago
  SolarWinds: “My intern had prepared a report on PCC’s profit potential, which I used to decide the purchasing price. It turns out the intern had not followed our financial analysis framework correctly. This mistake therefore lies squarely on the interns shoulders, please talk to them if you have any complaints. My company and I should not be blamed.”
- throwaway888abc 6y ago
  There is a man and Man.
908087 6y ago
What's amazing is that they thought this claim would make them look better.
Threeve303 6y ago
The same intern worked for Volkswagen as a developer and is solely responsible for the whole miles per gallon debacle.
andy_ppp 6y ago
It’s extremely weak leadership to blame the intern... makes me think it’s probably the CEO’s fault with such a lack of character.
jtdev 6y ago
This is modern American corporate culture after all. Lie, cheat, steal, win at all costs and if you lose blame someone at the bottom of the ladder.
itsdrewmiller 6y ago
@dang any chance this could be updated to the correct headline? So many comments in this thread calling for the firing of a CEO who already resigned.
- slickrick216 6y ago
  Dig him up and fire him again.
- sjg007 6y ago
  New CEO should be fired too. They are failing at crisis management 101.
colechristensen 6y ago
This is great and very publicly (unintentionally) shows exactly why SolarWinds got hacked. A CEO scapegoating an intern is what you would expect from a company with very deep systematic problems which led to such an embarrassing failure. If your least experienced employee can accidentally topple the entire company, it is the entire company at fault. There are cases where individuals can undermine a whole organization and be at fault, but that requires sophistication and corruption which go above and beyond solid safeguards.
cgb223 6y ago
Lol SolarWinds is never going to be able to find an intern again. Who would want to work for a company that throws them under the bus for the largest hack in recent history
gameswithgo 6y ago
ohhh really bad move blaming the person instead of the process, CEO.
sumoboy 6y ago
What's next, the janitor was responsible for the companies poor quarterly results?
say_it_as_it_is 6y ago
This is so outrageous a claim that it warrants the technology community responding to it. Where is Bruce Schneier?
adrianmonk 6y ago
Here's more context about what the password was for. https://twitter.com/vinodsparrow/status/1338431183588188160 https://twitter.com/vinodsparrow/status/1338431183588188160 This tweet has a screenshot of an email from Vinoth Kumar (named in the CNN article) to SolarWinds saying: Hi Team, I have found a public Github repo which is leaking ftp credential belongs to SolarWinds. Repo URL: https://github.com/ Downloads Url: http://downloads.solarwinds.com FTP Url: ftp://solarwinds.upload.akamai.com Username: Password: POC: http://downloads.solarwinds.com/test.txt I was able to upload a test POC. Via this any hacker could upload malicious exe and update it with release SolarWinds product. (The tweet blanks out some things including part of the github URL, the username, and the password.) My thoughts: (1) I assume this means when it comes to technical measures to prevent a weak password, SolarWinds would have to rely on Akamai. (2) The researcher was able to upload to the root directory of downloads.solarwinds.com. As an educated guess, this may have been a shared account and many people knew this password. When many people share an account, they tend to choose passwords that are easy to convey to someone else. If so, the intern probably didn't create the password and was only responsible for leaking it.
- jtsiskin 6y ago
  Also note from that thread: SolarWinds gave him $0 for finding and reporting this. That in itself is completely irresponsible and a reflection of the companies systemic issues, which can’t be blamed on one intern.
aszantu 6y ago
lol if an intern can leak critical passwords you prolly got a ton more problems than just the intern leaking critical passwords
corobo 6y ago
We still doing the blame the intern thing eh?
jzer0cool 6y ago
Let us step back one. What policy in place which allows such a password in the first place?
endymi0n 6y ago
Oh those interns... Few years ago, police launched an investigation into a European dating company. They were prohibited by court order to destroy any evidence. Turns out in an unfortunate and unforeseen turn of events, an intern wiped their production Hadoop cluster just a week later with the backups having some issues. He was fired pretty quickly, but I heard he hasn‘t been too bad off since... Don‘t hire any interns, they can do quite a lot of damage!
sjg007 6y ago
This CEO needs to go, he apparently has zero experience in crisis management. He has signaled to all of his employees that yes he will throw them under the bus. He has signaled to his customers that there is not enough oversight on operations. As such he has now lost all credibility. It's a shame really, to blow a totally manageable crisis where solar winds could emerge stronger as a result.
zeckalpha 6y ago
Next week: SolarWinds board blames CEO for password leak
skynet-9000 6y ago
We always use interns to do security critical work -- the less experience, the better. We also have exacting security standards, and a Critical Password Policy, since there are no non-critical passwords. Critical Password Policy Rule 1: All critical passwords must contain the company name, lower case, with no special characters or spaces. Rule 2: All critical passwords must have between 3 and 1024 randomly selected sequential integers appended, each of no less than 1 and no greater than 3. When you are creating your critical passwords, it is critical that you follow the rules in the exact sequence as stated above, and that you do not introduce any external sources of randomness or entropy. Failing to follow this Critical Password Policy may result in your dismissal and later blame before a Congressional committee.
- camjohnson26 6y ago
  That’s all well and good but you should really make sure to disable any kind of TLS on these connections so the security team can monitor for breaches. Any ops team worth its salt will make sure to secure all endpoints by physical device address and remove single points of access like gateways to widen the attack surface and ensure attackers can’t focus on one location. Store any critical data in a basic S3 bucket and remove any firewalls on production databases so that anyone on the team can fix a problem or restore a backup if needed. And you should really disable most logging, it’s a performance bottleneck and can tell attackers what’s happening in the system. I’ll type these notes up in a PowerPoint and we can deliver a presentation to key stakeholders.
jVinc 6y ago
> Confronted by Rep. Rashida Tlaib, former SolarWinds CEO Kevin Thompson said the password issue was "a mistake that an intern made." Imagine a bank where the CEO says "the problem with all the money going missing was that an intern dropped the keys to the security vault and we had told the guards to never question anyone who had the keys, just let them take whatever they want without question". Seems to me that you can't pin anything on the intern. The problem was the extreme lack of security practices at the company, which ultimately falls on the CEO, who's trying to blame his own incompetence on a intern.
j_barbossa 6y ago
Maybe it was the intern's fault to set a weak password but it is the CEOs fault for setting up an organization where things like these can slip through security review or monitoring.
awiesenhofer 6y ago
While this of course is pure BS, with the way they handled all this so far my trust in them has been lost quite a while ago. And I guess I am not the only one. My question though is what alternatives are people moving to now? Any experiences?
lr1970 6y ago
The fact that the SolarWinds CEO dared to blame the whole security breach on a single intern is a damning testament to the rotten security culture at SolarWinds. Would you trust a company where a single intern can compromise security of all of its clients? The fact that the CEO does not understand how damning his admission is, makes the whole situation hopeless.
IronWolve 6y ago
Remember the old nortel network equipment default passwords like admin/admin that telecoms didnt change? Good times.
dang 6y ago
Related ongoing thread: SolarWinds CEO blames intern for password leak - https://news.ycombinator.com/item?id=26284782 https://news.ycombinator.com/item?id=26284782
somerandomness 6y ago
They really should have tried harder to find a more convincing scapegoat.
rayiner 6y ago
The buck stops with intern. That’s why the intern gets paid like 20 million dollars a year.
andred14 6y ago
Come on, who believes anything CNN says anymore after the hundreds of lies in 2020?
abvdasker 6y ago
The only thing the CEO of SolarWinds is telling us by saying this is that he's a terrible leader. We don't even know that the hackers used the exposed password at this point. But even if they did, this is an obvious process failure and by blaming an individual intern, Sudhakar Ramakrishna is loudly demonstrating his own hopeless incompetence.
xyst 6y ago
poor intern is getting thrown under the bus
imron 6y ago
It was either an intern or a third-party contractor, and they went with intern.
mickael-kerjean 6y ago
A few years back, I caught a security consultant from Thales pushing his credentials on GitHub as he was working on what seemed to be a personal project during office hours and trying to get throught the corporate proxy... Considering Thales creates the most sensitive system, I completely lost trust in companies who claim to make secure products
spacecowboy 6y ago
This environment sounds like a case of no accountability and negligence. If a intern can make this type of decision without review and accountability, what other decisions are being made under the same environment of no review and accountability within the company?
nibsfive 6y ago
As a 20 person startup we have a KMS with only two authorized users. Seems like SolarWinds tried pretty hard to be hacked.
mbrodersen 6y ago
CEO’s are typically quick to claim large bonuses “because of the huge responsibility I have” and even quicker to not take responsibility (and keep their bonuses) when things go wrong.
toss1 6y ago
tl;dr: If this CEO's security systems were so weak that a single password leak by an intern can compromise his entire company, the entire US Govt and many others, and much of the Fortune500, then his security systems suck - all the way up to insane values of suckage. The fact that he is blaming the intern shows that he knows his systems suck and is attempting to divert. The entire management chain from CEO down to the intern's manager is at best too incompetent to work in the industry, and more accurately, a major threat to national security. Anyone who trusts these people to do so much as run out for a pizza is a fool.
retox 6y ago
Personally, I am more comfortable socializing benefits and privatizing (my own) mistakes. Which might not be the best way of phrasing it, but if someone else made a mistake, whether it's widely known or just something I've been privy to, I'm happier saying that "there was a mistake" rather than pointing the finger. However if I made the mistake, I'm content to say it was something I personally did wrong, it either case I will follow up with the steps we can take to prevent it happening again. Conversely, if I delivered something of noteworthy benefit for the project I will talk about it in terms of how 'we' achieved the success, and if it was someone else I will call them out by name and congratulate them on the solution. This ethos is probably career limiting as a dev, but it does help me sleep soundly.
xtat 6y ago
what a shitshow

305 points by tnolet 6y ago

rvz 6y ago

Should have used an encrypted complex password and a password manager. It is the fault of both of you. ¯\_(シ)_/¯

gavingmiller 6y ago
In no way is this an interns fault. If your entire infrastructure relies on the secure password of ... checks notes ... a single intern! then you're doing it wrong.
- rvz 6y ago
  > "secure password" Whatever that means. This would never have happened in the first place had they used an encrypted complex password and a simple password manager. The whole company takes the hit with blunders like this. It's everyone's fault responsible for the infrastructure allowing this to happen in the first place. Those who pass the blame on others very quickly are equally to blame which means the CEO is just as to blame as the 'intern'. Clearly the whole company doesn't train their interns.
  - frombody 6y ago
    The password was coded into a file that was checked in to git. The git repository just happened to be public. It's entirely reasonable to think that the person in question possibly didn't even stop to think that Solarwinds123 was an actual secret that needed to be kept, as it is the equivalent of common passwords that are published publicly in manufacturer documentation.
  - shoelessone 6y ago
    I think the point is at no point is it acceptable to be in a position to be able to do something deeply damaging to a company with something as simple as a intern leaking a password. The intern should never been put in a position where this was even possible. I’d say in all the ways that matters this was basically everybody BUT the interns fault.
mhh__ 6y ago
You can't really blame someone who is explicitly there to basically do what they're told and learn as much as they can for a short period of time.

kbd 6y ago

Good companies: “this was a failure of process, no individual is responsible”. SolarWinds: “the intern revealed our ‘solarwinds123’ password.”

jacquesm 6y ago
Yes, an individual was responsible: the CEO. Ultimately that job comes with the responsibility for everything happening inside the company and as a CEO you ensure that you have the proper culture and control mechanisms in place that that sort of thing does not happen, and that if it should happen that it gets detected.
- lawnchair_larry 6y ago
  By your standards, zero companies have a proper “culture” then. That’s just not how things work anywhere. You might think or even assume they should, but they don’t.

harryf 6y ago

The story line may be being twisted by CNN but the idea of a CEO blaming his staff is fundamentally wrong. The message should be “I created a culture which meant an intern could put the whole company at risk. This was my mistake”

loceng 6y ago
Ted Cruz claimed it was his children's fault for pressuring him into their trip, e.g. there is no lacking in incompetence and integrity in many systems, organizations, companies, communities, families. It starts with the individual which requires someone caring about how they look, about accountability, chain of command/responsibility. How do we solve this though? Who is the CEO responsible to - who do they care, if anyone, about how they appear to? How do we shift this behaviour for the examples of CEOs and politicians who maybe haven't been outed yet by such a security breach but are vulnerable to such lacking of a rigid structure of command/responsibility? Edit to add: I'm beginning to think HN is full of really lazy people.
- rootusrootus 6y ago
  > Edit to add: I'm beginning to think HN is full of really lazy people. Because of downvotes? That's probably just because you brought partisan politics into the discussion.
  - reagank 6y ago
    > Because of downvotes? That's probably just because you brought partisan politics into the discussion. Wut? Mentioning a politician’s name isn’t partisan politics. He said nothing about the politics of Cruz, he compared the Senator shifting blame onto his kids with this CEO blaming an intern. Don’t be like that. (Edited to correct typo)
    - loceng 6y ago
      Thank you for pointing this out - it's unfortunate that people can be immediately triggered to react after simply seeing a name that prevents/blocks them from continuing on to understand or critically think to understand what's actually said; it's definitely a sign of the times and state of our health and our thinking ability as a society.

switchstance 6y ago

Ultimately, the CEO should take responsibility for everything. Finger pointing is not a leadership skill.

duckmysick 6y ago
Somebody appointed the CEO, no?
- kowlo 6y ago
  Ah, yes. To blame someone, we must first invent the universe.
- tialaramex 6y ago
  Sure, SolarWinds is a corporation, it will have a Board of Directors representing the interests of its shareholders, it has at times been a private company (so its shareholders were some handful of private equity investment companies) and public (so they will have included funds and the great unwashed) but in either case the Directors are responsible. So in terms of specific people that's Bill Bock, the chairman of that board at the time. They did indeed replace their CEO. I would assume after agreeing to pay off the old one as this is the usual practice and, unlike some no-name intern, a handsomely compensated CEO can afford expensive lawyers if you try to kick them out uncompensated for incompetence based merely on the evidence that their inadequate oversight cost you billions of dollars.

lousken 6y ago

Intern, yea sure, the cheapest excuse ever

alfiedotwtf 6y ago

This is the equivalent of stock crashes being blamed on what the media has dubbed “fat fingers” of a trader

octetta 6y ago

What a good look... one of the people at the highest tier of a company blaming someone at the lowest tier. Are we all this blind?

loceng 6y ago
No, we're all mostly just distracted by so many problems and failures throughout our society. Someone commented the other day mentioning how it doesn't take a genius to see the West is degenerating. To me it's obvious it's the industrial complexes involved in regulatory capture - fuelled by bad actors, greed, targeting the richest nation in history - leading to bad policy for people, individuals, for too many decades now; including policy allowing the duopoly to exist, thrive, and the tightening and expanding conglomerates of a handful of mainstream media companies also then perpetuating the two main narratives of the duopoly, along with whatever other for-profit interests decide to influence/manipulate us through shallow, cheap advertising. Thank Goodness for the internet, technology, to allow anyone with a voice a better chance at gaining a following and a communication channel - so we can bypass the control of those systems to educate people, and share different narratives - truths.

carlisle_ 6y ago

Did these guys really think people would buy it when their excuse is a literal cliché? What’s next? Did a dog eat their homework?

mhh__ 6y ago

Always the honourable thing to do!

nfjrbrnnffk 6y ago

I worked at two big, well known software companies which had the same default password - company123, albeit for low security stuff, where you wanted wide access but a password was required for some reason.

chapium 6y ago
I find it interesting that some companies don't secure their test domains, but do rely on merging from test to prod.
- YarickR2 6y ago
  Well, a lot of companies are doing it in reverse , by not properly securing test sites, and bringing live sensitive data from prod to test . I don't know which is worse

llarsson 6y ago

The point of being a CEO is that you are ultimately responsible for what decisions are made. Even if decisions are delegated. Because guess what, you as CEO are responsible for delegating correctly. Failure to realize this is shameful.

iou 6y ago
+1
krapp 6y ago
There's a reason "shit rolls downhill" is a common idiom.
- sonotathrowaway 6y ago
  “The buck stops here” is also a well known saying. President Truman had a sign on his desk with the saying.
  - srswtf123 6y ago
    Alas, thats a relic of a bygone era. More recently, our politicians simply “don’t recall”, or worse directly lie to us. For C-level folks, its simpler. Take no responsibility ever, unless forced to by the courts. Even then, taking actual responsibility is so rare that I have no examples.
  - pkulak 6y ago
    Sadly, now the people with the most power also want the least responsibility (and the most money).
    - deleted 6y ago
      [deleted]
    - toss1 6y ago
      Yes, this describes nicely what sociopaths seek. That this is common in most organizations shows that most larger organizations end up being selection systems for filtering sociopathic to the top. It is not because their are deliberately designed this way, but because this is what sociopaths seek, and the organization fails to actively filter against it.

akadruid1 6y ago

I think they know this is a pretty desperate story. Some of the other coverage [1] suggest Solarwinds had so many different overlapping security failures they may never be able to attribute to a single cause. Still there's some interesting things that could help a (much smaller, less critical) software vendor decide where to focus their security efforts. Perhaps near the top of the list should be: 1. Who in your organisation has access to your build and distribution toolchain, and how secure are their credentials? 2. How good is your record keeping? Are all your builds traceable back to a specific revision in your source control, and are you keeping build logs somewhere they can't be tampered with? [1] https://www.bloomberg.com/opinion/articles/2021-02-26/deepening-mystery-and-fear-over-the-solarwinds-hack https://www.bloomberg.com/opinion/articles/2021-02-26/deepen...

ChrisMarshallNY 6y ago

Waitaminute...The password was “solarwinds123,” and the CEO is blaming an intern?

slickrick216 6y ago
Yeah because everyone knows you go all the way to 9 like you are slapping the piano keyboard in movie Big.

mkl95 6y ago

As an intern, you usually work for free or for subpar money, and expect to be taught the basics of the job and some good fundamentals. The fact this intern lacked the security knowledge any mom and pop company employee should have suggests the company simply wanted free labour from them.

TruthWillHurt 6y ago

And the same culture runs through Microsoft- Rep. Katie Porter: "You and your company were supposed to be preventing the Russians from reading DoD emails!" Microsoft President Brad Smith: "There is no indication, to my knowledge, that the DoD was attacked" That's accountability! dilligence! Microsoft at its best.

higeorge13 6y ago

This is so bad. A proper ceo, cto, executive, manager, leader does not point any fingers.

jorblumesea 6y ago

Would you blame the intern if they ruined the production DB after being given write access? I wouldn't. I'd blame senior technical leadership for putting processes into place that allow failures like that. Most especially, the CEO.

dmingod666 6y ago

If your companies existence relies on interns not making mistakes. Security breach is the smallest of your issues.. Company board is probably like, fingers crossed, hope the other interns are good... The last time an intern made such a disproportionate impact was in 1996.

jacquesm 6y ago

So, who is responsible for giving an intern - likely not even strictly under an employment contact with the company - access at that level. And who, in turn is responsible for failing in their oversight duty of this situation? And who in turn is responsible for a the department that this fell under where such oversight duty failures are possible? And so on. Before long, you're back at the CEO, but that time the question will stick. As a CEO you've got nobody to blame after your first 90 days of employment.

DoubleGlazing 6y ago
I don't think I have ever been more than one week in to a job before I was given some form of admin level rights. In small companies I can understand that. But I've had it also happen in government jobs, in big companies and startups that collect a lot of sensitive personal data. In one case I was with a startup that had names, addresses, DOBs, phone numbers and debit/card details in their DB. When they hired me they didn't ask for references or ID. I was given full admin rights to their Azure account on day one. If I were in charge I would at the very least want to validate a new employees ID before giving them any form of access to IT systems, and only elevate access privileges once they had an established track record within the company.
- jacquesm 6y ago
  As an intern? I would find it somewhat worrisome if an intern at some random company would be given access to mission critical systems and data within a week. That would definitely qualify as an oversight failure, and 'small' is a pretty relative affair, if the company has four people then I could see your point - maybe - but if it had 30 or more than it really isn't ok.

boh 6y ago

The fun thing about being in charge is that everything is your fault. Blaming the intern, the person least able to defend themselves, is ridiculous. If an intern can destroy your business, you're doing it wrong.

ed25519FUUU 6y ago
What’s more damaging is that a simple password is the only thing keeping your business from destruction, not to mention exposing all Americans with a security risk.
vardaro 6y ago
And its at the expense of a young professional just starting their career...

mensetmanusman 6y ago

That is a symptom of an organization that improperly allows interns to access important information.

hibbelig 6y ago

According to the article, SolarWinds doesn't seem to think that the password itself is a problem, only that it was leaked. And they "took it down", that sounds as if they deleted the leaked document. It doesn't sound as if they changed the password. Amazing.

reagank 6y ago
Seriously. Blaming the intern is a bad look, but here it reinforces the idea that they don’t even understand what was wrong. The fact that it leaked is only relevant is that it shows 1) how bad their password is and 2) how deficient their process is for dealing with (let’s face it, inevitable) leaks.

jpmattia 6y ago

> Current and former top executives at SolarWinds are blaming a company intern for a critical lapse in password security that apparently went undiagnosed for years. The SolarWinds execs have provided us with keen insight for the root cause of the SolarWinds attack, but the insight they conveyed is probably not the insight they intended to convey.

AdmiralAsshat 6y ago

> "They violated our password policies and they posted that password on an internal, on their own private Github account," Thompson said. So let's analyze the various scenarios under which the "intern" might have been responsible, and why each one is bullshit. 1) The intern exposed a company password on their own account. Counter: What kind of "password policies" allowed such a weak password in the first place? 2) The intern came up with the weak password themselves, thus violating "password policies" not just for secrecy but strength/security. This password was then used for several critical, production applications. Counter: Why was an intern in charge of deciding a password used for anything critical? 3) The intern came up with the weak password and exposed it, but it was only used for the intern's own corporate accounts (e.g. their Windows workstation). Counter: Why did an intern have a level of access such that their account being breached could lead to this level of compromise/exfiltration? Conclusion: There is no conceivable scenario under which this makes sense.

deleted 6y ago
[deleted]
laurent92 6y ago
Worst scenario: The intern is malevolent. Counter: Credentials should be given progressively as trust is built (and as lessons are learnt), and an intern can’t have access to production. I have an employee who I thought I could give more responsibilities, but he keeps not locking his computer when he walks away. He has very limited access to everything and it would impede his career if he didn’t also have the same attitude about other issues. (My question is, how do I make him diligent — It’s real potential wasted).
- ethanwillis 6y ago
  Have you explained this to him?
  - marcus_holmes 6y ago
    Further, have you really listened to his explanation of why he isn't bothered about locking his computer? It might be that in your organisation this is a cargo-cult security practice that he's not bothered by because he knows it's not an effective practice. Or it could be that he knows he doesn't have enough permissions to do any damage, so he doesn't bother locking his computer. Trusting him with a little responsibility might change that.

jmfldn 6y ago

It shouldn't be possible for an intern to leak a password, or if it is, the blast radius should be limited. In other words, it's the job of senior people to make this sort of thing hard or at least not super damaging when it occurs.

plandis 6y ago

Blaming the intern instead of root causing it as a systemic failure just leads me to wonder if they will actually take the correct steps to prevent this from happening in the future. Fwiw this is a good archive of mostly well done post mortems: https://github.com/danluu/post-mortems https://github.com/danluu/post-mortems

dartharva 6y ago

The article says the "former CEO" said that, not the current CEO.

dccoolgai 6y ago

Pretty terrible when you think of all the govt money this company took to provide these services. Likely charged at massively inflated rates for the benefit of their "expertise". Guarantee you that intern was probably billed out at 10-20x what it cost SolarWinds to employ them. Doesn't even pass a level 1 smell test.

toss1 6y ago

A chain is only as strong as its weakest link. This means that the CEO, CTO, and all management down the chain failed to create any kind of robustly secure system. Their entire system was never more than a single password leak away from complete failure for the security of the entire USG. With literally thousands of opportunities for such leaks per day, it is inevitable. So, a system must be designed such that when the inevitable occurs, the consequences are minimal. The most basic of fail-safe designs. These people, from the CEO on down, utterly failed to do this. Yet they made millions from false claims to have succeeded in creating a secure system, when they created a highway for global espionage. They literally could not have provided our adversaries a better avenue for espionage if they tried. The free world would literally be better off if all of that company and it's management had never existed. Meanwhile, as a small manufacturer who has some govt-related work, the blizzard of new security certification requirements for Controlled Militarily Critical Technical Data and the like coming down the pike is like nothing I've ever seen before. Sure, some of it is due to increased threats, but much of it is definitely because of a*$holes like those CxOs who so utterly and deliberately failed in their most basic duties, and should never work in the industry again.

citizenpaul 6y ago

Cool so its totally not his fault then. If only there was someone in charge that could handle this situation.

hedora 6y ago

So, if you want to compromise the security of 100,000’s of IT departments, get an internship at SolarWinds?

loveistheanswer 6y ago
This is an interesting way of looking at it. Undoubtedly there must be nation state actors using such attack vectors

sjreese 6y ago

The password is invalid on its face Eg - Maximum number of days a password may be used. Minimum number of days allowed between password changes. Number of days warning given before a password expires. Says that you have to use any password within 0+x days - otherwise it would have expired - the older the posting the more unlikely it would be valid. Would you risk detection with a password without mix case and consecutive numbers. Most would avoid traps or suspect a trap

bawolff 6y ago
Password expiries are not really considered good practise anymore.

ineedasername 6y ago

That really doesn't make them look any better. If a single intern's password mishap can breach a security company's systems on this level, they've lost the fight long before this incident.

zepto 6y ago

This just means that the CEO doesn’t understand security risks and must be fired. Any company that continues to rely on Solarwinds after this, if the CEO is not fired, is accepting that their security is only as good as a SolarWinds intern.

itsdrewmiller 6y ago
He already resigned - the actual CNN headline is "Former" CEO.
- deleted 6y ago
  [deleted]

womitt 6y ago

This is the weakest leadership reaction possible in a giant fuckup scenario like this

SecurityLagoon 6y ago

Wow, way to remove your product from consideration ever again. Security breaches happen - fix up the issues and show people your reform. Blaming the intern just makes you look like an ass whos company I never want to do business with again.

stanrivers 6y ago

If you have a system setup where a single intern can destroy your company, you are doing something wrong. Let's even say the intern was malicious and trying to do harm... that is still your fault. One person in the lowest position in the company can break everything? Again, you are setup incorrectly.

aklemm 6y ago

Somebody hasn’t read Extreme Ownership.

pjmorris 6y ago

"A loss of X dollars is always the responsibility of an executive whose financial responsibility exceeds X dollars." - Gerald Weinberg's 'First Principle of Financial Management' and 'Second Rule of Failure Prevention' [1] [1] 'First-Order Measurement', Quality Software Management, Volume 2, Gerald Weinberg, Dorset House Publishing, 1993

vinay_ys 6y ago

I prefer to remember only one long passphrase and enter it in only one place – my password manager. And then I trust my password manager to generate, store and submit the right password to the right portal and only upon my consent. This is vastly better than I remembering and entering the passwords myself. But it is still not perfect. I would rather all websites and apps universally moved to a standardized machine friendly authentication API and then I have a authentication user-agent (my password manager) do the actual authentication.

gizmo385 6y ago
Wouldn't that be introducing a globalized single point of failure?
- vinay_ys 6y ago
  The password managers can have many different strategies for keeping their seed secret – it could be local hardware backed or cloud based but locked to a few of the devices you own etc.
jtsiskin 6y ago
This exists - webauthn!

ldbooth 6y ago

Someone didn't see Spaceballs. "12345? That's amazing I've got the same combination on my luggage!"

quallzone 6y ago

There is a stark contrast in the former SolarWinds CEO response versus Berkshire Hathaway CEO Warren Buffet's attitude. I noticed an interesting statement in Buffet's 2020 earnings report: "The final component in our GAAP figure – that ugly $11 billion write-down – is almost entirely the quantification of a mistake I made in 2016. That year, Berkshire purchased Precision Castparts (“PCC”), and I paid too much for the company. No one misled me in any way – I was simply too optimistic about PCC’s normalized profit potential. Last year, my miscalculation was laid bare by adverse developments throughout the aerospace industry, PCC’s most important source of customers." . . . "I was wrong, however, in judging the average amount of future earnings and, consequently, wrong in my calculation of the proper price to pay for the business. PCC is far from my first error of that sort. But it’s a big one." WOW. I rarely see that level of accountability at senior organizational levels.

jtsiskin 6y ago
SolarWinds: “My intern had prepared a report on PCC’s profit potential, which I used to decide the purchasing price. It turns out the intern had not followed our financial analysis framework correctly. This mistake therefore lies squarely on the interns shoulders, please talk to them if you have any complaints. My company and I should not be blamed.”
throwaway888abc 6y ago
There is a man and Man.

908087 6y ago

What's amazing is that they thought this claim would make them look better.

Threeve303 6y ago

The same intern worked for Volkswagen as a developer and is solely responsible for the whole miles per gallon debacle.

andy_ppp 6y ago

It’s extremely weak leadership to blame the intern... makes me think it’s probably the CEO’s fault with such a lack of character.

jtdev 6y ago

This is modern American corporate culture after all. Lie, cheat, steal, win at all costs and if you lose blame someone at the bottom of the ladder.

itsdrewmiller 6y ago

@dang any chance this could be updated to the correct headline? So many comments in this thread calling for the firing of a CEO who already resigned.

slickrick216 6y ago
Dig him up and fire him again.
sjg007 6y ago
New CEO should be fired too. They are failing at crisis management 101.

colechristensen 6y ago

This is great and very publicly (unintentionally) shows exactly why SolarWinds got hacked. A CEO scapegoating an intern is what you would expect from a company with very deep systematic problems which led to such an embarrassing failure. If your least experienced employee can accidentally topple the entire company, it is the entire company at fault. There are cases where individuals can undermine a whole organization and be at fault, but that requires sophistication and corruption which go above and beyond solid safeguards.

cgb223 6y ago

Lol SolarWinds is never going to be able to find an intern again. Who would want to work for a company that throws them under the bus for the largest hack in recent history

gameswithgo 6y ago

ohhh really bad move blaming the person instead of the process, CEO.

sumoboy 6y ago

What's next, the janitor was responsible for the companies poor quarterly results?

say_it_as_it_is 6y ago

This is so outrageous a claim that it warrants the technology community responding to it. Where is Bruce Schneier?

adrianmonk 6y ago

Here's more context about what the password was for. https://twitter.com/vinodsparrow/status/1338431183588188160 https://twitter.com/vinodsparrow/status/1338431183588188160 This tweet has a screenshot of an email from Vinoth Kumar (named in the CNN article) to SolarWinds saying: Hi Team, I have found a public Github repo which is leaking ftp credential belongs to SolarWinds. Repo URL: https://github.com/ Downloads Url: http://downloads.solarwinds.com FTP Url: ftp://solarwinds.upload.akamai.com Username: Password: POC: http://downloads.solarwinds.com/test.txt I was able to upload a test POC. Via this any hacker could upload malicious exe and update it with release SolarWinds product. (The tweet blanks out some things including part of the github URL, the username, and the password.) My thoughts: (1) I assume this means when it comes to technical measures to prevent a weak password, SolarWinds would have to rely on Akamai. (2) The researcher was able to upload to the root directory of downloads.solarwinds.com. As an educated guess, this may have been a shared account and many people knew this password. When many people share an account, they tend to choose passwords that are easy to convey to someone else. If so, the intern probably didn't create the password and was only responsible for leaking it.

jtsiskin 6y ago
Also note from that thread: SolarWinds gave him $0 for finding and reporting this. That in itself is completely irresponsible and a reflection of the companies systemic issues, which can’t be blamed on one intern.

aszantu 6y ago

lol if an intern can leak critical passwords you prolly got a ton more problems than just the intern leaking critical passwords

corobo 6y ago

We still doing the blame the intern thing eh?

jzer0cool 6y ago

Let us step back one. What policy in place which allows such a password in the first place?

endymi0n 6y ago

Oh those interns... Few years ago, police launched an investigation into a European dating company. They were prohibited by court order to destroy any evidence. Turns out in an unfortunate and unforeseen turn of events, an intern wiped their production Hadoop cluster just a week later with the backups having some issues. He was fired pretty quickly, but I heard he hasn‘t been too bad off since... Don‘t hire any interns, they can do quite a lot of damage!

sjg007 6y ago

This CEO needs to go, he apparently has zero experience in crisis management. He has signaled to all of his employees that yes he will throw them under the bus. He has signaled to his customers that there is not enough oversight on operations. As such he has now lost all credibility. It's a shame really, to blow a totally manageable crisis where solar winds could emerge stronger as a result.

zeckalpha 6y ago

Next week: SolarWinds board blames CEO for password leak

skynet-9000 6y ago

We always use interns to do security critical work -- the less experience, the better. We also have exacting security standards, and a Critical Password Policy, since there are no non-critical passwords. Critical Password Policy Rule 1: All critical passwords must contain the company name, lower case, with no special characters or spaces. Rule 2: All critical passwords must have between 3 and 1024 randomly selected sequential integers appended, each of no less than 1 and no greater than 3. When you are creating your critical passwords, it is critical that you follow the rules in the exact sequence as stated above, and that you do not introduce any external sources of randomness or entropy. Failing to follow this Critical Password Policy may result in your dismissal and later blame before a Congressional committee.

camjohnson26 6y ago
That’s all well and good but you should really make sure to disable any kind of TLS on these connections so the security team can monitor for breaches. Any ops team worth its salt will make sure to secure all endpoints by physical device address and remove single points of access like gateways to widen the attack surface and ensure attackers can’t focus on one location. Store any critical data in a basic S3 bucket and remove any firewalls on production databases so that anyone on the team can fix a problem or restore a backup if needed. And you should really disable most logging, it’s a performance bottleneck and can tell attackers what’s happening in the system. I’ll type these notes up in a PowerPoint and we can deliver a presentation to key stakeholders.

jVinc 6y ago

> Confronted by Rep. Rashida Tlaib, former SolarWinds CEO Kevin Thompson said the password issue was "a mistake that an intern made." Imagine a bank where the CEO says "the problem with all the money going missing was that an intern dropped the keys to the security vault and we had told the guards to never question anyone who had the keys, just let them take whatever they want without question". Seems to me that you can't pin anything on the intern. The problem was the extreme lack of security practices at the company, which ultimately falls on the CEO, who's trying to blame his own incompetence on a intern.

j_barbossa 6y ago

Maybe it was the intern's fault to set a weak password but it is the CEOs fault for setting up an organization where things like these can slip through security review or monitoring.

awiesenhofer 6y ago

While this of course is pure BS, with the way they handled all this so far my trust in them has been lost quite a while ago. And I guess I am not the only one. My question though is what alternatives are people moving to now? Any experiences?

lr1970 6y ago

The fact that the SolarWinds CEO dared to blame the whole security breach on a single intern is a damning testament to the rotten security culture at SolarWinds. Would you trust a company where a single intern can compromise security of all of its clients? The fact that the CEO does not understand how damning his admission is, makes the whole situation hopeless.

IronWolve 6y ago

Remember the old nortel network equipment default passwords like admin/admin that telecoms didnt change? Good times.

dang 6y ago

Related ongoing thread: SolarWinds CEO blames intern for password leak - https://news.ycombinator.com/item?id=26284782 https://news.ycombinator.com/item?id=26284782

somerandomness 6y ago

They really should have tried harder to find a more convincing scapegoat.

rayiner 6y ago

The buck stops with intern. That’s why the intern gets paid like 20 million dollars a year.

andred14 6y ago

Come on, who believes anything CNN says anymore after the hundreds of lies in 2020?

abvdasker 6y ago

The only thing the CEO of SolarWinds is telling us by saying this is that he's a terrible leader. We don't even know that the hackers used the exposed password at this point. But even if they did, this is an obvious process failure and by blaming an individual intern, Sudhakar Ramakrishna is loudly demonstrating his own hopeless incompetence.

xyst 6y ago

poor intern is getting thrown under the bus

imron 6y ago

It was either an intern or a third-party contractor, and they went with intern.

mickael-kerjean 6y ago

A few years back, I caught a security consultant from Thales pushing his credentials on GitHub as he was working on what seemed to be a personal project during office hours and trying to get throught the corporate proxy... Considering Thales creates the most sensitive system, I completely lost trust in companies who claim to make secure products

spacecowboy 6y ago

This environment sounds like a case of no accountability and negligence. If a intern can make this type of decision without review and accountability, what other decisions are being made under the same environment of no review and accountability within the company?

nibsfive 6y ago

As a 20 person startup we have a KMS with only two authorized users. Seems like SolarWinds tried pretty hard to be hacked.

mbrodersen 6y ago

CEO’s are typically quick to claim large bonuses “because of the huge responsibility I have” and even quicker to not take responsibility (and keep their bonuses) when things go wrong.

toss1 6y ago

tl;dr: If this CEO's security systems were so weak that a single password leak by an intern can compromise his entire company, the entire US Govt and many others, and much of the Fortune500, then his security systems suck - all the way up to insane values of suckage. The fact that he is blaming the intern shows that he knows his systems suck and is attempting to divert. The entire management chain from CEO down to the intern's manager is at best too incompetent to work in the industry, and more accurately, a major threat to national security. Anyone who trusts these people to do so much as run out for a pizza is a fool.

retox 6y ago

Personally, I am more comfortable socializing benefits and privatizing (my own) mistakes. Which might not be the best way of phrasing it, but if someone else made a mistake, whether it's widely known or just something I've been privy to, I'm happier saying that "there was a mistake" rather than pointing the finger. However if I made the mistake, I'm content to say it was something I personally did wrong, it either case I will follow up with the steps we can take to prevent it happening again. Conversely, if I delivered something of noteworthy benefit for the project I will talk about it in terms of how 'we' achieved the success, and if it was someone else I will call them out by name and congratulate them on the solution. This ethos is probably career limiting as a dev, but it does help me sleep soundly.

xtat 6y ago

what a shitshow