12 ms·
For me a company of their size and, what I would expect, maturity, this new announcement does not satisfy me or provide me much assurance. Consequently I am sti
by LilBytes 5y ago
For me a company of their size and, what I would expect, maturity, this new announcement does not satisfy me or provide me much assurance. Consequently I am still happy I have been recommending people against Ubiquiti since the original announcement from Krebs.
* Why was it so easy for a lead engineer to get access to a root AWS user without anyone else being notified? I.e. AWS GuardDuty provides FREE alerting for when an AWS root IAM account is logged in or used, this account should be under lock and key and when used, confirmed and audited by relevant persons or teams.
Start edit/
* Furthermore on the root account being easily accessed, the root account in the companies I've worked at had MFA enabled, and the QR code is locked in a safe only accessible by two people agreeing it needs to be accessed in a break glass situation, where warranted.
/End edit
* Why was he also able to delete critical CloudTrail logs and reduce their retention to 1 day? I.e. These logs should be in a S3 bucket or other environment where such changes cannot be made. Alternatively, they should be shipped to a redundant service that manages this risk to prevent data deletion
* Why did Ubiquti not announce they were compromised sooner? The hack started in early December, Ubiquiti noticed the compromise on Dec. 28. Ubiquiti told the market on January 11th. Is that a satisfactory turn around? Giving them some credit for the XMas break I'll say this partially understandable.
All the AWS configuration I'm speaking of above, I would describe as Security 101.
Most of these settings can be set and managed from AWS Organisations for free, and backed up with alarming and alerts for Guard Duty trivially. That a company of Ubiquiti's size and maturity had such basic risks not managed is still a concern.
I understand AWS Organisations can be difficult to set up for legacy AWS accounts, but even with that said, setting the alarms and monitoring up that would help manage the risk associated to the questions above is not difficult and should have been in place.
That Ubiqitui would only find relief ultimately from the developers poor OpSec rather than Ubiquiti's own security policies and procedures provides a commending perspective of their internal security posture.
- techdragon 5y agoWhile I agree this is demonstrative of overall less than adequate security practices, I’m unsurprised that a company that started with making hardware and only later added cloud functionality beyond a website/store was not initially setup as you described. What you described is the sort of setup I’ve been involved in the transition from what Ubiquiti has and what you just described, and it can be quite a lot of work to make that transition, if the industry involved didn’t have regulatory risk management drives, it’s entirely possible that it would have been considered more expensive than it was worth. Ubiquiti now has public evidence their security posture is inadequate and there will be pressure for them to demonstrate they have changed this situation. It’s very difficult to completely prevent malicious actors engaging in deliberate efforts to infiltrate and plan actions like this once they have any measure of access and trust. What matters now is how they respond to it.
- hvidgaard 5y agoIt is next to impossible to prevent malicious actions from insiders. But it is a well understood risk and there are plenty of canaries and deterrents possible that it should stop all but the most dedicated adversaries. Those same measures will notify multiple other people when it happens so you can act accordingly.
- qsi 5y agoI agree it reflects poorly on the organization, but the previous story on HN [1] mentioned that the malicious insider had scared the CEO into giving access by manufacturing some sort of security incident. Had this not happened, he would not have been able to pull it off IIRC. That would also point to the CEO being too powerful when it comes to security, which is also a knock against the organization, but in a different way. [1] https://news.ycombinator.com/item?id=29411775 https://news.ycombinator.com/item?id=29411775
- jusssi 5y ago> Most of these settings can be set and managed from AWS Organisations for free, and backed up with alarming and alerts for Guard Duty trivially. Makes me wonder, why are they even settings? Why aren't they just always on?
- jiggawatts 5y agoI've found that every one of these security incidents always has someone come in and say: "Why didn't they just ..." Where the problem is that the "..." is the subset of non-default security configurations that would have stopped this specific insider(!) attack. They never mentioned that: - You can't predict which attack you'll be hit with, so you have to implement every non-default / optional security setting or feature in order to be "protected". This is a metric ass-ton of work, typically on the order of multiple man years of up front effort, and then with ongoing multiple FTEs required (e.g.: for separation of roles). - While you're not actually under insider attack, this provides zero business benefit that can be measured in dollars. - The people implementing the protection against insider attack are the same people that the protection would typically be designed to stop. No one person can be trusted to do this. Not even any one team. Just organising the Byzantine security required for this is a project manager's nightmare. - Insider attacks by senior technical staff are stupidly difficult to protect against. I've never seen any org that could survive an intelligent, motivated attacker that already has significant admin rights. Even the FAANGS would take a significant "hit" in this scenario. Ubiquiti is nowhere near their scale or capabilities in this space. - All that stuff described like "QR codes in a safe" are hilarious to me. Every time I've tried to implement even 1/10th of that, everyone who ought to be responsible for the secure handling of this kind of key material ran away screaming from any personal responsibility. Literally nobody ever wanted to deal with anything even vaguely similar. - Many of complex delegation systems and IAM technologies suffer from permission-role inversion. Often the CEO/CIO/CTO has virtually no access to the technology stack, but the junior intern from the subcontractor has the keys to the castle and could delete the whole org for shits and giggles. But, you see, he's not "allowed to". But he has the physical permissions. That he's not supposed to use. Because he was told to. You see? - Now you're going to say that the senior staff should have the admin rights, and delegate the appropriate rights to the junior staff. Oh, you sweet summer child. That requires responsibility! (see above). It also requires that they learn, manage, and monitor something as technically complex as AWS IAM. But you see, senior people are busy people. They're busy with meetings. With memos. And more importantly, they're busy playing politics and jostling for the next promotion. The fiddly security rule stuff they just delegate to the juniors. That's the ticket to the next pay grade!
- _pplp 5y agoHoo boy, this is gonna be a fun one. For reference, I spent a year (mid-2018 to mid-2019) running the UniFi Network team and worked with Nick during that time. > * Why was it so easy for a lead engineer to get access to a root AWS user without anyone else being notified? I.e. AWS GuardDuty provides FREE alerting for when an AWS root IAM account is logged in or used, this account should be under lock and key and when used, confirmed and audited by relevant persons or teams. The "Cloud Lead" that Nick took over from gave zero fucks. He ran all the AWS stuff for Ubiquiti under his personal AWS account. Nick came in and started putting "proper" AWS structure and security in place, primarily by scaring Robert (the CEO) into giving him the keys to the castle (my own personal opinion of Robert is... not the greatest). One thing to understand about Ubiquiti (at least during those times) is that the company had zero C-level execs. There was Robert.... and then nobody knows. I asked repeatedly why we didn't have a CTO, or a COO, or a CFO, or CMO or ANYTHING and I got nothing but shrugs and "idunno" as a response for the whole year I was there. So when Nick came in, a very... let's just say "forceful" personality, he immediately won over Robert and ended up with carte blanche over pretty much all of Ubiquiti's cloud accounts. Which were basically... everything. All the UniFi Network services, UniFi Protect services, you name it. If it was connected to the cloud in any way, Nick had access to it. So why wasn't anybody else notified? Simple. Because he was basically "god". If anybody was gonna be notified, it would've been Nick. He was the top of the totem pole company-wide when it came to AWS. Also, for some perspective, at that time Ubiquiti kept all the hardware signing keys in a private GitHub repo that every employee had read access to. And they were in plain-text. So... yeah. > * Furthermore on the root account being easily accessed, the root account in the companies I've worked at had MFA enabled, and the QR code is locked in a safe only accessible by two people agreeing it needs to be accessed in a break glass situation, where warranted. See above for the quality of security processes and practices this company had in place. > * Why was he also able to delete critical CloudTrail logs and reduce their retention to 1 day? I.e. These logs should be in a S3 bucket or other environment where such changes cannot be made. Alternatively, they should be shipped to a redundant service that manages this risk to prevent data deletion See above. (re: "god") Nick answered only to Robert. And he'd already successfully hoodwinked him. He could do whatever he wanted. Eventually he fell from Robert's good graces, but seeing as Ubiquiti as a company didn't really have a ton of checks and balances, he kept his god-level access far longer than he should've. > * Why did Ubiquti not announce they were compromised sooner? The hack started in early December, Ubiquiti noticed the compromise on Dec. 28. Ubiquiti told the market on January 11th. Is that a satisfactory turn around? Giving them some credit for the XMas break I'll say this partially understandable. Simple. Fear of share price falling. I was constantly given this as a reason we couldn't be transparent. Not by Robert, nor where he could hear. But it was pretty much well known that the company kept shit quiet for fear of the share price dipping. > All the AWS configuration I'm speaking of above, I would describe as Security 101. To keep with the metaphor, Ubiquiti couldn't even get Pre-school level security in place, much less 101. I have no idea how something even more massive hasn't happened yet. Must be dumb luck. Speaking of, by the time I left the company, the team that was handling the door entry-way systems (UniFi "Access" I guess) had been caught with numerous security issues, not the least of which was logging user credentials in plain text (not just storing, but logging, in response to authentication events). They were also based in China and subject to Chinese laws around government access, so take that how you will. And that doesn't really even cover most of it. That year took a toll on my physical, mental, and emotional health, not to mention put a crazy strain on my marriage. I'd rather honestly forget it, but the schadenfreude of what's going on is too delicious to ignore.
- unionpivo 5y agoYou assume anyone except the hacker knew anything about AWS. Our firm subcontracts to other firms. I am among other thing resident Sysadmin which means I get to do everything related to AWS. Out of 6 contracts we had, where firm had their project on aws I got root credentials in 4 cases. out of 4 i only managed to convince 2 to take their account security more seriously and lock things down. What I am trying to say is that what happened at Ubiquiti is far more comon than people realize. Because when all is said and done you still need someone who thinks like sysadmin.
- LilBytes 5y agoFrankly after reading the other comments in this chain, as well as experience in other businesses and industries. You're likely correct. I'm giving too many companies far too much credit.
- ex_ubiquiti 5y ago> You assume anyone except the hacker knew anything about AWS. There were a lot of smart and experienced people at Ubiquiti when I worked there. Nick Sharp manipulated his way into total control over everything and wouldn't let anyone outside of his isolated team touch it. Nick was hired out of his job at Amazon because he was supposed to be the AWS expert. He used that to lock out anyone but himself and a trusted team. It didn't matter that others knew better. Nick controlled it.
- _pplp 5y ago> Nick was hired out of his job at Amazon because he was supposed to be the AWS expert. This always cracked me up. From what I can tell, he was a mid level dev on the Alexa web api team. He knew AWS sure, but he did not have the cred at all to justify the position and responsibility he was given at Ubiquiti.
- gonzo 5y agoThis means the fall in Ubiquiti’s share price was entirely deserved. Management abdicated its responsibility.
- lars_francke 5y ago> * Why was it so easy for a lead engineer to get access to a root AWS user without anyone else being notified? I.e. AWS GuardDuty provides FREE alerting for when an AWS root IAM account is logged in or used, this account should be under lock and key and when used, confirmed and audited by relevant persons or teams. I had never heard of GuardDuty so I got curious. It doesn't seem like there's a free tier available for it: https://aws.amazon.com/guardduty/pricing/ https://aws.amazon.com/guardduty/pricing/ In fact at larger scales it seems pretty expensive even.
- LilBytes 5y agoYou're absolutely right. I had not noticed. I'd say this was to do with GuardDuty isn't free but has consistently been such a small line item on my AWS bills that I thought it was free. Thank you for the correction. All the same the point I was trying to drive is when the service is incredibly effective, both in practice and cost, it should in nearly all instances be implemented. Edit: I wanted to amend my statement but the window to make those changes has passed.
- lars_francke 5y agoThanks. This seems like a service that _should_ be free and enabled by default at least with a subset of options for the root account.
- dmos62 5y agoThose are good points, but Ubiquiti lost me at centralized, private management of my network devices. Someone might, you know, abuse that; plus, I'm not in control of what's on my device.
- deadbunny 5y agoYou don't have to use their cloud services, everything can be managed 100% locally. With that said I'm moving away from Ubiquity after years of broken pointless updates, years long outstanding bugs, and after this thread obvious massive lacking security.
- salmo 5y agoThey’ve pretty much broken that in practice since the last major firmware update. I just only turn my controller on when I’m working now and occasionally have to re-adopt. And alternatives likely have similar issues. It’s what naturally occurs when small businesses get large, and especially when companies go from embedded to SaaS development.
- nonameiguess 5y agoIt's mostly meant for enterprise use cases. Think something like hotel WiFi. Which is easier to secure? The cloud accounts, or physical access to the several hundred devices in the hotel? Even in private buildings, if you're deploying thousands of access points, you may not trust the minimum wage third party contractors doing all the installations and don't want on-device configuration to be possible.
- dmos62 5y agoThat's a good point.
- salmo 5y agoThe problem is that they set a bar there. There is a gulf between consumer and enterprise APs that is full of only Unifi-esque gear. “Cloud management” is an OOB requirement and primary use case for SOHO gear these days. It’s a fleet management issue, just wish it could be done with proper config management and not phoning home. But they’re selling to small shops doing IT for 10s of law firms or doctor’s offices. I’d never use Ubiquiti’s switches, routers, etc. There are great alternatives there. But when I went to replace my APs earlier this year I still could not find anything less shady that still did what I needed. Is there an alternative that does PoE w/ multiple APs that hand-off well? And decent hardware… TP-Link has a similar offering but with similar problems. OpenWRT on [consumer gear] is not an answer here. More effort goes into the plastic than the hardware. Never dealing with cheap NICs, bad SoCs, inadequate memory, garbage drivers, etc. again.
- sbuk 5y agoSee https://news.ycombinator.com/item?id=29412262 https://news.ycombinator.com/item?id=29412262 Obviously we have to take the linked comment at face value.
- tlogan 5y agoYes and all the things. But Nick was the guy responsible for all these things. You really cannot have good security if the goal of your “cloud lead” is to ransomware you. So yes - they did not have good security so they hired him to fix it :)
- bigyellow 5y ago> Why was it so easy for a lead engineer to get access to a root AWS user without anyone else being notified? Oh my sweet summer child. You haven't worked in large organizations with thousands of employees before, have you? Surely not if you think this is "Security 101".
- LilBytes 5y agoUp to and including companies of 100,000 staff, to my current employer of < 100. At around ~1,000 staff though you quickly lose "insight" into platform security as it's delegated out to sub-orgs within a larger organisation. Though I've not worked at any FAANG companies where I'd hope security is... erm, of a certain 'maturity'. E.g. Google and BeyondCorp. But I've no desire to rehash leet code exams or get back in that space so likely will never go there :).
- chaostheory 5y agoI've written this before, but someone on HN posted a link on Ubiquiti Glassdoor reviews where multiple people mentioned that the company is hostile to automated testing. Have things changed? If not, why, just why?
- deleted 5y ago[deleted]