9 ms·
Ubiquiti accused of covering up ‘catastrophic’ breach– and it’s not denying it
- akersten 6y agoWhat I want to know is the likelihood that customer devices were compromised. Frankly, I could care less about whether my data/password/whatever were lost in the breach. Happens all the time, who cares. The problem with the Ubiquiti case is that this compromise has the potential to have poisoned official firmwares, with the silly source code/customer data bitcoin ransom as a smokescreen. Remember, there were VMs spun up in their cloud that they couldn't account for. That doesn't look like someone just tooling around to me, that looks like someone recreating a build process. Is anyone monitoring their Ubiquiti devices in their homelabs to watch for any suspicious traffic? Being part of a nation-state botnet is not a very comforting idea.
- inetknght 6y ago> Is anyone monitoring their Ubiquiti devices in their homelabs to watch for any suspicious traffic? With modern software good luck filtering suspicious noise from signal.
- luma 6y agoThe problem is that UBNT now requires you to cloud-enable many of their modern devices as part of the setup process. You can later disable that after setup, but I think it's safe to assume most people don't know that (they certainly don't go out of their way to tell you that), and even fewer people will go through the effort. So, this hack also implies that a substantial number of devices out there now have their remote admin credentials leaked to an unknown, presumed malicious, third party.
- fossuser 6y agoIf you have 2FA enabled are you protected from this? I suppose an automatic firmware update with bad code in it could still hit me?
- cj 6y ago> If you have 2FA enabled are you protected from this? Very hard to say based on the limited information released. Part of what was released mentions "SSO cookie secrets" and "remote access". Best to assume 2FA isn't protecting you if the attackers were possibly able to forge cookies.
- 1cvmask 6y agoThe admin panel does not have 2fa support.
- cj 6y agoWhen I managed a cloud key a couple years ago, login happened via SSO through ubnt, which has 2fa. (But yes, you can also login directly on the cloudkey locally which doesn't support 2fa) The concern is whether their remote login / SSO authentication via cloudkey was compromised.
- tnorthcutt 6y ago...this seems completely and totally insane
- teilo 6y agoYes it does, both on the local device, and on the Unifi cloud service. This is true as of recent firmware versions. But 2FA in this case is useless. The hackers owned the UBNT servers. If they harvested user logins, then one can assume that they also harvested the corresponding TOTP secrets. To make matters worse: When you setup a Gen2, Gen2+, UDM, or UDM-Pro, you are required to login to the UBNT cloud service to activate your device. Once you do this, your device now has an "Owner" account cannot be disabled. If you have enabled 2FA on your Unifi account, then you can now login to the device locally using your Unifi credentials + 2FA, with a nice false sense of security. So in other words, if your account creds and TOTP secret got leaked, and you did not change them, anyone who gains access to your local device's login page can still own you. This remains true even if you disable Remote Access. (And in fact, as of recent firmwares, the ONLY way to disable remote access is through the Unifi cloud panel. You can't even do it locally.) It will still call out to Unifi's cloud service and authenticate you. There's no way to turn off this SSO account. The best you can do is block all access to Unifi at the firewall, and download all firmware updates manually. And good luck with that if you are using a Unifi device as your gateway.
- WWLink 6y agoThere is no way to disable cloud login to the unifi dream machine's web console. Not the network controller, but the web console itself. Like, the screen you get if you hit up your UDM's public IP, or go to the network controller (private IP) and open settings -> controller settings -> dream machine
- lsllc 6y agoI have 2 cloud keys that I only use for UniFi Protect -- mostly because they abandoned the UniFi Video that I could self host; my Network controllers are all containers that are mine (I do have 2FA set up for the UniFi Protect). I don't see how to remove the remote login from the Cloud Key, I tried creating a new Super-admin, but there's seemingly no way to remove the user. I guess I'll just have to factory reset them and start over with all new credentials.
- count 6y agoNot to discount the threat, but 'vms spun up in their cloud' is ...exactly what coinminining attackers do. It's so frequent, it's even called out as a 'thing to watch your bill to notice' in amazon security talks/etc. Gain access, launch N VM's, where N is as many as your credential and the cloud account limits in place allow, mine coins 'for free', and move on.
- ericbarrett 6y agoI agree, but APTs have gotten clever enough that I could also see them doing this as a distraction. Like burgling somebody's house to find blackmail material, but dressing it up as a common "smash & grab" theft so as not to raise their deeper suspicions.
- akersten 6y agoThat's fair - and I hope that is the case that it's just mining. But I also hope Ubiquiti performs a full audit (ideally by an expert 3rd party) to ensure the integrity of their build systems and reassure customers that the worst hasn't happened. Because it's been radio silence on that part, and that they haven't even announced they're planning on it makes it seem like a blindspot.
- carlisle_ 6y ago>That doesn't look like someone just tooling around to me, that looks like someone recreating a build process. This is quite the leap in logic.
- akersten 6y agoIt's speculation as I stated. But it's really not a "leap". The whistleblower's claim: > They were able to get cryptographic secrets for single sign-on cookies and remote access, full source code control contents, and signing keys exfiltration And what Ubiquiti has further stated: > we have well-developed evidence that the perpetrator is an individual with intricate knowledge of our cloud infrastructure. So someone with intricate knowledge of their infrastructure and access to their source code + signing keys, is doing what exactly with those VMs? I'm not trying to 'prove' that the firmware is compromised. We can't do that with the information we have. I'm just highlighting strong evidence that the possibility needs to be thoroughly investigated.
- croutonwagon 6y agoWhile my controller is on 6.0.43 and APs (uap-ac-lite and uap-ac-hd and uap-pro gen1) are on the corresponding firmware. I believe this controller version is December 2020 and predates the January issue disclosure date(though possibly not the incident). I have suricata taps on my net that are clean. Aps and controller are predictably trying to get to their telemetry call homes but nothing more. That said my Ubnt controller and aps have almost no internet access and haven’t for a long time. I only enable it when performing upgrades and disable it. I’ll pull the firewall logs and see specifically the IPs they are attempting to hit.
- nkotov 6y agoI'm pretty sad about this because I personally liked Ubiquiti products and the quality of them.
- TheCapn 6y agoAgreed. I often tell people when they ask for company/product recommendations that I can only really recommend somebody after they've screwed up. Any company can deliver product without hassle, but the ones you want to trust are the ones who handle mistakes properly. In this case, I just lost all my confidence in Ubiquiti because they've done the worst thing when it comes to maintaining my trust, lie and deceive.
- adkadskhj 6y agoYea. Sucks too because yes, this is gross but their _hardware_ might still be good.. if only you could easily run it at home without their cloud. But it seems that's been getting more and more difficult. Yeesh.
- SparkyMcUnicorn 6y agoFor most of their devices you can run 100% locally. That said, my UAP-AC-PRO at home has been on the fritz lately and I'm a bit lost on what I should upgrade to. This news comes on top of a lot of issues I've had with them lately, mostly around firmware updates and broken promises.
- xienze 6y ago> if only you could easily run it at home without their cloud. Am I missing something with all these comments talking about “needing” a cloud account? I have two of their APs and a USG, and I don’t do anything in the cloud. I have a Cloud Key on my network, but that was just for convenience. I could’ve run their management software locally.
- gh123man 6y agoI'm in the same boat. I bought my Edge Router Lite probably 8 years ago and it's easily been the best $99 I ever spent on networking gear and it's still working amazingly well. Before the Edge Router I couldn't imagine having a home network that didn't need a router reboot once/week. I've grown into many of the ER features over the years. Also fun fact - the storage is literally a USB flash drive plugged into a standard USB-A port under the cover! (which is totally upgradable). I will forever be impressed with this product.
- annoyingnoob 6y agoI'm hoping this spurs Ubiquiti into reconsidering its cloud-only push. I know I will be looking beyond Ubiquiti the next time I need to expand the network.
- dwater 6y agoI'm hoping this spurs the board to oust the CEO who seems to be the one getting the blame for Ubiquiti's shift in recent years away from making highly reliable prosumer focused products to pushing half-baked cash grabs out the door.
- xoa 6y agoThe CEO is also founder and held onto a super majority share of the stock. It's his board essentially, which was a governance issue I remember seeing identified as an investment risk years ago. If it wasn't for that he might have been pushed on a long ways back, but there it is. There's risks to both full submission the shareholders and full independence of shareholders.
- kiwijamo 6y agoYesteday I ordered new MicroTik gear to replace the last of my Ubiquiti gear only to find there's a two week wait for it to be shipped! Looks like there's been increased demand for products made by Ubiqiti's competitors. Thanks Ubiquiti for a good few years but time to move on. You've lost my trust big time.
- sliken 6y agoWhat did you replace? With what?
- wil421 6y agoMikroTik is always saying delayed shipping. Amazon has been the only place I’ve seen them in stock but from 3rd party sellers. MikroTik isn’t even in the same league as UniFi. Their interfaces and docs are terrible. The only reason I ever considered buying something from them is because they had a good price for a 10gb switch. If you want a dumb device you can’t beat their price. Check out a few of the reviews on Amazon. Seed and Don’s reviews are very critical. https://www.amazon.com/dp/B07NFXN4SS https://www.amazon.com/dp/B07NFXN4SS
- fossuser 6y agoThis is the main reason I’ve kept my Unifi stuff and cloud key despite the growing company issues. Is there anything in this space that is good hardware with a decent UI? When I looked the options were consumer crap or very expensive enterprise hardware with no UI.
- jabroni_salad 6y agoLast summer I installed a bunch of second hand Arubas for a non-profit. it was dead simple and the first AP you set up acts as the controller for the others, no cloudkey equivalent needed. I'm not sure if the UI is 'consumer friendly' but I didn't need the commandline and I had never worked with any HP networking gear prior to that, and had no difficulties.
- phatfish 6y ago
- caminocorner 6y agoUbiquiti is publicly traded and operates in many states with privacy laws. I wonder if the SEC or states' AGs will get involved
- orev 6y agoThey will get involved as much as they have with every other breach — i.e. not at all. Even Equifax only got an extremely minor dog and pony show for the complete disaster they created.
- sliken 6y agoSuggestions on what to replace a Edgerouter 6p with? Ideally something with regular security updates and zero cloud. I was considering a the Odroid-h2+ from hardkernel. It has an Intel J4115 (4 cores, 10 watt TDP), 2 Sodimms (max 32GB ram), and two 2.5Gbit ethernet ports. Looks like it would make a decent router. There's an optional board to add 4 more 2.5Gbit ethernet ports. Wasn't sure on the OS, OpenWRT is definitely in the running. Or maybe VyOS (the open fork of vyatta), which is relatively close to what Ubiquiti ships.
- akeck 6y agoI'm strongly considering OpenWRT on PC Engines hardware for my next AP.
- _joel 6y agoI built a number of VPN endpoints uning these for PoP's / Remote Offices. They're superb, not used any RF stuff with them but for the price and performance you can't go wrong.
- Naac 6y agoGrab one of these: https://www.amazon.com/dp/B01H2QJTM4/ref=cm_sw_r_cp_apa_glc_fabc_XYT3FK54453RCFP2JESZ?_encoding=UTF8&psc=1 https://www.amazon.com/dp/B01H2QJTM4/ref=cm_sw_r_cp_apa_glc_... And put Opnsense on it. Guaranteed security updates, clean GUI interface, open source. Set it up once then forget it. If you want an AP there are many options on Amazon. For maximum security prevent the AP from being able to access the internet.
- dustinmoris 6y agoWhat was the main drive for Ubiquiti customers to buy Ubiquiti? I don’t mean this as a troll question, but as an engineer myself I always preferred my TP-Link mesh which serves my entire house and garden perfectly blazing fast internet and I never had to go to the cloud like Ubiquity customers. What do they offer for so many technical people to willingly undoubtedly compromise their security by using their cloud as well?
- dwater 6y agoI don't use their cloud. I have a router, 2 switches, and 2 APs from the Unifi line. I run the controller software on one of my servers and nothing phones home out of my LAN. The APs have good coverage, everything is very reliable (as long as you don't upgrade to new firmware), and there single pane of glass management console provided by the controller software is awesomely convenient with just enough options and features for my needs. but all of this hardware is a few years old now and no longer what they want to sell.
- phalangion 6y agoUbiquiti doesn't have to go to the cloud. I manage my Ubiquiti stuff locally. The reason I bought Ubiquiti is because they offered pro-ish level quality and reliability at closer to consumer prices. At a time when the only consumer options were pretty crappy, Ubiquiti offered higher quality at a small premium without having to go all the way to Cisco APs, for example. I haven't really bought any new gear for several years, so I don't know if the landscape has changed.
- somethingwitty1 6y agoYeah, this is exactly it. They used to produce enterprise-ish level hardware and at a reasonable price. Their devices, in my experience, were pretty rock solid. Setup took a little bit more work than the consumer (edited) offerings, but then you were done. They just ran. They could handle more packets, at least at the time, than the majority of consumer hardware. They've now shifted to targeting consumers and their new stuff doesn't even have the basic features of other consumer hardware. Want DDNS? Something found in every other consumer router I've used. Nope. Want an admin console to configure or update without your smartphone (or when your wifi is down again)? Nope. Want to set a static IP for a device? Ok, but we restart the router each time you set one. Want your network to be bigger than /24? Nope. You can checkout their forums and see how long they've been aware consumers want these things and they've delivered on so little.
- Waterluvian 6y agoI have a Unifi LR AP I want to replace. I have a three level home and this AP really struggles to provide coverage. I think my mistake was getting a directional AP. Are there any decent consumer alternatives if I'm done with Ubiquiti?
- sliken 6y agoI've looked, haven't found anything clearly better. Just got a Unifi Wifi-6 LR AP. Seems reasonable to just block APs from internet traffic and run the AP management software locally (also with traffic blocked from the internet). If you find something better, please post.
- ehutch79 6y agoYou really will likely need multiple APs. one of the consume mesh setups like eero or orbi or whatever will help
- pixl97 6y agoI'm going to agree with ehutch on this. Don't try to cover a multi story house with a single AP. Would you attempt to cover the house with one music radio? No, it would be way to loud near the stereo and way too quite far away. Also you spill signal into neighbour's location and blast excess noise into the spectrum. 3 APs on a lower setting would give you much better performance.
- Waterluvian 6y agoI appreciate yours and ehutch's feedback. 1 AP is a hard requirement, sadly. I have 6 locations in my house I can plug in my cable modem and for now there shall be no hole drilling and Ethernet cable running. My current setup works, but I'm surprised I cannot just centrally locate 1 Wifi AP. We did that when I was younger in a far larger house. But I guess back then wifi was far slower and more robust?
- boatsie 6y agoThe problem is 5ghz has much better bandwidth but less range than 2.4ghz. Also there is more interference these days with everyone having routers.
- softwaredoug 6y ago> At this point, we have well-developed evidence that the perpetrator is an individual with intricate knowledge of our cloud infrastructure. That's quite interesting. You can imagine a disgruntled, former employee that didn't lose Lastpass or AWS access... or someone else very close to the company?
- skue 6y agoYou can also imagine an executive team trying to shift suspicion towards the whistleblower, who happens to be an individual with intricate knowledge of their cloud infrastructure.
- ed25519FUUU 6y agoWhat's the best option for POE access points that aren't Ubiquiti? They seem to have a corner on this for the home market.
- softwaredoug 6y agoThe #1 thing Ubiquiti trades on is trust and it’s shocking how they don’t see that. They need absolute transparency directly to their security experts, which need to operate independently of management, to earn and maintain that trust. Even little hints of prevaricating won’t be seen charitably in any way. They’ll be seen as hiding something. It’s close to importance to vaccines. Did you know vaccine safety regulators see the data from a trial before the company management? They operate independently because trust and transparency is so absolutely crucial for both the product and public health. We rely on routers quite a bit in society to secure our identities, personal transactions, and critical infrastructure: we should be applying the same mindset that we do to vaccine safety to router security!
- dang 6y agoThis is reporting on the Krebs article which was discussed yesterday: Whistleblower: Ubiquiti Breach “Catastrophic” - https://news.ycombinator.com/item?id=26638145 https://news.ycombinator.com/item?id=26638145 - March 2021 (770 comments) Previously: Ubiquiti Networks Breach - https://news.ycombinator.com/item?id=25735032 https://news.ycombinator.com/item?id=25735032 - Jan 2021 (467 comments)
- AdmiralAsshat 6y agoI think the story here (at least my intent when I submitted it) was that Ubiquiti released a "statement" in response to the Krebs article and, distressingly, they don't seem to challenge any of the article's core assertions. Their response seems to be "Oh, um, yes...that happened. But hey! We don't think they accessed any customer information! So...please stop looking at this."
- nhance 6y agoI manage a few of these. One I have installed at a customer location acted very strangely until I ran it through a full factory reset. I observed the UI was hiding the ability to apply updates, SSH was enabled, but it didn't allow me to connect and the name of it was changed to what looked like the MAC address. No permissions were changed, the UDM was sitting unattended and the other administrative user did not update their password prior to this happening. I am very paranoid and ended up doing a full reset as I was only able to partially administrate it. Has anyone else had a similar experience? P.S. The full reset did seem to resolve it