21 ms·

Ubiquiti all but confirms breach response iniquity

609 points by parsecs 6y ago

ksec 6y ago
HN probably get tired of me banging on about it. But is about time Apple come back to Wireless and Router business.
Normal_gaussian 6y ago
So ubiquiti can't be trusted. What are the suggestions for running a ssries if home and small office networks in rented buildings (no cabling?). A UDM + nano ap / flex HD as wireless bridges & mesh wifi gave VLANS, performance monitoring, and an ease of use that let even a junior UI dev implement use it easily and correctlywhile complying with all lease req's. With the world of work at home exploding there seems to be a big missing link here. I'm sitting with a big list of q's that I'm not sure I have a decent amount of time to answer. Does switching to pfsense/openwrt/something open source work with mesh? With ease of set up? Do enterprise brands offer anything worthwhile here? Do I have to regress to letting machines connect to unsecured networks?
- simple_phrases 6y ago
  OpenWRT does mesh networking and OpenWISP allow centralized management of networking appliances. The latter is compatible with the former.
efitz 6y ago
You get great insight into the character of the leaders of a company watching how breaches are handled. Companies that put the customer first are transparent, and quickly take action (even if painful to customers) to ensure that customers’ data and systems stay intact and confidential. Companies that try to gloss over, hide or downplay things indicate that the leadership does not respect their customers and is only interested in maximizing profit/minimizing loss.
- rossipedia 6y ago
  If I can vent for a second, this company has no leadership. None. Things may have changed in 2 years, but I doubt it. I was messaged almost daily by random employees asking wtf was going on with the company. They were afraid for their jobs. Practically no one respected the CEO, and he was the only C-suite exec. There. Was. No. Leadership. There was no company wide communication, and all communication channels were made private, and if you sent an email to more than a couple people you were directly rebuked by the CEO. Nobody felt like they were trusted, and the norm was for most engineers to have absolutely zero idea of what was happening in the company outside of their direct project. Teams were constantly at odds and pitted against each other, and the CEO never resolved any conflicts between teams or employees. The company (at least the software side) was treated like Thunderdome. Some team leads and office managers took care of their people, but most people were just beaten down. I don't think I'd ever seen a less motivated, more dejected group of software developers than I did during my time there. IMO, this kind of bullshit clown show starts from the top. And as long as the top doesn't want to fix it, it won't get fixed. And since software almost invariable ends up reflecting the structure of the organization that produced it, you get this kind of security shit show. I hope this is the last one and they get their act together. But realistically I can't believe that'll happen.
  - ex_ubiquiti 6y ago
    > There was no company wide communication, and all communication channels were made private I couldn't understand why the ex-Amazon cloud lead was also in charge of Slack. When he made all channels private and put a Slackbot in every channel to monitor conversations, I knew it was all over. I'm worried his Slackbot logs are part of the leak. Guy had his hands in everything :( Same guy who took over GitHub and forced everyone into his self hosted source control because he couldn't trust Github. That decision didn't pay off.
    - sngz 6y ago
      > Same guy who took over GitHub and forced everyone into his self hosted source control because he couldn't trust Github. sounds smart to me. I wouldn't trust github either.
kbumsik 6y ago
I was about to buy Ubiquiti products and it is disappointing. Are there good alternatives other than DIYs like PfSense/BSD?
- dt3ft 6y ago
  Fortinet?
- EricE 6y ago
  Cisco small business line is shaping up nicely. You can get used kit on ebay pretty reasonably, update firmware without needing an account or support contract (yes, I am still talking about Cisco!) and their management console is maturing. For new installs I'm likely to go in that direction since UBNT seems lost and directionless, plus regressing on their former major selling points like no cloud required. At this point it wouldn't shock me if they announced subscription required for firmware updates and at that point screw it - might as well go Meraki or Ruckus and get actual support as well as a lot better performance.
rosege 6y ago
Opened HN to look to see what everyone was saying about the FB hack, stayed for the Ubiquiti one.
xyzzy21 6y ago
"The Cloud" absolutely can NOT be trusted with anything serious. I'm still amazed serious people actually think it's a smart or wise idea. It's become a "Go to the fridge and get the box" type of mindless laziness by far too many marketers and developers.
- Black101 6y ago
  It's going to get much worst before it gets better.
haberman 6y ago
Can companies be held responsible for damages from data breaches? If they could, it seems like it would incentivize more caution about what data is collected, and more investment in the security of that data. I also imagine an insurance industry, where the insurers then have expectations about what kinds of security must be in place to get reasonable premiums.
- redler 6y ago
  Yup, this is more or less how “cyber security” policies work.
  - elliekelly 6y ago
    Unless it’s changed in the last two or three years cyber security insurance policies seem only to cover the cost of notifying customers of the breach and paying for credit monitoring for whatever period of time is required in each customer’s specific jurisdiction. (When last I looked, in most states it’s none.) Every time I looked into cyber security insurance it wasn’t worthwhile at all because it didn’t provide any meaningful coverage. Maybe for a small startup with a lot of PII it would make sense but I think most companies would probably come to the same conclusion.
    - boston_clone 6y ago
      It may be worth reviewing cybersecurity insurance policies with your legal team! At a former company, we had a nasty case of BEC with a vendor that ultimately cost us well over six figures - over 90% of the loss was recouped by filing a claim with our insurance.
lazyweb 6y ago
Mentioned it before, but since a few days ago my unifi devices (2 wifi APs, a small switch, plus one Debian VM with the controller, all on it's on VLAN) are not allowed to do outbound traffic anymore, with the exception of NTP, DNS and one trusted apt mirror. Looking at the firewall logs it seems the devices try to ping (ICMP type 8) a bunch of AWS IPs every few hours. The controller tries to connect 80/443 on different AWS IPs a lot more often, even without me navigating the web interface. Other than that, no ill effects. Device firmware update notifications are gone, just says "up to date" now. Interestingly, I still see the ad for their "dream machine" on the dashboard, as it seems to be baked into the controller. It's also trying to load external resources from "net-fe-static-assets.network-controller.svc.ui.com" while navigating the new web interface. The "classic" interface still seems to be truly self-contained. Using the latest controller version as of today (6.1.71-15061-1). Condensed firewall logs for reference below. Not that it matters much, but why not. Unifi controller VM: zgrep unifidrop /var/log/syslog\* | grep "SRC=$unificontroller" | awk '{print $12, $21}' | sort | uniq -c | sort -h 5 DST=13.224.246.17 DPT=443 5 DST=143.204.174.59 DPT=443 5 DST=143.204.174.83 DPT=443 5 DST=34.210.116.187 DPT=80 5 DST=34.211.38.191 DPT=80 5 DST=34.218.198.60 DPT=80 5 DST=99.84.5.14 DPT=80 5 DST=99.84.5.24 DPT=80 5 DST=99.84.5.51 DPT=80 5 DST=99.84.5.82 DPT=80 7 DST=13.224.246.67 DPT=443 7 DST=13.225.74.11 DPT=443 7 DST=13.227.220.19 DPT=443 7 DST=13.227.220.38 DPT=443 15 DST=54.201.165.155 DPT=443 25 DST=44.239.243.150 DPT=443 28 DST=44.238.226.202 DPT=443 28 DST=52.89.51.163 DPT=443 28 DST=54.218.175.125 DPT=443 Unifi devices (all ICMP 8): zgrep unifidrop /var/log/syslog\* | grep -v "SRC=$unificontroller" | awk '{print $12}' | sort | uniq -c | sort -h 2 DST=13.224.230.94 2 DST=143.204.9.24 4 DST=99.84.6.169 6 DST=52.84.94.172 6 DST=54.230.54.165 24 DST=52.222.138.169
- electro_blah 6y ago
  LOL sounds like somebody installed a rogue device on your network.
- xyst 6y ago
  I have said this before, but would like to reiterate that I am never touching or buying anything branded as Ubiquiti or owned by Robert Pera. This hardware is far from cheap and consumers are literally paying for adware/spyware. I really hope Ubiquiti stock takes a nosedive over the next year.
imwillofficial 6y ago
I used to be a die hard Ubiquiti fan. They have fallen from grace in a big way. Disappointing.
arbitrage 6y ago
So, what happens now? Will Ubiquiti be held to task, by anyone?
- LgWoodenBadger 6y ago
  I’m done buying ubiquiti equipment. 6 devices, and 3 family members I recommended ubiquiti to who also have multiple devices. Clearly the market exists for what they’re offering. I am surprised at the serious lack of alternatives.
  - commandar 6y ago
    HPE seems to be aiming squarely at Ubiquiti with their Aruba Instant On line (which is distinct from the Aruba Instant line because what's life without some confusing branding?). I installed some of their APs and switches in my home a few weeks ago and it's working well. It ended up a little less expensive than comparable UBNT gear would have been and there's actual availability on their Wifi6 APs. The APs are cloud managed only, but, personally, I trust HPE not to make a complete clown show of things the way UBNT has. But that's absolutely going to be a deal-breaker for a lot of people (especially with the way UBNT's now poisoned the well to a degree). The big hole in that lineup is a gateway device. They just don't have anything comparable to the UDM (Pro). I'd be surprised if they don't eventually introduce something, though, given how the rest of the lineup seems pointed directly at the niche Ubiquiti is currently occupying.
    - sigg3 6y ago
      > The APs are cloud managed only, but, personally, I trust HPE not to make a complete clown show That is unwise on the long-term, though. The technology is there so you don't have to trust anyone, so why do you choose to trust them when they're not offering cloudfree solutions?
      - commandar 6y ago
        Because it was the best available option in my price range in all other aspects and I honestly don't care all that much where the AP controller for my home network is located. The APs will continue to work while offline and I'm unlikely to even look at the controller interface more than a handful of times a year provided the vendor doesn't do anything that will outright break things. I honestly don't care if it's hosted inside my network or not. Ubiquiti has a history of poor software releases that break things and now of trying to gloss over a serious security breach. I'm less worried about HPE in that regard.
gvkhna 6y ago
I’m still on board with Uniquiti, tons of equipment and it wouldn’t make sense to switch everything over for small operations. But this is extremely disappointing, they’re definitely moving in a little bit of a different direction then where many of us would hope. More shiny products that increase bottom line is great but many IT officials rely on UniFi as well, I wonder how they’re responding to enterprise customers. I just hope this incident will at least get them to put some emphasis on security again as well.
- liaukovv 6y ago
  I wonder if you could extract costs of migration from ubiquity with a lawsuit
  - madeofpalk 6y ago
    Sounds like a pain that’s not worth it.
  - nomadiccoder 6y ago
    You shouldn’t.
    - liaukovv 6y ago
      Why not?
- neartheplain 6y ago
  >I’m still on board with Uniquiti Freudian slip?
- ex_ubiquiti 6y ago
  > they’re definitely moving in a little bit of a different direction then where many of us would hope it pains me to say this because I was there for the UniFi glory days: The old Ubiquiti is dead and gone. Almost everyone I know quit. I hope they can land on their feet and return to the glory days but I don't have much hope. The company got toxic fast at the end
teeray 6y ago
What I’m curious about is, if I run my own controller on my own hardware, do I need to be concerned about this? I could understand supply chain concerns... I’ve held off updating anything while this plays out. But all these “breach! breach!” stories fail to spell out who is affected and what they need to do.
- gerdesj 6y ago
  You probably don't need to be concerned(ish). I run a controller for 32 "sites" across the UK with 1 to 13 APs per site and a few switches. I keep it behind HAProxy but with fairly minimal changes (from memory.) I have stuck with controller 5.13.32 rather than moving to 6.x just yet. It's an LTS version and I'm still waiting for the whinging to stop on the forums. I also watch the AP firmware and that has had some interesting times over the last few months. I've confirmed dodgy AP versions on my sites and backrevved and held accordingly. I treat the whole thing the same way I do any other system. I come out in spots when people mention clouds and IT in the same sentence, so I have not knowingly enabled any cloudy integrations from my controller to UBNT. Specifically, I have not enabled "Remote Access".
- ev1 6y ago
  Force pushed updates overnight turned local controllers into requiring ui.com single sign on, iirc.
- Nextgrid 6y ago
  If the compromise is widespread enough then the attackers might have gained control of the update infrastructure allowing them to push out malicious firmware to your devices.
  - js2 6y ago
    These blanket statements don’t apply to everyone. It depends which Ubiquiti hardware you own and how you’ve configured it. For example, I run the UniFi controller on my FreeNAS server. There are no forced updates to it. It doesn’t update unless I update it. The firmware on my APs doesn’t update unless I update them from my controller.
brian-armstrong 6y ago
Has anyone looked at Ubiquiti's firmware signing? Would it be possible to patch it to retain the drivers and kernel but replace the configuration layers? Being able to homebrew some config would make the equipment more valuable to us I think.
- gertrunde 6y ago
  People have been running OpenWRT on Ubiquiti gear for quite a long time iirc. [https://openwrt.org/toh/ubiquiti/start https://openwrt.org/toh/ubiquiti/start]
  - Hikikomori 6y ago
    Afaik performance will be abysmal on edge router series as the npu isn't used.
    - KirillPanov 6y ago
      From firsthand experience: performance is in fact awesome on the edgerouters (4, 6, 8, and 12) using plain-vanilla Linux. It's a big honking MIPS chip with firehose connections to the ethernet PHYs. Precisely the kind of device you want for a router.
      - Hikikomori 6y ago
        Then you are better off buying something with a beefier cpu that costs less since it doesn't have an npu.
      - joshspankit 6y ago
        Awesome at what level? Are we talking DPI at 1Gbps symmetric?
      - sigg3 6y ago
        Please elaborate. What are the thruput measurements on OpenWRT when compared to ER-.. stock firmware, with hw accel or with DPI..? I have an ER-4 to be able to use the entire wan connection, but on stock firmware I must disable DPI to enable hw acceleration (otherwise the thruput floors). I don't use DPI atm, so no big loss. Can you utilize the hw accel in openWRT too?
vr46 6y ago
So this week, I have gone from having a single little USG and a massive order planned for loads of kit to stopping them automatically updating the firmware and dropping that order. Extremely annoying, but not as annoying as if this had happened in a couple of weeks.
- tifadg1 6y ago
  So what are vendor are you changing to now?
  - vr46 6y ago
    None! Going to keep my jerry-rigged-Heath-Robinson networks with the existing mesh and switches until things resolve to a satisfactory juncture.
- EricE 6y ago
  Why anyone has - of all things - automatic firmware updates for networking gear always blows my mind away. Also you can still run all your UBNT kit completely disconnected from the cloud - way better than anyone else. The single management pane is still very useful. Actually I run everything but the USG - their router/firewall is absolutely the weakest link in their product suite. The APs are a good value and solid performers, especially if you stay on earlier, stable firmwares (and DON'T auto update). Their switches are a fair value. Lack of progress with layer 3 routing, especially on their bigger/pro switches is annoying and skews the value prop for those away from UBNT. Cisco's Small Business line is starting to look pretty good and they have a management console that is maturing. And at the rate they are going they should easily be able to catch and blow past UBNT within a year. And so far (despite being from cisco) it's also one of the few product families in this space that does NOT require a subscription.
  - vr46 6y ago
    Good points. I got the auto firmware thing wrong, it was actually the controller update, which I run on a Pi. And fair point on running it disconnect from the cloud, which it is now. Going to check out the Cisco stuff.
xvector 6y ago
Ubiquiti has lost my business. And with the recent issues with Netgate/PfSense [1], it looks like OpnSense is the way to go. [1]: https://arstechnica.com/gadgets/2021/03/buffer-overruns-license-violations-and-bad-code-freebsd-13s-close-call/ https://arstechnica.com/gadgets/2021/03/buffer-overruns-lice...
- jessebarton 6y ago
  why would you not just run OpenBSD with PF.
  - bpye 6y ago
    Why should I choose OpenBSD over FreeBSD or even Linux with nftables?
    - dijit 6y ago
      If you’re really asking, and not making a point; PF is created and primarily maintained by OpenBSD OpenBSD’s base system (without extra packages) includes PF and has a focus on security. PF in freebsd is several major versions old. nftables (like iptables before it) is rule based and not bucket based. So high numbers of rules will not affect pf’s performance like it does with nftables. But, for home users, probably not noticeable. Though I prefer the syntax of PF personally.
      - hyperpl 6y ago
        Wireguard has also been stable on OpenBSD which helped me with my throughput on my apu2d router hardware.
      - fuzzy2 6y ago
        Could you expand on what you mean by "bucket based"? Maybe the so-called "tables"? They sound pretty identical to ipset on Linux.
        6y ago
rossipedia 6y ago
> Ubiquiti also hinted it had an idea of who was behind the attack, saying it has “well-developed evidence that the perpetrator is an individual with intricate knowledge of our cloud infrastructure. As we are cooperating with law enforcement in an ongoing investigation, we cannot comment further.” I personally don't believe this. IMO, this is a company who is looking for a fall guy, and _most likely_ it's going to be somebody who raised a stink about all the security problems during their time there. Form your own opinion, I'm just a guy who worked at Ubiquiti for a year, raising all kinds of hell about the security, architectural, and operational problems that I saw while I was there. But what do I know...
- judge2020 6y ago
  Given they were stupid enough to spin up some VMs, I doubt it was someone that knew what they had access to. A skilled attacker would stay dormant sucking up all data accessible via the AWS API (including s3 stuff) and potentially keep access to the infrastructure for years.
  - smashed 6y ago
    There is no evidence that this did not also happen.
    - brippalcharrid 6y ago
      And if it is happening, we might hear about that in a few years' time, if it's discovered, and if it's brought to light in circumstances that are conducive to the vendor making a public disclosure (eg. which are impossible to cover up).
  - deleted 6y ago
    [deleted]
  - throwaway8581 6y ago
    This kind of analysis is basically worthless because you don’t know whether they are operating at multiple levels of deception by, e.g., making you think they are a stupid script kiddie and that you successfully wiped them out.
tpmx 6y ago
By now we'll have to ask: Is it realistic to expect hardware-oriented companies to build secure software? (Yes, Apple exists.)
- eertami 6y ago
  I think your question is wrong, it should be: Is it realistic to trust any company to build completely secure software? I don't see your point about Apple, unless you're being sarcastic for comedic effect. Apple release software with security flaws too. In fact a zero-click security vulnerability present in the Apple email client was posted on this very page only three days ago[1]. [1]: https://news.ycombinator.com/item?id=26664714 https://news.ycombinator.com/item?id=26664714
- ryandrake 6y ago
  Most hardware companies don’t care in the slightest about software quality. To them, software is just another line item on the Bill Of Materials, like a bolt or piece of sheet metal. You either have some overworked intern who knows C cobble something together that barely works or you buy it from the least expensive supplier. When the build is ramping, at the end of the assembly line somebody is going to flash something on the device, and they are not going to stop the line to worry about a security hole.
  - deleted 6y ago
    [deleted]
- jnwatson 6y ago
  This wasn't about the security of their "software", as in the thing that's running on your device. This was about their backend security. That's a much, much tougher call to make.
- simple_phrases 6y ago
  iOS exploits are cheaper than Android exploits because iOS exploits are so plentiful in comparison[1]. [1] https://www.cyberscoop.com/ios-zero-day-zerodium-high-supply/ https://www.cyberscoop.com/ios-zero-day-zerodium-high-supply...
d-funct 6y ago
What no one seems to be really discussing is how paranoid should people be around this breach? Is it a case of you probably want to rebuild machines that have default usernames/passwords? Or is it more whatever can be seen in the Ubiquiti UI might be been accessed by third parties?
- rovr138 6y ago
  > Is it a case of you probably want to rebuild machines that have default usernames/passwords? I mean, regardless, most probably, the answer to this is yes.
bcrescimanno 6y ago
It's disappointing to see a breach like this and even more disappointing to see what (at least on the surface) appears to be a lackadaisical response. At someone who runs a UniFi network in my home with just 4 pieces of hardware (gateway, wired switch, and 2 PoE WAPs) I'm really curious if there are solid alternatives for a managed home network. UniFi really hit a sweet spot of price/performance that made it a somewhat pricey; but, not totally unreasonable option for the home. Any suggestions from the HN crowd?
- e40 6y ago
  That is my exact configuration, too. Would love to have alternatives.
  - heavymark 6y ago
    I’m not aware of any alternatives that are designed as well, and if you switch the new option could just as easily be hacked or if so it on it could also be hacked but you may never realize. Though it’s good for all these people to pretend to threaten to leave since maybe that will get the company to be a little more forth right which is all we can really ask for these days.
- ThatPlayer 6y ago
  I've heard good things about TP-Link's Omada series. Their controller even looks like a clone of Unifi's
  - ed25519FUUU 6y ago
    Isn’t TP-link a Chinese company?
    - catblast01 6y ago
      Is ubiquiti a Chinese company? Really, what a low effort idiotic post.
deleted 6y ago
[deleted]
ta20210405 6y ago
Amateur hour at Ubiquiti. Sadly they leave us without a decent replacement. Mikrotik is the only contender and they leave a lot to be desired.
- dt3ft 6y ago
  I recommend fortinet as replacement for USG. Not as cheap, but you get what you pay for.
dmix 6y ago
Looks like a basic ransom request but pushing malware to the ‘85 million’ devices through automated automated would be far more damaging.
smiley1437 6y ago
Anyone know if Apple will be putting out a wifi mesh system, maybe integrated into Homepod Minis? Apple already 'owns' me, I might as well have them run my Wifi too and ditch my unifi gear. At least Apple seems to care about privacy and security, even if it is a self-serving marketing scheme.
- lostlogin 6y ago
  Their wifi line used to be excellent. Having APs that could be hardwired would be a requirement for me. The less wifi the better.
hrgiger 6y ago
I keep one 6p behind isp router to manage home network, they have good hardware but i didnt like the idea exposing to cloud, only allowed local dns, ntp. And removed all port listeners from ubi in sbin then touched a new file with same name. Latest firmware complained a lot but worked at some point. I am not sure i am fully secure but quite happy with performance
wnevets 6y ago
breaches can happen to anyone however as a customer the way Ubiquiti has been handling this is really disconcerting.
spurgu 6y ago
Am I the only one annoyed with the expression "all but"? To me it sounds like the complete opposite. "All but confirms" to me sounds like they're "doing everything else than confirming" / "all other things except confirming".
- CRConrad 6y ago
  It means ~"Comes so close to confirming that it does everything up to it, except actually confirming" -- the actual confirmation is all that's missing. Once I realised that, it stopped annoying me. Auttoiko?
TwoNineFive 6y ago
On this subject, does anyone know what is up with the reddit sub, r/ubiquiti? Seems to be run by u/briellie. She(?) seems like a really toxic person with some kind of business relationship with Ubiquiti like a reseller or something. The Reddit sub seems like they are actively trying to suppress discussion of this issue. There's some allegations of censorship on the sub, but I'm not seeing it... which might actually just be confirmation that they are censoring. I don't know.
- moxzyros 6y ago
  There was a recent discussion[1] on the sub with 1K upvotes and over 500 comments about the breach and I routinely see unabated salty posts and comments about Ubiquiti's downward spiral. I have lurked on the sub for several years (I manage a bunch of Ubiquiti gear) and I never got the impression people were being censored or moderated into submission. Are there any particular examples of suppression or links to the allegations of censorship? The sub did recently begin allowing equipment picture posts again by popular demand. [2] I suppose an uncharitable interpretation is that that move was appeasement to distract from the breach issue. [1] https://www.reddit.com/r/Ubiquiti/comments/mgm4o7/whistleblower_ubiquiti_breach_catastrophic_krebs/ https://www.reddit.com/r/Ubiquiti/comments/mgm4o7/whistleblo... [2] https://www.reddit.com/r/Ubiquiti/comments/mi0679/rule_change_equipment_picture_posts_are_allowed/ https://www.reddit.com/r/Ubiquiti/comments/mi0679/rule_chang... Edit: Formatting
worik 6y ago
I can believe that they do not keep logs of the database access. As brain dead as it sounds. I have been in the position of implementing a client on a API I do not control. The owners of the servers (colleagues but in a different country) do not seem to know what logs are. We get random failures from the server. I can pin down to the second when they occur (not closer because of network lag). I suspect that the server is failing under load, but the way I would find out is to... Read the logs. My foreign colleagues do not respond to me, ghost me entirely, when I ask them to inspect the logs. Perhaps it is a Windows/Azure thing?
- jiggawatts 6y ago
  Logs are typically off by default in most Enterprise software, or goes nowhere by default which is basically the same thing. Logs cost money to both collect and store. Not everyone is cheerfully burning through VC capital. Some people have budgets. Speaking of log collection, simply dumping the logs into a central repository is the same as taking the garbage to the landfill. Collecting trash just results in a big collection of trash. Extracting useful information from a huge pile of logs is non-trivial. You'd have to know at least one query language, probably several if you work on big enterprise systems. You'd have to know a bunch of esoteric things like how to convert the long decimal strings to a number so that you can interpret as any one of a dozen timestamp formats as an actual datetime in UTC, and then convert that to local timestamp so that you can tell when something actually happened. And so on. All of this is merely a prerequisite for finding a specific instance of what you know is there. You know what you'll never find in logs? Things you didn't know to look for! That's unfortunately about 90-99% of what you actually need to know, making logs typically about 1-10% as useful as you'd like. Did I mention they cost money? Have you seen how many arms and legs Splunk charges these days? Why would you pay that much for something that is less than a tenth as useful as it could be? The fundamental problem is that discipline doesn't scale. You can't expect a hundred thousand IT operations guys to all do everything all of the time, in spite of the non-technical finance guy holding on to the purse strings like he's gripping a life preserver after going overboard in a raging storm. PS: I turned logging on extensively for a recent Azure project. Some logs cost more than the service they were monitoring. I mean sure, I could turn off the unnecessary logs, but how do I know ahead of time which ones will be necessary or not? I don't have a time machine! I can't go back and turn off the logs I won't need... later. PPS: Have you noticed all logging companies charge by the gigabyte? What incentive do you imagine they have for improving the efficiency of the log transfer and storage formats?
lifeisstillgood 6y ago
Off topic but is there a good guide to middle level home network setup - something like using OpenWRT on (Rpis?) and turning that into a router and couple of access points. I was going to press buy on the setup for some ubiquiti products till a couple of days ago :-(
- EricE 6y ago
  Unifi is still great for APs or switches. Their routers are OK if you just want the basics. If you set the controller up in a VM or raspberry pi you can stay 100% local all the time. Leave auto updates OFF, only upgrade the firmware/controller until you have a real reason to - something important gets fixed, a new feature is released you want to use - that sort of stuff. I have sites that firmware hasn't been updated in years and doesn't need to. It always amuses me in one thread people bitch about MS, Windows 10 and forced updates then in threads like this I read people fessing to having auto updates enabled voluntarily. Ugh. As for alternatives, Cisco's small business line is looking pretty reasonable. You can find used gear on ebay easily, you can upgrade firmware without a subscription or even needing an account (yes, I'm still talking about Cisco!) and while their management console is a bit weak, it's maturing. I think it will mature faster than UBNT who was strong out the gate but delivered nothing new or significant for almost 7 years now in routing, switching has never delivered layer 3 - especially useful for larger switches. APs seem to take longer to get bugs resolved in their firmware - but like I said if you only upgrade when you really need to that won't even effect you.
ghostpepper 6y ago
I would love to see a competitor spring up targeting the same enthusiast/prosumer segment. It seems like there are quite a few ex-employees with knowledge of how to build it.
tgpc 6y ago
Long-time Ubiquiti fan here Their lack of Wifi 6 across the range, and the security problems drive me to look at alternatives. Found the Netgear WAX610. Very happy with them.
aneutron 6y ago
I'm thinking it won't be long before folks roll their own distro of Unifi APs and switches.
jimnotgym 6y ago
Hang on a minute there > Ubiquiti’s IoT gear includes things like WiFi routers I understood IoT to mean wifi toasters, TVs and other home appliances. Since when was a router an IoT device? Are we going to call all nework devices IoT now. This strikes me as taking rather too much journalistic license. In fact wtf is a WiFi Router. I use Unifi to deploy Wireless Acess Points on a LAN with centralized control. It is possible to do this without them having internet access at all, but it makes it rather harder to update everything. This is miles away from IoT. Describing Ubiquity as a IoT company is like calling Cisco, Juniper, Mikrotik and Aruba IoT companies. This sounds like an attempt to feed the narrative that the IoT is going to eat us alive. Let us focus instead on what Ubiquity actually did wrong, isn't that bad enough?
- EricE 6y ago
  Indeed - throwing around the IoT buzzword is just an unnecessary (and irrelevant) distraction.
- abestic9 6y ago
  I believe the IoT distinction is that the platform reports to a cloud service, where parts of that are implemented as SaaS, if only for SSO and remote management. I think this is a fairly loose definition though if the underlying device data is still on-prem. Wi-Fi router I would take to mean one of those consumer all-in-one boxes, which is what the UDM fits the description of. I agree that this distinction is unnecessary. When we consider who wrote the article and who it's for, we should expect a bit more granularity than foggy acronyms.
GekkePrutser 6y ago
Ubiquiti should really stop making cloud logins mandatory. The latest stuff (UDM/UDM Pro, Cloud Key G2) must be connected to their cloud at installation time. Remote access can be turned off but an admin account connected to their cloud remains. Without those ties to their infrastructure, this breach would not be as severe. It would just cause an attacker to see what I've bought from them, nothing else. I'm glad I can still use the unifi controller in docker without any ties to UI.com however their later stuff like Unifi protect, access, talk etc no longer works with that.
- ex_ubiquiti 6y ago
  I worked there and I didn't even understand why we had to force cloud logins on Dream Machine. In the early days we were all about letting people run their own controller hardware and not requiring cloud logins. No one could ever tell us why we had to force everyone to the cloud. It was a mandate from above
  - GekkePrutser 6y ago
    Thanks for that insight! I guess it's for the usual reasons. Telemetry / product improvement, and also more marketing data. Data is the new gold :)
  - EricE 6y ago
    They utterly flushed away their biggest selling point - the only vendor in that space that didn't require a cloud login. Beyond stupid.
dustinmoris 6y ago
I find it really strange that so many claim that they need Ubiquiti and that there is "sadly" no other good alternative. What are people doing with their home networks? What are they comparing it with? Has anyone actually tried some of the mesh networks from TP-Link or other brands? I have one at home and honestly I don't even know what the admin management looks like because I never have to go there and do something. What are people doing? Is it that I am so ignorant to some needs which people have that they constantly need to tweak their networks at home or is it just a symptom of Ubiquity kit that requires users to constantly do something with it that now they think they need all that fancy management stuff because they got used to do so much maintenance work on something that should just work without ever having to touch it again?
MindTooth 6y ago
After seeing that they did not capture the logs. What is the “proper” way of storing said logs? I guess you need a remote logserver like logstash to store them. But what service does actually send the logs from the server to a central storage. Looking into Loki, Graphite, etc. But I’m a bit at a loss where to begin.

609 points by parsecs 6y ago

ksec 6y ago

HN probably get tired of me banging on about it. But is about time Apple come back to Wireless and Router business.

So ubiquiti can't be trusted. What are the suggestions for running a ssries if home and small office networks in rented buildings (no cabling?). A UDM + nano ap / flex HD as wireless bridges & mesh wifi gave VLANS, performance monitoring, and an ease of use that let even a junior UI dev implement use it easily and correctlywhile complying with all lease req's. With the world of work at home exploding there seems to be a big missing link here. I'm sitting with a big list of q's that I'm not sure I have a decent amount of time to answer. Does switching to pfsense/openwrt/something open source work with mesh? With ease of set up? Do enterprise brands offer anything worthwhile here? Do I have to regress to letting machines connect to unsecured networks?

simple_phrases 6y ago
OpenWRT does mesh networking and OpenWISP allow centralized management of networking appliances. The latter is compatible with the former.

efitz 6y ago

You get great insight into the character of the leaders of a company watching how breaches are handled. Companies that put the customer first are transparent, and quickly take action (even if painful to customers) to ensure that customers’ data and systems stay intact and confidential. Companies that try to gloss over, hide or downplay things indicate that the leadership does not respect their customers and is only interested in maximizing profit/minimizing loss.

rossipedia 6y ago
If I can vent for a second, this company has no leadership. None. Things may have changed in 2 years, but I doubt it. I was messaged almost daily by random employees asking wtf was going on with the company. They were afraid for their jobs. Practically no one respected the CEO, and he was the only C-suite exec. There. Was. No. Leadership. There was no company wide communication, and all communication channels were made private, and if you sent an email to more than a couple people you were directly rebuked by the CEO. Nobody felt like they were trusted, and the norm was for most engineers to have absolutely zero idea of what was happening in the company outside of their direct project. Teams were constantly at odds and pitted against each other, and the CEO never resolved any conflicts between teams or employees. The company (at least the software side) was treated like Thunderdome. Some team leads and office managers took care of their people, but most people were just beaten down. I don't think I'd ever seen a less motivated, more dejected group of software developers than I did during my time there. IMO, this kind of bullshit clown show starts from the top. And as long as the top doesn't want to fix it, it won't get fixed. And since software almost invariable ends up reflecting the structure of the organization that produced it, you get this kind of security shit show. I hope this is the last one and they get their act together. But realistically I can't believe that'll happen.
- ex_ubiquiti 6y ago
  > There was no company wide communication, and all communication channels were made private I couldn't understand why the ex-Amazon cloud lead was also in charge of Slack. When he made all channels private and put a Slackbot in every channel to monitor conversations, I knew it was all over. I'm worried his Slackbot logs are part of the leak. Guy had his hands in everything :( Same guy who took over GitHub and forced everyone into his self hosted source control because he couldn't trust Github. That decision didn't pay off.
  - sngz 6y ago
    > Same guy who took over GitHub and forced everyone into his self hosted source control because he couldn't trust Github. sounds smart to me. I wouldn't trust github either.

kbumsik 6y ago

I was about to buy Ubiquiti products and it is disappointing. Are there good alternatives other than DIYs like PfSense/BSD?

dt3ft 6y ago
Fortinet?
EricE 6y ago
Cisco small business line is shaping up nicely. You can get used kit on ebay pretty reasonably, update firmware without needing an account or support contract (yes, I am still talking about Cisco!) and their management console is maturing. For new installs I'm likely to go in that direction since UBNT seems lost and directionless, plus regressing on their former major selling points like no cloud required. At this point it wouldn't shock me if they announced subscription required for firmware updates and at that point screw it - might as well go Meraki or Ruckus and get actual support as well as a lot better performance.

rosege 6y ago

Opened HN to look to see what everyone was saying about the FB hack, stayed for the Ubiquiti one.

xyzzy21 6y ago

"The Cloud" absolutely can NOT be trusted with anything serious. I'm still amazed serious people actually think it's a smart or wise idea. It's become a "Go to the fridge and get the box" type of mindless laziness by far too many marketers and developers.

Black101 6y ago
It's going to get much worst before it gets better.

haberman 6y ago

Can companies be held responsible for damages from data breaches? If they could, it seems like it would incentivize more caution about what data is collected, and more investment in the security of that data. I also imagine an insurance industry, where the insurers then have expectations about what kinds of security must be in place to get reasonable premiums.

redler 6y ago
Yup, this is more or less how “cyber security” policies work.
- elliekelly 6y ago
  Unless it’s changed in the last two or three years cyber security insurance policies seem only to cover the cost of notifying customers of the breach and paying for credit monitoring for whatever period of time is required in each customer’s specific jurisdiction. (When last I looked, in most states it’s none.) Every time I looked into cyber security insurance it wasn’t worthwhile at all because it didn’t provide any meaningful coverage. Maybe for a small startup with a lot of PII it would make sense but I think most companies would probably come to the same conclusion.
  - boston_clone 6y ago
    It may be worth reviewing cybersecurity insurance policies with your legal team! At a former company, we had a nasty case of BEC with a vendor that ultimately cost us well over six figures - over 90% of the loss was recouped by filing a claim with our insurance.

lazyweb 6y ago

Mentioned it before, but since a few days ago my unifi devices (2 wifi APs, a small switch, plus one Debian VM with the controller, all on it's on VLAN) are not allowed to do outbound traffic anymore, with the exception of NTP, DNS and one trusted apt mirror. Looking at the firewall logs it seems the devices try to ping (ICMP type 8) a bunch of AWS IPs every few hours. The controller tries to connect 80/443 on different AWS IPs a lot more often, even without me navigating the web interface. Other than that, no ill effects. Device firmware update notifications are gone, just says "up to date" now. Interestingly, I still see the ad for their "dream machine" on the dashboard, as it seems to be baked into the controller. It's also trying to load external resources from "net-fe-static-assets.network-controller.svc.ui.com" while navigating the new web interface. The "classic" interface still seems to be truly self-contained. Using the latest controller version as of today (6.1.71-15061-1). Condensed firewall logs for reference below. Not that it matters much, but why not. Unifi controller VM: zgrep unifidrop /var/log/syslog\* | grep "SRC=$unificontroller" | awk '{print $12, $21}' | sort | uniq -c | sort -h 5 DST=13.224.246.17 DPT=443 5 DST=143.204.174.59 DPT=443 5 DST=143.204.174.83 DPT=443 5 DST=34.210.116.187 DPT=80 5 DST=34.211.38.191 DPT=80 5 DST=34.218.198.60 DPT=80 5 DST=99.84.5.14 DPT=80 5 DST=99.84.5.24 DPT=80 5 DST=99.84.5.51 DPT=80 5 DST=99.84.5.82 DPT=80 7 DST=13.224.246.67 DPT=443 7 DST=13.225.74.11 DPT=443 7 DST=13.227.220.19 DPT=443 7 DST=13.227.220.38 DPT=443 15 DST=54.201.165.155 DPT=443 25 DST=44.239.243.150 DPT=443 28 DST=44.238.226.202 DPT=443 28 DST=52.89.51.163 DPT=443 28 DST=54.218.175.125 DPT=443 Unifi devices (all ICMP 8): zgrep unifidrop /var/log/syslog\* | grep -v "SRC=$unificontroller" | awk '{print $12}' | sort | uniq -c | sort -h 2 DST=13.224.230.94 2 DST=143.204.9.24 4 DST=99.84.6.169 6 DST=52.84.94.172 6 DST=54.230.54.165 24 DST=52.222.138.169

electro_blah 6y ago
LOL sounds like somebody installed a rogue device on your network.
xyst 6y ago
I have said this before, but would like to reiterate that I am never touching or buying anything branded as Ubiquiti or owned by Robert Pera. This hardware is far from cheap and consumers are literally paying for adware/spyware. I really hope Ubiquiti stock takes a nosedive over the next year.

imwillofficial 6y ago

I used to be a die hard Ubiquiti fan. They have fallen from grace in a big way. Disappointing.

arbitrage 6y ago

So, what happens now? Will Ubiquiti be held to task, by anyone?

LgWoodenBadger 6y ago
I’m done buying ubiquiti equipment. 6 devices, and 3 family members I recommended ubiquiti to who also have multiple devices. Clearly the market exists for what they’re offering. I am surprised at the serious lack of alternatives.
- commandar 6y ago
  HPE seems to be aiming squarely at Ubiquiti with their Aruba Instant On line (which is distinct from the Aruba Instant line because what's life without some confusing branding?). I installed some of their APs and switches in my home a few weeks ago and it's working well. It ended up a little less expensive than comparable UBNT gear would have been and there's actual availability on their Wifi6 APs. The APs are cloud managed only, but, personally, I trust HPE not to make a complete clown show of things the way UBNT has. But that's absolutely going to be a deal-breaker for a lot of people (especially with the way UBNT's now poisoned the well to a degree). The big hole in that lineup is a gateway device. They just don't have anything comparable to the UDM (Pro). I'd be surprised if they don't eventually introduce something, though, given how the rest of the lineup seems pointed directly at the niche Ubiquiti is currently occupying.
  - sigg3 6y ago
    > The APs are cloud managed only, but, personally, I trust HPE not to make a complete clown show That is unwise on the long-term, though. The technology is there so you don't have to trust anyone, so why do you choose to trust them when they're not offering cloudfree solutions?
    - commandar 6y ago
      Because it was the best available option in my price range in all other aspects and I honestly don't care all that much where the AP controller for my home network is located. The APs will continue to work while offline and I'm unlikely to even look at the controller interface more than a handful of times a year provided the vendor doesn't do anything that will outright break things. I honestly don't care if it's hosted inside my network or not. Ubiquiti has a history of poor software releases that break things and now of trying to gloss over a serious security breach. I'm less worried about HPE in that regard.

gvkhna 6y ago

I’m still on board with Uniquiti, tons of equipment and it wouldn’t make sense to switch everything over for small operations. But this is extremely disappointing, they’re definitely moving in a little bit of a different direction then where many of us would hope. More shiny products that increase bottom line is great but many IT officials rely on UniFi as well, I wonder how they’re responding to enterprise customers. I just hope this incident will at least get them to put some emphasis on security again as well.

liaukovv 6y ago
I wonder if you could extract costs of migration from ubiquity with a lawsuit
- madeofpalk 6y ago
  Sounds like a pain that’s not worth it.
- nomadiccoder 6y ago
  You shouldn’t.
  - liaukovv 6y ago
    Why not?
neartheplain 6y ago
>I’m still on board with Uniquiti Freudian slip?
ex_ubiquiti 6y ago
> they’re definitely moving in a little bit of a different direction then where many of us would hope it pains me to say this because I was there for the UniFi glory days: The old Ubiquiti is dead and gone. Almost everyone I know quit. I hope they can land on their feet and return to the glory days but I don't have much hope. The company got toxic fast at the end

teeray 6y ago

What I’m curious about is, if I run my own controller on my own hardware, do I need to be concerned about this? I could understand supply chain concerns... I’ve held off updating anything while this plays out. But all these “breach! breach!” stories fail to spell out who is affected and what they need to do.

gerdesj 6y ago
You probably don't need to be concerned(ish). I run a controller for 32 "sites" across the UK with 1 to 13 APs per site and a few switches. I keep it behind HAProxy but with fairly minimal changes (from memory.) I have stuck with controller 5.13.32 rather than moving to 6.x just yet. It's an LTS version and I'm still waiting for the whinging to stop on the forums. I also watch the AP firmware and that has had some interesting times over the last few months. I've confirmed dodgy AP versions on my sites and backrevved and held accordingly. I treat the whole thing the same way I do any other system. I come out in spots when people mention clouds and IT in the same sentence, so I have not knowingly enabled any cloudy integrations from my controller to UBNT. Specifically, I have not enabled "Remote Access".
ev1 6y ago
Force pushed updates overnight turned local controllers into requiring ui.com single sign on, iirc.
Nextgrid 6y ago
If the compromise is widespread enough then the attackers might have gained control of the update infrastructure allowing them to push out malicious firmware to your devices.
- js2 6y ago
  These blanket statements don’t apply to everyone. It depends which Ubiquiti hardware you own and how you’ve configured it. For example, I run the UniFi controller on my FreeNAS server. There are no forced updates to it. It doesn’t update unless I update it. The firmware on my APs doesn’t update unless I update them from my controller.

brian-armstrong 6y ago

Has anyone looked at Ubiquiti's firmware signing? Would it be possible to patch it to retain the drivers and kernel but replace the configuration layers? Being able to homebrew some config would make the equipment more valuable to us I think.

gertrunde 6y ago
People have been running OpenWRT on Ubiquiti gear for quite a long time iirc. [https://openwrt.org/toh/ubiquiti/start https://openwrt.org/toh/ubiquiti/start]
- Hikikomori 6y ago
  Afaik performance will be abysmal on edge router series as the npu isn't used.
  - KirillPanov 6y ago
    From firsthand experience: performance is in fact awesome on the edgerouters (4, 6, 8, and 12) using plain-vanilla Linux. It's a big honking MIPS chip with firehose connections to the ethernet PHYs. Precisely the kind of device you want for a router.
    - Hikikomori 6y ago
      Then you are better off buying something with a beefier cpu that costs less since it doesn't have an npu.
    - joshspankit 6y ago
      Awesome at what level? Are we talking DPI at 1Gbps symmetric?
    - sigg3 6y ago
      Please elaborate. What are the thruput measurements on OpenWRT when compared to ER-.. stock firmware, with hw accel or with DPI..? I have an ER-4 to be able to use the entire wan connection, but on stock firmware I must disable DPI to enable hw acceleration (otherwise the thruput floors). I don't use DPI atm, so no big loss. Can you utilize the hw accel in openWRT too?

vr46 6y ago

So this week, I have gone from having a single little USG and a massive order planned for loads of kit to stopping them automatically updating the firmware and dropping that order. Extremely annoying, but not as annoying as if this had happened in a couple of weeks.

tifadg1 6y ago
So what are vendor are you changing to now?
- vr46 6y ago
  None! Going to keep my jerry-rigged-Heath-Robinson networks with the existing mesh and switches until things resolve to a satisfactory juncture.
EricE 6y ago
Why anyone has - of all things - automatic firmware updates for networking gear always blows my mind away. Also you can still run all your UBNT kit completely disconnected from the cloud - way better than anyone else. The single management pane is still very useful. Actually I run everything but the USG - their router/firewall is absolutely the weakest link in their product suite. The APs are a good value and solid performers, especially if you stay on earlier, stable firmwares (and DON'T auto update). Their switches are a fair value. Lack of progress with layer 3 routing, especially on their bigger/pro switches is annoying and skews the value prop for those away from UBNT. Cisco's Small Business line is starting to look pretty good and they have a management console that is maturing. And at the rate they are going they should easily be able to catch and blow past UBNT within a year. And so far (despite being from cisco) it's also one of the few product families in this space that does NOT require a subscription.
- vr46 6y ago
  Good points. I got the auto firmware thing wrong, it was actually the controller update, which I run on a Pi. And fair point on running it disconnect from the cloud, which it is now. Going to check out the Cisco stuff.

xvector 6y ago

Ubiquiti has lost my business. And with the recent issues with Netgate/PfSense [1], it looks like OpnSense is the way to go. [1]: https://arstechnica.com/gadgets/2021/03/buffer-overruns-license-violations-and-bad-code-freebsd-13s-close-call/ https://arstechnica.com/gadgets/2021/03/buffer-overruns-lice...

jessebarton 6y ago
why would you not just run OpenBSD with PF.
- bpye 6y ago
  Why should I choose OpenBSD over FreeBSD or even Linux with nftables?
  - dijit 6y ago
    If you’re really asking, and not making a point; PF is created and primarily maintained by OpenBSD OpenBSD’s base system (without extra packages) includes PF and has a focus on security. PF in freebsd is several major versions old. nftables (like iptables before it) is rule based and not bucket based. So high numbers of rules will not affect pf’s performance like it does with nftables. But, for home users, probably not noticeable. Though I prefer the syntax of PF personally.
    - hyperpl 6y ago
      Wireguard has also been stable on OpenBSD which helped me with my throughput on my apu2d router hardware.
    - fuzzy2 6y ago
      Could you expand on what you mean by "bucket based"? Maybe the so-called "tables"? They sound pretty identical to ipset on Linux.
      - 6y ago

rossipedia 6y ago

> Ubiquiti also hinted it had an idea of who was behind the attack, saying it has “well-developed evidence that the perpetrator is an individual with intricate knowledge of our cloud infrastructure. As we are cooperating with law enforcement in an ongoing investigation, we cannot comment further.” I personally don't believe this. IMO, this is a company who is looking for a fall guy, and _most likely_ it's going to be somebody who raised a stink about all the security problems during their time there. Form your own opinion, I'm just a guy who worked at Ubiquiti for a year, raising all kinds of hell about the security, architectural, and operational problems that I saw while I was there. But what do I know...

judge2020 6y ago
Given they were stupid enough to spin up some VMs, I doubt it was someone that knew what they had access to. A skilled attacker would stay dormant sucking up all data accessible via the AWS API (including s3 stuff) and potentially keep access to the infrastructure for years.
- smashed 6y ago
  There is no evidence that this did not also happen.
  - brippalcharrid 6y ago
    And if it is happening, we might hear about that in a few years' time, if it's discovered, and if it's brought to light in circumstances that are conducive to the vendor making a public disclosure (eg. which are impossible to cover up).
- deleted 6y ago
  [deleted]
- throwaway8581 6y ago
  This kind of analysis is basically worthless because you don’t know whether they are operating at multiple levels of deception by, e.g., making you think they are a stupid script kiddie and that you successfully wiped them out.

tpmx 6y ago

By now we'll have to ask: Is it realistic to expect hardware-oriented companies to build secure software? (Yes, Apple exists.)

eertami 6y ago
I think your question is wrong, it should be: Is it realistic to trust any company to build completely secure software? I don't see your point about Apple, unless you're being sarcastic for comedic effect. Apple release software with security flaws too. In fact a zero-click security vulnerability present in the Apple email client was posted on this very page only three days ago[1]. [1]: https://news.ycombinator.com/item?id=26664714 https://news.ycombinator.com/item?id=26664714
ryandrake 6y ago
Most hardware companies don’t care in the slightest about software quality. To them, software is just another line item on the Bill Of Materials, like a bolt or piece of sheet metal. You either have some overworked intern who knows C cobble something together that barely works or you buy it from the least expensive supplier. When the build is ramping, at the end of the assembly line somebody is going to flash something on the device, and they are not going to stop the line to worry about a security hole.
- deleted 6y ago
  [deleted]
jnwatson 6y ago
This wasn't about the security of their "software", as in the thing that's running on your device. This was about their backend security. That's a much, much tougher call to make.
simple_phrases 6y ago
iOS exploits are cheaper than Android exploits because iOS exploits are so plentiful in comparison[1]. [1] https://www.cyberscoop.com/ios-zero-day-zerodium-high-supply/ https://www.cyberscoop.com/ios-zero-day-zerodium-high-supply...

d-funct 6y ago

What no one seems to be really discussing is how paranoid should people be around this breach? Is it a case of you probably want to rebuild machines that have default usernames/passwords? Or is it more whatever can be seen in the Ubiquiti UI might be been accessed by third parties?

rovr138 6y ago
> Is it a case of you probably want to rebuild machines that have default usernames/passwords? I mean, regardless, most probably, the answer to this is yes.

bcrescimanno 6y ago

It's disappointing to see a breach like this and even more disappointing to see what (at least on the surface) appears to be a lackadaisical response. At someone who runs a UniFi network in my home with just 4 pieces of hardware (gateway, wired switch, and 2 PoE WAPs) I'm really curious if there are solid alternatives for a managed home network. UniFi really hit a sweet spot of price/performance that made it a somewhat pricey; but, not totally unreasonable option for the home. Any suggestions from the HN crowd?

e40 6y ago
That is my exact configuration, too. Would love to have alternatives.
- heavymark 6y ago
  I’m not aware of any alternatives that are designed as well, and if you switch the new option could just as easily be hacked or if so it on it could also be hacked but you may never realize. Though it’s good for all these people to pretend to threaten to leave since maybe that will get the company to be a little more forth right which is all we can really ask for these days.
ThatPlayer 6y ago
I've heard good things about TP-Link's Omada series. Their controller even looks like a clone of Unifi's
- ed25519FUUU 6y ago
  Isn’t TP-link a Chinese company?
  - catblast01 6y ago
    Is ubiquiti a Chinese company? Really, what a low effort idiotic post.

deleted 6y ago

[deleted]

ta20210405 6y ago

Amateur hour at Ubiquiti. Sadly they leave us without a decent replacement. Mikrotik is the only contender and they leave a lot to be desired.

dt3ft 6y ago
I recommend fortinet as replacement for USG. Not as cheap, but you get what you pay for.

dmix 6y ago

Looks like a basic ransom request but pushing malware to the ‘85 million’ devices through automated automated would be far more damaging.

smiley1437 6y ago

Anyone know if Apple will be putting out a wifi mesh system, maybe integrated into Homepod Minis? Apple already 'owns' me, I might as well have them run my Wifi too and ditch my unifi gear. At least Apple seems to care about privacy and security, even if it is a self-serving marketing scheme.

lostlogin 6y ago
Their wifi line used to be excellent. Having APs that could be hardwired would be a requirement for me. The less wifi the better.

hrgiger 6y ago

I keep one 6p behind isp router to manage home network, they have good hardware but i didnt like the idea exposing to cloud, only allowed local dns, ntp. And removed all port listeners from ubi in sbin then touched a new file with same name. Latest firmware complained a lot but worked at some point. I am not sure i am fully secure but quite happy with performance

wnevets 6y ago

breaches can happen to anyone however as a customer the way Ubiquiti has been handling this is really disconcerting.

spurgu 6y ago

Am I the only one annoyed with the expression "all but"? To me it sounds like the complete opposite. "All but confirms" to me sounds like they're "doing everything else than confirming" / "all other things except confirming".

CRConrad 6y ago
It means ~"Comes so close to confirming that it does everything up to it, except actually confirming" -- the actual confirmation is all that's missing. Once I realised that, it stopped annoying me. Auttoiko?

TwoNineFive 6y ago

On this subject, does anyone know what is up with the reddit sub, r/ubiquiti? Seems to be run by u/briellie. She(?) seems like a really toxic person with some kind of business relationship with Ubiquiti like a reseller or something. The Reddit sub seems like they are actively trying to suppress discussion of this issue. There's some allegations of censorship on the sub, but I'm not seeing it... which might actually just be confirmation that they are censoring. I don't know.

moxzyros 6y ago
There was a recent discussion[1] on the sub with 1K upvotes and over 500 comments about the breach and I routinely see unabated salty posts and comments about Ubiquiti's downward spiral. I have lurked on the sub for several years (I manage a bunch of Ubiquiti gear) and I never got the impression people were being censored or moderated into submission. Are there any particular examples of suppression or links to the allegations of censorship? The sub did recently begin allowing equipment picture posts again by popular demand. [2] I suppose an uncharitable interpretation is that that move was appeasement to distract from the breach issue. [1] https://www.reddit.com/r/Ubiquiti/comments/mgm4o7/whistleblower_ubiquiti_breach_catastrophic_krebs/ https://www.reddit.com/r/Ubiquiti/comments/mgm4o7/whistleblo... [2] https://www.reddit.com/r/Ubiquiti/comments/mi0679/rule_change_equipment_picture_posts_are_allowed/ https://www.reddit.com/r/Ubiquiti/comments/mi0679/rule_chang... Edit: Formatting

worik 6y ago

I can believe that they do not keep logs of the database access. As brain dead as it sounds. I have been in the position of implementing a client on a API I do not control. The owners of the servers (colleagues but in a different country) do not seem to know what logs are. We get random failures from the server. I can pin down to the second when they occur (not closer because of network lag). I suspect that the server is failing under load, but the way I would find out is to... Read the logs. My foreign colleagues do not respond to me, ghost me entirely, when I ask them to inspect the logs. Perhaps it is a Windows/Azure thing?

jiggawatts 6y ago
Logs are typically off by default in most Enterprise software, or goes nowhere by default which is basically the same thing. Logs cost money to both collect and store. Not everyone is cheerfully burning through VC capital. Some people have budgets. Speaking of log collection, simply dumping the logs into a central repository is the same as taking the garbage to the landfill. Collecting trash just results in a big collection of trash. Extracting useful information from a huge pile of logs is non-trivial. You'd have to know at least one query language, probably several if you work on big enterprise systems. You'd have to know a bunch of esoteric things like how to convert the long decimal strings to a number so that you can interpret as any one of a dozen timestamp formats as an actual datetime in UTC, and then convert that to local timestamp so that you can tell when something actually happened. And so on. All of this is merely a prerequisite for finding a specific instance of what you know is there. You know what you'll never find in logs? Things you didn't know to look for! That's unfortunately about 90-99% of what you actually need to know, making logs typically about 1-10% as useful as you'd like. Did I mention they cost money? Have you seen how many arms and legs Splunk charges these days? Why would you pay that much for something that is less than a tenth as useful as it could be? The fundamental problem is that discipline doesn't scale. You can't expect a hundred thousand IT operations guys to all do everything all of the time, in spite of the non-technical finance guy holding on to the purse strings like he's gripping a life preserver after going overboard in a raging storm. PS: I turned logging on extensively for a recent Azure project. Some logs cost more than the service they were monitoring. I mean sure, I could turn off the unnecessary logs, but how do I know ahead of time which ones will be necessary or not? I don't have a time machine! I can't go back and turn off the logs I won't need... later. PPS: Have you noticed all logging companies charge by the gigabyte? What incentive do you imagine they have for improving the efficiency of the log transfer and storage formats?

lifeisstillgood 6y ago

Off topic but is there a good guide to middle level home network setup - something like using OpenWRT on (Rpis?) and turning that into a router and couple of access points. I was going to press buy on the setup for some ubiquiti products till a couple of days ago :-(

EricE 6y ago
Unifi is still great for APs or switches. Their routers are OK if you just want the basics. If you set the controller up in a VM or raspberry pi you can stay 100% local all the time. Leave auto updates OFF, only upgrade the firmware/controller until you have a real reason to - something important gets fixed, a new feature is released you want to use - that sort of stuff. I have sites that firmware hasn't been updated in years and doesn't need to. It always amuses me in one thread people bitch about MS, Windows 10 and forced updates then in threads like this I read people fessing to having auto updates enabled voluntarily. Ugh. As for alternatives, Cisco's small business line is looking pretty reasonable. You can find used gear on ebay easily, you can upgrade firmware without a subscription or even needing an account (yes, I'm still talking about Cisco!) and while their management console is a bit weak, it's maturing. I think it will mature faster than UBNT who was strong out the gate but delivered nothing new or significant for almost 7 years now in routing, switching has never delivered layer 3 - especially useful for larger switches. APs seem to take longer to get bugs resolved in their firmware - but like I said if you only upgrade when you really need to that won't even effect you.

ghostpepper 6y ago

I would love to see a competitor spring up targeting the same enthusiast/prosumer segment. It seems like there are quite a few ex-employees with knowledge of how to build it.

tgpc 6y ago

Long-time Ubiquiti fan here Their lack of Wifi 6 across the range, and the security problems drive me to look at alternatives. Found the Netgear WAX610. Very happy with them.

aneutron 6y ago

I'm thinking it won't be long before folks roll their own distro of Unifi APs and switches.

jimnotgym 6y ago

Hang on a minute there > Ubiquiti’s IoT gear includes things like WiFi routers I understood IoT to mean wifi toasters, TVs and other home appliances. Since when was a router an IoT device? Are we going to call all nework devices IoT now. This strikes me as taking rather too much journalistic license. In fact wtf is a WiFi Router. I use Unifi to deploy Wireless Acess Points on a LAN with centralized control. It is possible to do this without them having internet access at all, but it makes it rather harder to update everything. This is miles away from IoT. Describing Ubiquity as a IoT company is like calling Cisco, Juniper, Mikrotik and Aruba IoT companies. This sounds like an attempt to feed the narrative that the IoT is going to eat us alive. Let us focus instead on what Ubiquity actually did wrong, isn't that bad enough?

EricE 6y ago
Indeed - throwing around the IoT buzzword is just an unnecessary (and irrelevant) distraction.
abestic9 6y ago
I believe the IoT distinction is that the platform reports to a cloud service, where parts of that are implemented as SaaS, if only for SSO and remote management. I think this is a fairly loose definition though if the underlying device data is still on-prem. Wi-Fi router I would take to mean one of those consumer all-in-one boxes, which is what the UDM fits the description of. I agree that this distinction is unnecessary. When we consider who wrote the article and who it's for, we should expect a bit more granularity than foggy acronyms.

GekkePrutser 6y ago

Ubiquiti should really stop making cloud logins mandatory. The latest stuff (UDM/UDM Pro, Cloud Key G2) must be connected to their cloud at installation time. Remote access can be turned off but an admin account connected to their cloud remains. Without those ties to their infrastructure, this breach would not be as severe. It would just cause an attacker to see what I've bought from them, nothing else. I'm glad I can still use the unifi controller in docker without any ties to UI.com however their later stuff like Unifi protect, access, talk etc no longer works with that.

ex_ubiquiti 6y ago
I worked there and I didn't even understand why we had to force cloud logins on Dream Machine. In the early days we were all about letting people run their own controller hardware and not requiring cloud logins. No one could ever tell us why we had to force everyone to the cloud. It was a mandate from above
- GekkePrutser 6y ago
  Thanks for that insight! I guess it's for the usual reasons. Telemetry / product improvement, and also more marketing data. Data is the new gold :)
- EricE 6y ago
  They utterly flushed away their biggest selling point - the only vendor in that space that didn't require a cloud login. Beyond stupid.

dustinmoris 6y ago

I find it really strange that so many claim that they need Ubiquiti and that there is "sadly" no other good alternative. What are people doing with their home networks? What are they comparing it with? Has anyone actually tried some of the mesh networks from TP-Link or other brands? I have one at home and honestly I don't even know what the admin management looks like because I never have to go there and do something. What are people doing? Is it that I am so ignorant to some needs which people have that they constantly need to tweak their networks at home or is it just a symptom of Ubiquity kit that requires users to constantly do something with it that now they think they need all that fancy management stuff because they got used to do so much maintenance work on something that should just work without ever having to touch it again?

MindTooth 6y ago

After seeing that they did not capture the logs. What is the “proper” way of storing said logs? I guess you need a remote logserver like logstash to store them. But what service does actually send the logs from the server to a central storage. Looking into Loki, Graphite, etc. But I’m a bit at a loss where to begin.