7 ms·
Barracuda urges replacing, not patching, its email security gateways
- mikece 3y agoCareful what you ask for Barracuda: we were already looking at replacing ours with non-Barracuda hardware and now that you are saying old gear cannot be updated all I can say is thank you for making my case to my managers.
- jokowueu 3y agoThey note that "In a statement, Barracuda said it will be providing the replacement product to impacted customers at no cost" Would you still switch to a non barracuda device ?
- insanitybit 3y agoIf you have to go through the cost of a replacement anyways - as in, the time and effort to actually do it - the cost of the hardware matters a lot less.
- Urist-Green 3y agoI suspect there is a higher level question at play stemming from the "deny" comments. Do you think Barracuda is uniquely vulnerable to these types of threats and, if not, would their competitors make similar efforts to recover?
- veave 3y agoIf this is the mindset of some people it's no wonder that companies would rather deny the problem and stick their heads in the sand.
- insanitybit 3y agoThe mindset that changing a system out has risks and that engineers/ops time is very expensive? I mean, I hear you, it sucks that being honest about the issue is leading to punishment, but rationally this is just how people are going to end up responding. If they really want to avoid this, don't give it to them for free, pay them for it.
- kortilla 3y agoNo it’s not. Considering changing vendors because a vendor did the right thing and sent out free replacements for potentially compromised hardware is not something any sane client would do.
- genocidicbunny 3y agoThe GP didn't say that though. They said they were already considering replacing one vendor with another, prior to this incident. As I understood it, the calculus was 'do nothing and keep the current barracuda stuff for a while longer or find another vendor and replace the barracuda stuff?' Now the former option is gone, and the calculus is 'replace current barracuda stuff with barracuda or another vendor?' So if you were already considering switching vendors, now is an opportune time to do so. You're going to be paying the eng/ops cost now anyhow, and as many in this thread have pointed out, one or two fairly off-the-shelf hw appliances are not exactly a big spend.
- kortilla 3y ago>You're going to be paying the eng/ops cost now anyhow, No you’re not. The people in this thread saying that have never actually managed technology because a vendor change that also requires a hardware swap is significantly more expensive than the drop in hardware swap. The only exception to this is if you were already in the middle of the transition and were planning to phase it out shortly anyway.
- kortilla 3y agoThis notion is detached from reality. A drop-in replacement of the same thing is an order of magnitude less work than switching vendors. Even adequately researching alternatives, let alone doing some trial testing, will take far more time than a swap.
- insanitybit 3y agoThey've already been researching alternatives.
- phpisthebest 3y agothe physical cost of the equipment is often the cheapest part of a deployment.
- MattGaiser 3y agoGiven that the replacement will be free, doesn't that make your case harder to make?
- blantonl 3y agoHe's already made up his mind, so he'll probably just leave out this important tidbit when discussing the issue with "management"
- katbyte 3y agoIt’s certainly not free to arrange getting the new device, swap, reconfigure, and hopefully you have HA to avoid downtime.
- JumpCrisscross 3y ago> certainly not free to arrange getting the new device, swap, reconfigure, and hopefully you have HA to avoid downtime Compared to migrating to a new vendor?
- thanksgiving 3y agoIt is an email server. You can set up some kind of system that “catches” incoming email while you take your hardware offline and replace it. Your internal users won’t be able to send or receive emails but anyone outside the organization would be none the wiser that your system was down. If you notify ahead of time and can keep downtime to under an hour, will it matter?
- loloquwowndueo 3y agoWhich you would need anyway if you switch to hardware from a different vendor.
- vyst44 3y agoCan I ask - which competing devices are you planning to replace these Barracuda email security gateway servers with? Thank you.
- technion 3y agoA large portion of organic at this point run their mail with Google or Microsoft, and are already paying for capabilities in those systems on par with third party offerings. There is often a counter "but I still get some spam with exchange online/google workspace", to which I would counter that you will still get spam with a barracuda.
- throwaway98345 3y agoYou’ll get false positives and negatives with any email gateway - even ML or “AI” backed ones. The offerings from both Google and Microsoft, while adequate, are someways off the efficacy of other solutions.
- atesti 3y agoIs the message archiver also affected?
- upon_drumhead 3y ago> “No other Barracuda product, including our SaaS email solutions, were impacted by this vulnerability,” the company said.
- nubinetwork 3y agoWe know already... https://news.ycombinator.com/item?id=36061772 https://news.ycombinator.com/item?id=36061772 https://news.ycombinator.com/item?id=36136705 https://news.ycombinator.com/item?id=36136705 https://news.ycombinator.com/item?id=36143926 https://news.ycombinator.com/item?id=36143926 https://news.ycombinator.com/item?id=36156908 https://news.ycombinator.com/item?id=36156908 https://news.ycombinator.com/item?id=36233472 https://news.ycombinator.com/item?id=36233472 https://news.ycombinator.com/item?id=36238822 https://news.ycombinator.com/item?id=36238822 https://news.ycombinator.com/item?id=36248328 https://news.ycombinator.com/item?id=36248328 https://news.ycombinator.com/item?id=36255901 https://news.ycombinator.com/item?id=36255901 https://news.ycombinator.com/item?id=36261519 https://news.ycombinator.com/item?id=36261519 https://news.ycombinator.com/item?id=36263581 https://news.ycombinator.com/item?id=36263581 https://news.ycombinator.com/item?id=36267639 https://news.ycombinator.com/item?id=36267639
- sbierwagen 3y agoNone of those submissions have more than 13 points.
- deleted 3y ago[deleted]
- throwaway54_56 3y ago"We" doing some heavy lifting here
- DirectorKrennic 3y agoAh, the Royal "We".
- dools 3y agoThe editorial
- Prickle 3y ago
- wazoox 3y agoWhat's funny is that obviously, the Barracuda "appliance" is a standard Supermicro 2U server :)
- ocdtrekkie 3y agoThis is often the case. VM appliances are all little Linux VMs where the user just doesn't have access to the internals. Heck, that's basically Docker containers too. Also, Barracuda cheaps out on these massively especially at the low end. I've seen, in the modern Core i years, Barracuda 1U appliances powered by Pentium III processors. I suppose they are powerful enough for the job Barracuda is asking of them, but it's worth a chuckle to see how many years old the chipset they're shipping is. The hardware is absolutely the cheapest part of the stack for them.
- znpy 3y agoI wonder what bcantrill’s take on this will be. He’s been on his personal holy war on firmware for years now, I’m not joking, I’m curious to read his opinions on this issue. Maybe barracuda could use some kind of standalone 2u oxide server instead of supermicro servers? ;)
- brirec 3y agoWith all due respect (and I really mean that — bcantrill is absolutely deserving of tremendous respect), why would using an oxide server for the hardware be any different (better or worse) than a SuperMicro server? And further, is it even possible to get Oxide equipment yet? Is there even a timeline? Or is it still vaporware?
- astrange 3y ago> why would using an oxide server for the hardware be any different (better or worse) than a SuperMicro server? Secure hardware can attest that its firmware is what it should be and doesn't have any persistent implants in it.
- yjftsjthsd-h 3y agoThen something has to safeguard that firmware.
- astrange 3y agoI don't know if you're trying to prove secure hardware is impossible by induction or something, but it does work.
- deleted 3y ago[deleted]
- bcantrill 3y agoA lot of questions in there! Taking these in order: 1. We aren't making standalone servers: the Oxide compute sled comes in the Oxide rack. So are not (and do not intend to be) a drop in replacement for extant rack mounted servers. 2. We have taken a fundamentally different approach to firmware, with a true root of trust that can attest to the service processor -- which can turn attest to the system software. This prompts a lot of questions (e.g., who attests to the root of trust?), and there is a LOT to say about this; look for us to talk a lot more about this 3. In stark contrast (sadly) to nearly everyone else in the server space, the firmware we are developing is entirely open source. More details on that can be found in Cliff Biffle's 2021 OSFC talk and the Hubris and Humility repos.[0][1][2] 4. Definitely not vaporware! We are in the process of shipping to our first customers; you can follow our progress in our Oxide and Friends podcast.[3] [0] https://www.osfc.io/2021/talks/on-hubris-and-humility-developing-an-os-for-robustness-in-rust/ https://www.osfc.io/2021/talks/on-hubris-and-humility-develo... [1] https://github.com/oxidecomputer/hubris https://github.com/oxidecomputer/hubris [2] https://github.com/oxidecomputer/humility https://github.com/oxidecomputer/humility [3] https://oxide-and-friends.transistor.fm/ https://oxide-and-friends.transistor.fm/
- r3trohack3r 3y agoThis actually seems great. They’re taking the hard path here admitting the devices are compromised, that the state can’t be trusted, and that the machine needs to be tossed. Then they’re offering to replace the vulnerable devices. I’m used to companies taking the “deny, deny, deny” route and these articles are then written by their community saying the response is insufficient.
- xen2xen1 3y agoAgreed. Better response than normal. Also gives them an avenue to see how many have been actively used? Maybe that's just me.
- stefan_ 3y agoWho would want a replacement "email security gateway" after the previous version was found to do the equivalent of system(<untrusted email input>)? When is our industry going to agree on a standard where we go "no, that is too dumb, too far"? This "oh there was an issue but it's fixed now" doesn't begin to address the severity of the fuckup this is.
- Hnrobert42 3y agoIt doesn’t take much for a machine to become irrevocably tainted, so the fact that they are doing a recall doesn’t, in and of itself, say much about the magnitude. Do you have some other data to substantiate your ire or are you just piling on?
- fsh 3y agoFrom the security advisory: > The vulnerability stemmed from incomplete input validation of user supplied .tar files as it pertains to the names of the files contained within the archive. Consequently, a remote attacker could format file names in a particular manner that would result in remotely executing a system command through Perl's qx operator with the privileges of the Email Security Gateway product. I would not be comfortable to continue using a product that contained such an egregious flaw.
- Ccubidu 3y agoSeems like they can only patch the application remotely - and not the OS - at least not at the same time (patch OS - then after reboot - let the app look for updates. The time between could have been enough to compromise it again)
- jhugo 3y agoAlternatively, they can patch both but the malware achieves persistence in firmware. Which may not even be a sign of particular sophistication, depending on what protections Barracuda had in place.
- dezgeg 3y agoOr it could be as simple as that the software update application on the device has been patched to re-add the malware after flashing the software update.
- PragmaticPulp 3y agoBarracuda is providing replacement hardware at no cost. This critical piece of information is buried halfway down the article: > In a statement, Barracuda said it will be providing the replacement product to impacted customers at no cost Obviously the time and effort to replace a device isn't free, but at least they're doing the right thing by acknowledging the issue and doing what they can to fix it definitively.
- samstave 3y agoWay cheaper than a lawsuit! But correct - its the right thing to do regardless of the true, ultimate motivations... But what I'd like to know is the impact of this? Like - how much corporate opportunity loss may have been created through information breaches which were unknown... That will never be known, unless we can assuredly say 'none' which is doubtful... - >Barracuda said the vulnerability existed in the Barracuda software component responsible for screening attachments for malware Heh -- uhm... Isn't that like _THE_CORE_ component of the devices job? Aside from ensuring filters on attachement egress blah blah... -- >No other Barracuda product, including our SaaS email solutions, were impacted by this vulnerability I'd like to hear them directly say that this specific ESG device line was NOT used in their Email SaaS offerings? And to know exactly what ESG kit they are using? Seems like a reasonable request if you're a large (or any) customer of theirs...
- ipython 3y agoYou’d be surprised how many “virus scanners” are implemented with poorly coded C parsers running as privileged user with no sandboxing… just saying, kind of asking for trouble.
- nickpeterson 3y agoI remember awhile back the lead of the chrome dev team getting into a twitter fight with some engineer from a popular antivirus tool because, as I recall, AV tools are basically just more attack surface. I tend to agree.
- nickdothutton 3y agoGiven that this looks to be just a particular build of Supermicro server, I wonder what mechanism the malware uses to achieve persistence such that a reformat or FS restore wouldn’t take care of it. Does anyone know if these devices have supermicro IPMIs on them? Those are notoriously insecure (like most lights out managers) and a great place to hide malware persistently. Edit: Typo.
- rvba 3y agoIs this the same super micro that hadd physical backdoors inplemeted by china? https://www.bloomberg.com/news/features/2018-10-04/the-big-hack-how-china-used-a-tiny-chip-to-infiltrate-america-s-top-companies#xj4y7vzkg https://www.bloomberg.com/news/features/2018-10-04/the-big-h...
- ocdtrekkie 3y agoYes except that article has remained wholly unsubstantiated by facts for five years. It seems to have been bunk.
- greggsy 3y agoThat article captured the imagination of the public, but has been derided by any and every cyber security professional worth their salt almost from day dot. Despite this, Bloomberg have refused to retract it or supply any credible sources, presumably because and continues to draw traffic.
- technion 3y agoAnd yet "do we have any backdoored supermicro hardware" is a common question from non technical management, even today. It's infuriating that it hasn't been retracted as I'm still talking about it. It's also at the forefront of insurance questions
- RealityVoid 3y agoJust goes to show how much of this word is theater rather than pragmatic improvement.
- shrubble 3y agoIs it likely they will analyze the firmware on the ones that come back, then wipe them and resell them either thru a refurbisher or even put the new software on the known-good server and use elsewhere, such as in the datacenter where they have their SaaS offering?
- jokoon 3y agoWhy doesn't the NSA just AUDITS actual source code of the software that it deems it deserves to increase its security? I mean it's obviously an interest to national security. They could do that while maintaining the backdoors they want to keep so they can have an edge on cyber warfare.
- EdwardDiego 3y agoWhat makes you think this wasn't the NSA?
- jamiek88 3y agoNSA can’t even stop their own tools from leaking, never mind take on this.
- _kbh_ 3y ago> NSA can’t even stop their own tools from leaking, never mind take on this. The NSA has kept plenty of vulnerabilities under wraps, the tool leaks were a pretty rare occurrence of the KGB/GRU trying to burn NSA ops without directly involving themselves.
- hgsgm 3y agoWhy would a vendor give their source code to the NSA to attack and also risk leak? The NSA does not protect the public's computers, they attack them.
- xen2xen1 3y agoTechnically, they do both. Very confusing.
- bluedino 3y agoOne of the first things I virtualized was a Barracuda spam appliance, the fans were extremely loud and the AMD chip was slow, so the interface to your spam inbox was slow, etc. Pulled the hard drive and loaded it up into VMmware ESX 2(?). It flew and there was one less server/appliance in my rack. Barracuda was one of this early companies that made a very useful product out of free software. I idolized that and wanted to work there. Then they were taken over by some outside company, started making all kinds of products that weren't in their core, support faltered...I'm kind of surprised they are still around.
- esjeon 3y agoThe vulnerability found was an RCE, and apparently at least 5% of their devices are infected. > The pivot from patch to total replacement of affected devices is fairly > stunning and implies the malware the threat actors deployed somehow achieves > persistence at a low enough level that even wiping the device wouldn’t > eradicate attacker access. Very interesting. I'd like to think it's probably some technical limitations in the recovery options rather than some weird magickery to persist beyond drive reset. Because, if it's not, everyone is screwed.
- sacnoradhq 3y agoThis is a problem of bad engineering: not using an immutable OS firmware model with a spare. There ought to be 3 physically-separate flash devices: bootloader (never changed), OS (managed by bootloader - updates deleted unless signed correctly), and data (wipeable by a physical reset button). If you don't do this, then you're inviting implants a priori.
- tiberious726 3y agoAnd what do you do when a bootloader vulnerability is discovered then?
- libpcap 3y ago>“One of the goals of malware is to be hard to remove, and this suggests the malware compromised the firmware itself to make it really hard to remove and really stealthy,” Weaver said. “That’s not a ransomware actor, that’s a state actor." China?
- Woodi 3y agoNice, first time in my life I consider buying something from Barracuda. But looks they sell to 1500+ desktops && 5+ officess. Oh well... ;) But: "somehow achieves persistence at a low enough level that even wiping the device wouldn’t eradicate attacker access" It started. Barrakuda at least is nice to replaces hw but soon vendors will be bored by replacing shit they themselve making and customers will be left with voulnerable systems. Simple x86/PC ecosystem is shit, intentionally crafted or not. And, very, very sadly on whole globe only EU is capable of enforcing something to be better. But chances are minuscule...
- rustybolt 3y agoThis could have been a software update if they had used an FPGA.
- Kwpolska 3y agoWhat does this appliance do? Is it just a SMTP proxy with a virus scanner and spam filter? How is it better than software filters on the mail server?
- denton-scratch 3y ago> SMTP proxy I don't think it's a proxy; it's an ordinary SMTP server, isn't it? All SMTP servers can be configured as relays. I imagine people buy these Barracuda appliances for CYA reasons, and because there's a support contract.
- londons_explore 3y agoIf I'm reading between the lines correctly... After a device is compromised, patching it won't help. Their software team (quite reasonably) says that after anyone else has got root on the system, their software patch system probably can't reliably be sure to get rid of the malware. So, they are offering to replace any device in this position. The replacement is probably a refurbished one from another customer, but has gone back to the factory to be flashed by some debug port - and using that you can wipe all the flash memory and be pretty sure there is nothing evil left.