22 ms·
Hackers Were Inside Citrix for Five Months
- moepstar 7y agoWhat i find particularly embarrassing for Citrix (although only marginally touched in the article) is the amount of time taken for them to close the hole that was in their Netscaler/ADC components. I mean, this is not a one-man show and an open-source project....
- thedance 7y agoHow do you know? There are lots of commercial products that have no staffing at all. "1 man" might even be above the median staffing.
- montalbano 7y agoCitrix had 8200 employees as of 2018 according to Wikipedia. https://en.wikipedia.org/wiki/Citrix_Systems https://en.wikipedia.org/wiki/Citrix_Systems
- thedance 7y agoI meant the NetScaler project in particular.
- frollo 7y agoThey still have nearly 10k employees and are a big, affluent company. None of their project should be a one-man project, but, even if it were, they should still be able to find enough engineers to work on it in case of emergencies. If it can be done by a small startup with a total of 3 devs in the entire company (and I have seen it done), it can be done by a company the size of Citrix.
- thedance 7y agoI guess you'd be surprised at how easy it is for an organization to just de-staff a launched product.
- throwiay987 7y agowhy are we defending gross incompetence and laid back attitudes again?
- mynameisvlad 7y agoWho is defending, exactly? thedance is pointing out that we have no idea of the inner workings of Citrix, and how they staff their projects. It's not completely unreasonable to believe a non-core project has minimal staffing levels.
- Spooky23 7y agoThe vulnerability in Netscaler was an old bug in a web server that let you trivially break out of the virtual directory. So you could navigate to <URL>/vpn/../path and read/write files. The device is a freaking load balancer. It took the company a month from public disclosure to push out a rule to workaround to stop most exploits on some devices, and 6 weeks to patch it. Something went horribly wrong. Netscaler was a potential spinoff product awhile back — my guess is that they stripped the division in the anticipation of a sale to make the numbers look good.
- streb-lo 7y ago> In March 2019, the Federal Bureau of Investigation (FBI) alerted Citrix they had reason to believe cybercriminals had gained access to the company’s internal network. The FBI told Citrix the hackers likely got in using a technique called “password spraying,” a relatively crude but remarkably effective attack that attempts to access a large number of employee accounts (usernames/email addresses) using just a handful of common passwords. Pretty bad when the FBI has to step in and alert you that someone has brute forced their way into your servers.
- zigzaggy 7y ago(This is slightly off the topic of this story) Believe it or not this happens often. I work with R&D / high-tech small businesses that have 1-2 patents or some other IP. They provide services to the government but they aren't exactly flush with cash - meaning their infosec programs are slim. So they stay in contact with the FBI and other TLAs (three letter agencies) who will contact them from time to time to say "hey you may want to check your network, we think someone's broken in." Basically the FBI is the internet police for these infrastructure / science / tech / etc. firms. It's not hard to understand why this information isn't out on the street more.
- stingraycharles 7y agoYes but we’re talking about Citrix here, not some random small business supplier. In Citrix’ case it is bad to be in the position that the FBI had to tell you someone was brute forcing your app.
- deleted 7y ago[deleted]
- zigzaggy 7y agoTrue, and I agree it is bad. But I stand by my original comment. I believe all this happens far more often than we know right now. I predict we'll be finding out in the not so distant future that we've all been targeted and breached. All our data are belong to them.
- jerry1979 7y agoI have always felt very uneasy about Citrix in the workplace, especially around PHI.
- Swtrz 7y agoI know of...three major HI providers in the midwest using citrix
- noident 7y agoI was relieved to see that only internal employee information was impacted. You don't even want to know how many banks, hospitals, and power plants rely on Citrix Receiver for remote desktop access.
- tyingq 7y ago"I was relieved to see that only internal employee information was impacted" Believing that puts a lot of credence in their analytical/forensic/security skills. Which doesn't align well with "inside Citrix for Five Months".
- XOPJ 7y agoI felt like this article was implying that other firms are claiming this is bigger and deeper with impact to their clients. It looks like maybe Citrix being exposed to their internal networks also impacted their customers but wasn't originally disclosed. Maybe they didn't do enough analysis after the breach to discover this themselves?
- jens-h 7y agoOn its own website, Citrix is using the real costs of a data breach as an argument to buy its products: https://www.citrix.com/de-de/products/citrix-workspace/resources/weighing-the-risks-infographic.html https://www.citrix.com/de-de/products/citrix-workspace/resou... But because of the widespread use of the software, a recent Citrix vulnerability puts 80K companies at risk: https://www.infosecurity-magazine.com/news/citrix-vulnerability-puts-80k/ https://www.infosecurity-magazine.com/news/citrix-vulnerabil... It is always dangerous if a single company has a monopoly.
- deleted 7y ago[deleted]
- arminiusreturns 7y agoOne thing I have learned as a sysadmin who has had the privledge to see the inside of hundreds of companies from medium sized law firms to F500 oil companies: There is a lot more incompetence than you would ever want to believe, and it's not always where you think. I've traced most of it to a failure of connection/communication between IT departments and C-levels/boards. The CTO/CIO and the person immediately below them (and the person immediately below them) are the "buck stops here" people for these kinds of issues, but often are either one of two types. 1) Too much MBA, not enough tech. 2) Too much tech, not enough MBA. Both tend to have pretty similar results.
- stcredzero 7y agoThe CTO/CIO and the person immediately below them (and the person immediately below them) are the "buck stops here" people for these kinds of issues, but often are either one of two types. 1) Too much MBA, not enough tech. 2) Too much tech, not enough MBA. 1) Knows how to get people to do things, but doesn't know what needs to be done. 2) Knows exactly what would fix security, but doesn't know how to get people to do those things.
- 1000units 7y agoMBA types are great at conning others into self-serving false insights like (2) isn't perfectly capable of enforcing sound security policies given the appropriate authority. Their jobs are directly about justifying their own existence, usually by song and dance.
- kevin_thibedeau 7y ago1) Has been told what needs to be done and decided to nix the idea because security is a cost center that cuts into their bonus potential.
- mox1 7y agoEh, while this happens its getting rarer and rarer. Execs know that if a cybersecurity issue is brought to them they kinda have to take it seriously (in 2020 anyways) or be the next Experian. Edit: whoops Equifax, not Experian
- PaulHoule 7y agoPeople still use Citrix?
- westmeal 7y agoHonestly, you would be surprised how many giant companies still use Citrix. Mostly medical. Terrifying.
- dman 7y agoAnd financial companies
- gyc 7y agoAnd law firms big and small.
- asmr 7y agoHospitality as well.
- jacquesm 7y agoAnd places that use RDP like setups for development because they don't want their employees to walk out with the crown jewels on their laptops.
- PaulHoule 7y agoWhy not just use RDP? It is because it as hard to get an enterprise to stop paying for a software license as it is to get one to start?
- technion 7y agoI've sat in many pitches where, ironically, the argument is that Citrix is more secure. We use multiple desktop products where the modern cloud version was literally just a Citrix cluster per customer, running the same old app. It's absolutely endemic across enterprise products as a "cloud solution".
- euroclydon 7y agoHow much source code was committed in that span? Time to audit it all. Plus the binaries and other resources that are pulled off network shares at build time. Plus the compilers...
- jmiskovic 7y agoYou already hinted at it, but anyone interested in computer security should read Ken Thompson's short but majestic "Reflections on trusting trust".
- s_dev 7y agoStories like this really show Citrix and Cisco are so far behind Huawei. If Huawei chips were similarly vulnearable the CIA and FBI would be disclosing this as vocally as possible.
- yourcssisbroke 7y agospotted the commie!
- whatsmyusername 7y agoHow's the weather in Beijing?
- vb6sp6 7y agoThe FBI or CIA would keep any vulnerabilities secret. But here is a recent example: https://www.businessinsider.com/us-accuses-huawei-of-spying-through-law-enforcement-backdoors-2020-2 https://www.businessinsider.com/us-accuses-huawei-of-spying-...
- Stierlitz 7y agoIf the hack was conducted through the use of account hijacking then why didn't anybody at Citrix notice. Unless once the hackers got in they migrated within the network using as yet unknown vulnerabilities. Makes one wonder of there are backdoors in all networking equipment. So as the various state security entities can keep an eye on us.
- amaccuish 7y agoI've always found Citrix perform way better than RDP (feels more responsive, handles multimedia better) I don't understand how after all these years and being originally based on the same tech, how RDP hasn't caught up.
- rodgerd 7y agoMaybe Microsoft have been spending time on trying to improve security.
- Spooky23 7y agoMicrosoft does a mediocre job with RDP because windows pc was the cash cow. Pretty sure that Citrix invented RDP and licensed it back. Now they are changing, and are partnering with Citrix and VMWare in Azure. Eventually, they’ll crush both.
- LilBytes 7y agoPCoIP has been a huge shift in the landscape for remote access protocols for this reason.
- jokoon 7y agoWell, no official government-supported agency have even stepped in to establish a list of norms involving software security, to force large corporations to abide by them. While the private sector is the sole responsible for their own cyber security, and while the NSA wants to keep the upper hand in cybersecurity by holding a cyber-weaponry supremacy, events like these will keep happening. Cyber chaos will continue because the NSA is obviously holding massive advancements in cyber weapons. The day the NSA will have an adversary that can be at least 50% as good as the NSA, you can be sure you will see cyber security standards being passed into law. If you think about it, having cyber supremacy is a good way to have total power over the world. When you have all the information, you have everything you need to do whatever you want. That sort of describes the US right now.
- idoh 7y agoThe government does actually have security policies / norms, see NIST 800-171 as an example.
- Craighead 7y agoThis is a bad take. Russian exploit-to-admin rights is 12 minutes
- xp84 7y ago>list of norms involving software security These norms are already well-known. However they are never followed 100% because they all rest upon a sandy foundation of: "Don't be clueless." Social engineering/phishing will always eventually work on someone somewhere in a company with more than 20 employees. Not to mention the other part, where passwords are allowed to be a vital link in the security chain, yet software vendors like Citrix simultaneously discourage password managers by getting in the way of using them and by forcing perfectly good passwords to be cycled endlessly. Resulting in people using passwords like Citrix20! (will change to Citrix21! in 90 days... iron clad security there guys.)
- blintz 7y agoI hope we can move to a world with ubiquitous two-factor and hardware roots of trust (FIDO2, U2F, etc) across enterprises. That is the only way I see things like this ending.
- anonsivalley652 7y agoThe core of OSes need to be treated more like read-only firmware that only gets updated as-needed and is unable to be over-written by itself, e.g., send a request to the BIOS to look for a valid public-key signed image file to be applied on reboot. Flash is so cheap, 64 GiB mirrored SSD devices should be available for operating system images on system boards. Leave OS images as signed squashfs files on a dumb flash FS like exFAT. Delta updates can be applied by stripping-out entropic-metadata, patching and recompressing a previous release to arrive at the valid signature of a complete latest release. Mixing operating systems, configuration, programs and user data in together is a recipe for fail.
- H8crilA 7y agoIsn't this normal? I mean after you break in keeping a low profile and staying undetected for as long as possible sounds like a no brainer to me. I wouldn't be surprised if some APTs were inside some "worst" companies for even 2 years at a time. Edit: from the Wikipedia page on Advanced Persistent Threats: > The median "dwell-time", the time an APT attack goes undetected, differs widely between regions. FireEye reports the mean dwell-time for 2018 in the Americas is 71 days, EMEA is 177 days and APAC is 204 days.[4] This allows attackers a significant amount of time to go through the attack cycle, propagate and achieve their objective.
- deleted 7y ago[deleted]
- tomrod 7y agoI look forward to seeing this covered on Darknet diaries. Just found the podcast and it rocks! Having used Citrix this really, really doesn't surprise me.
- aSplash0fDerp 7y agoSo instead of the old days of having a sign that says "We haven't had a workplace accident in XX days" they need "We haven't had a security breach in XX days". How times change...
- stef25 7y agoNot sure if there's more breaches or just more exposure because people started caring. Reading the The Cuckoo's Egg at the moment, apparently it was common in the 1980's for military networks to have guest/guest logins, or even ones with no passwords at all.
- justlexi93 7y agoNetworking software giant Citrix Systems says malicious hackers were inside its networks for five months between 2018 and 2019, making off with personal and financial data on company employees, contractors, interns, job candidates and their dependents.
- anonsivalley652 7y agoI worked at a big name university in the IT department for housing and dining. Long before I got there, one of the Oracle database servers for meal-related activities had been pwned for years because it hadn't been behind a firewall and it had a routed public IP address. It was running Windows so it had accumulated a number of interesting malware including obscure rootkits with no antivirus patterns. I once booted it up off of clean media, ran some forensics tools and found a warez dumpsite on it. This box "couldn't be down" so all the happened to it was it was place behind a bidirectionally-restricted firewall. It still kept limping along with funky malware because they didn't want to spend time or money fixing it. Sigh. If it were my box, it would've been an immediate disconnection, image hardening, wipe and reinstall from backups (data-only). I remember sending some binaries and other deets over to Mark Russinovich at then SysInternals, who's now the CTO of Azure.
- doublerabbit 7y agoWouldn’t even do backups, you couldn’t trust them. If a box is hacked, it’s hosed. Cast fire and rebuild start again.
- tinus_hn 7y agoFrom what I’ve heard their advice on securing servers running their software used to be to ‘make good backups’. I’m not surprised by an incident like this happening to cowboys.