13 ms·
Apt Encounters of the Third Kind
- gue5t 6y agoPlease change the posting title to match the article title and disambiguate between APT (Advanced Persistent Threats, the article subject) and Apt (the package manager).
- jwilk 6y agoFWIW, the package manager is also spelled APT.
- gue5t 6y agoYou're right... what an annoying namespace collision. On the other hand, stylizing software as Initial Caps is much more acceptable than stylizing non-software acronyms that way, so it would still be less misleading to change the capitalization.
- IgorPartola 6y agoWould you say that these things aren’t aptly named?
- diogenesjunior 6y agoha
- milliams 6y agoEspecially where the article doesn't define it or even use the term "APT" except in the title.
- diogenesjunior 6y agoI thought we couldn't edit titles?
- kencausey 6y agoYes, the poster can for a limited time, 2 hours I think.
- anamexis 6y agoAlso the mods can and often do.
- lormayna 6y agoPoster here. Do you think I need to edit the title? This title was funny to me, but probably just because I am a security guy and I know what is an APT.
- airstrike 6y agoIt should match the original, unless you have a strong reason not to i.e. it breaks the guidelines somehow https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- 8note 6y agoSwitching Apt to APT would add lot of clarity while barely changing the title
- dopidopHN 6y agoThanks, I don’t work in security but I use APT a lot. I thought it was a unfunny joke? Like ... APT provide some of those packages? Ok. That make more sense. The author did a good job at making that readable. Is it often like that?
- NotEvil 6y agoMost security analysis are easily readable (the ones I read were) but there may be outliers
- tgv 6y agoAnd I thought it was the common English word "apt". It's a trivial ambiguity, not click-bait.
- h2odragon 6y agoCNA? > On March 21, 2021, CNA determined that it sustained a sophisticated cybersecurity attack. The attack caused a network disruption and impacted certain CNA systems, including corporate email. Upon learning of the incident, we immediately engaged a team of third-party forensic experts to investigate and determine the full scope of this incident, which is ongoing. + [CNA suffers sophisticated cybersecurity attack](https://www.cna.com/ https://www.cna.com/)
- FDSGSG 6y agoNo, CNA was hit by ransomware.
- mwcampbell 6y agoI'm trying to find the lesson in here about how to prevent this kind of incident in the first place. The nearest I can find is: don't build any production binaries on your personal machine.
- rcxdude 6y agoReproducible builds can go a long way, along with a diverse set of build servers which are automatically compared. Whether you use your personal machine or a CI system there's still the risk of it being compromised (though your personal machine is probably at a little more risk of that since personal machines tend to have a lot more software running on them than CI systems or production machines).
- h2odragon 6y agoI'm paranoid, and I'd have considered the efforts described here to be pretty secure. I'll say the only counter to this grade of threat is constant monitoring, by a varied crew of attentive, inventive, and interested people. Even then, there's probably going to be a lot of luck needed.
- zozbot234 6y agoOne sensible mitigation to this grade of threat; avoid running Windows, even as a VM host as the dev did. It's a dumpster fire.
- xmodem 6y agoI think you may have misinterpreted that part of the post - my understanding is that the Linux laptop that was being used was compromised, and there was a 3 month gap when that developer switched to a Windows machine before that became compromised too. Specifically it would be fascinating to learn whether the Windows host was compromised or if it was only the Linux VM.
- ducktective 6y ago
- motohagiography 6y agoSuperb work. The "who" of attribution is more likely related to the actual PII they were after than any signature you'll get in the code. Seems like a lot of effort and risk of their malware being discovered for PII instead of being an injection point into those users machines. I rarely hear security people talk about why a system was targeted, and once you have that, you can know what to look for, inject canaries to test etc.
- afrcnc 6y agoFrom Twitter chatter, this appears to be Chinese APT malware, something related to PlugX
- kjjjjjjjjjjjjjj 6y ago>Chinese APT Wow, surprising!
- mwcampbell 6y ago> Chinese APT malware, Why is it necessary to point out the foreign origin? Doesn't that just encourage our innate xenophobia?
- renewiltord 6y agoMy interpretation, not knowing anything about the field, is that this is a nation state actor or sponsored by such.
- FpUser 6y agoIt might as well be but since you know nothing about the field what is the value of your interpretation? To me your interpretation reads like: something is wrong, must be (substitute with your enemy du jour)
- renewiltord 6y ago
- cmeacham98 6y agoConspiracy theory: the fact the POC insisted on the writer checking out the traffic suggests they knew about (or were suspicious of) the fact that PII was being leaked.
- clankyclanker 6y agoProbably, but is that a conspiracy theory so much as an insurance policy? Being able to competently complete that sort of nightmare investigation is probably why the investigator was re-hired annually. A packet capture of the config files would show something was up to anyone suspicious, but knowing what to do about it is a completely different story.
- cmeacham98 6y agoThe 'conspiracy' part of my conspiracy theory is not that they hired a security consultant, but that they explicitly guided him to the exact hardware[1] with the correct metric to detect it[2] asking him to test for a surprisingly accurate hypothetical[3], even going so far as to temporarily deny the suggestion of the person they're paying to do this work[4]. This is weirdly specific assuming they had no knowledge of the compromise. Of course, I have no non-circumstantial evidence and this could all be a coincidence, which is why my comment is prefixed with "conspiracy theory". 1: "However, he asked me to first look at their cluster of reverse gateways / load balancers" 2: Would have likely been less likely to find the issue with active analysis given the self destruct feature 3: "Specifically he wanted to know if I could develop a methodology for testing if an attacker has gained access to the gateways and is trying to access PII" 4: "I couldn't SSH into the host (no SSH), so I figured we will have to add some kind of instrumentation to the GO app. Klaus still insisted I start by looking at the traffic before (red) and after the GW (green)"
- bombcar 6y agoIt sounded to me like they had a suspicion and specifically wanted the contractor to use his expertise in a limited way that would catch if the suspicion was right. Perhaps they had noticed the programs restarting and when trying to debug triggered it.
- xmodem 6y agoThis is truly the stuff of nightmares, and I'm definitely going to review our CI/CD infrastructure with this in mind. I'm eagerly awaiting learning what the initial attack vector was.
- lovedswain 6y ago9 times out of 10, through the front door. Some shit in a .doc, .html or .pdf. The Google-China hack started with targetted pdfs
- ducktective 6y agoHow such an attack is even possible? A bug in the LibreOffice, browser, or Evince?
- yjftsjthsd-h 6y agoPDF is a nightmare format, including such gems as javascript IIRC; it's not surprising that it can be used to make exploits in reader software.
- ducktective 6y agoSo the attacker has to have exploits in every pdf reader app on linux? Since it is not Adobe only and there are quite a few. Or maybe a common backend engine (mupdf and popler)...
- yjftsjthsd-h 6y agoYeah, I suspect that a rather lot of the options use the same libraries; https://en.wikipedia.org/wiki/Poppler_(software) https://en.wikipedia.org/wiki/Poppler_(software) claims that poppler is used by Evince, LibreOffice 4.x, and Okular (among others).
- npsimons 6y ago
- ivanstojic 6y agoI think the old “Mossad is gonna Mossad” thing is still true. Good security practices are mandatory, and will keep you safe 99% of the time. But when you have what appear to be state level actors using 0 day exploits... you will not stop them.
- kjjjjjjjjjjjjjj 6y agoMore like Chinese state sponsored hackers
- satyanash 6y agoThanks for making me look up "Mossad is gonna Mossad" -> Schneier -> Mickens' essay titled "This World of Ours". https://www.usenix.org/system/files/1401_08-12_mickens.pdf https://www.usenix.org/system/files/1401_08-12_mickens.pdf
- Arrath 6y agoThat was very entertaining, thank you.
- jeffrallen 6y agoThank you for this. Helps put my career choices into perspective. (I just quit security work to be a stay at home dad.)
- GSGBen 6y agoThanks, this is such good writing. Reminds me a little of Douglas Adams.
- sophacles 6y agoBoth are good authors. If you like the humor aspect that's mostly Mickens - one of my favorites from him: https://www.usenix.org/system/files/1311_05-08_mickens.pdf https://www.usenix.org/system/files/1311_05-08_mickens.pdf
- 6y ago
- anticristi 6y agoCall me naïve, but who is such a hot target to warrant so much effort to exfiltrate PII? Defense? FinTech? Government?
- draugadrotten 6y agoI'd add medical to that list. Vaccine test results are hot stuff.
- clankyclanker 6y agoNot just vaccines, but basically all your data, including billing and disease history. Perfect for both scamming and extortion. Keep in mind that you actually want your medical provider to have that data, so they can treat you with respect to your medical history, without killing you in the process.
- anticristi 6y agoTrue. However, reading between the lines, the exfiltration "project" was targeted (i.e. one-off), skilled and long. I would put the cost anywhere between 1 megabuck and 10 megabucks. Given risks and dubious monetization, I would assume the "sponsor" demands at least a 10x ROI. Is medical data really that valuable?
- webnrrd2k 6y agoHow about psychiatric data from the area around Washington DC? Hospitals/practices that are frequented by New York CEO-types? I can picture that being quite valuable to the right parties.
- daveslash 6y agoI think you're right that it's medical. The author calls out PII was the target. Sure, there's PII in Defense/Fintech/Government, but it's probably not the target in those sectors and PII doesn't have the same spotlight on it as in the Medical world (e.g. HIPPA & GDPR).
- sloshnmosh 6y agoWow! What an amazing write-up and find! It’s also amazing that they noticed the subtle difference in the NFS packet capture. I can’t wait for the rest to be published. Bookmarked
- npsimons 6y agoThis is the kind of content I come to HN for! I don't get to do a lot of low level stuff these days, and my forensics skills are almost non-existent, so it's really nice to see the process laid out. Heck, just learning of binwalk and scapy (which I'd heard of, but never looked into) was nice.
- h2odragon 6y agoConsider the possibility that its fiction. Would you be upset? I wouldn't, perhaps a bit disappointed not to learn more. This certainly fits into "worthy of itself".
- ericbarrett 6y agoOne of the most fascinating breach analyses I've ever read. Reading between the lines, I sense the client didn't 100% trust Mr. Bogdanov in the beginning, and certainly knew there was exfiltration of some kind. Perhaps they had done a quick check of the same stats they guided the author toward. "Check for extra bits" seems like a great place to start if you don't know exactly what you're looking for. Their front-end architecture seemed quite locked down and security-conscious: just a kernel + Go binary running as init, plain ol' NFS for config files, firewalls everywhere, bastion hosts for internal networks, etc. So already the client must have suspected the attack was of significant sophistication. Who was better equipped to do this than their brilliant annual security consultant? Which is completely understandable to me, as this hack is already of such unbelievable sophistication that resembles a Neil Stephenson plot. Since the author did not actually commit the crime, and in fact is a brilliant security researcher, everything worked out.
- eeZah7Ux 6y ago> just a kernel + Go binary running as init This is hardly reducing the attack surface compared to a good distro with the usual userspace. It's been decades since attackers relied on a shell, or unix tools in general, or on being to write to disk and so on: it's risky and ineffective. Many attack tools run arbitrary code inside the same process that has been breached and extract data from its memory. They don't try to snoop around or write to disk and so on. Rather, move to another host. The only good mitigation is to split your own application in multiple processes based on the type of risk and sandbox each of them accordingly.
- ericbarrett 6y ago> This is hardly reducing the attack surface compared to a good distro with the usual userspace. Run `tcpdump -n 'tcp and port 80'` on your frontend host and you'll still see PHP exploit attempts from 15 years ago. Not every ghost who knocks is an APT. A singleton Go binary running on a Linux kernel with no local storage is objectively a smaller attack surface than a service running in a container with /bin/sh, running on a vhost with a full OS, running on a physical host with thousands of sleeping VMs—the state of many, many websites and APIs today.
- alfiedotwtf 6y agoStarted slow to get me hooked, then bam... slapped you in the face with a wild ride. Reading this, I know of places that have no hope against someone half as decent as this APT. The internet is a scary place
- lucb1e 6y agoOne thing I didn't get is this magical PII thing. How does the author look at a random network packet -- nay, just packet headers -- and assign a PII:true/false label? I think many corporations would sacrifice the right hand of a sysadmin if that was the way to get this tech. The article just says: > I wrote a small python program to scan the port 80 traffic capture and create a mapping from each four-tuple TLS connection to a boolean - True for connection with PII and False for all others. Is it just matching against a list of source IPs? And perhaps the source port, to determine whether it comes from e.g. a network drive (NFS in this case)? Not sure what he uses the full four-tuple for, if this is the answer in the first place. It's very hand-wavy for what is an integral part of finding the intrusion and kind of a holy grail in other situations as well.
- lettergram 6y agoWe just open sourced one of our libraries used to detect PII: https://github.com/capitalone/DataProfiler https://github.com/capitalone/DataProfiler Amazon and Microsoft also have their own offerings, but can be quite expensive for network packets (and pretty slow). Most projects / teams will use some basic regular expressions to capture basics like SSN, credit card numbers or phone numbers. They’re typically just strings of a specific length. More difficult if you’re doing addresses, names, etc.
- ngngngng 6y ago> YOLO rename label Love the most recent commit
- lucb1e 6y agoThat's great for you but... how's that relevant to the article? The author never speaks of using this sort of thing. I saw these regex matchers in school but don't understand them. They go off all day long because one in a dozen numbers match a valid credit card number, even in the lab environment the default setup was clearly unusable. But perhaps more my point: who'd ever upload the stolen data plaintext anyhow? Unencrypted connections have not been the default for stolen data since... the 80s? If your developers are allowed to do rsync/scp/ftps/sftp/https-post/https-git-smart-protocol then so can I, and if they can't do any of the above then they can't do their work. Adding a mitm proxy is, aside from a SPOF waiting to happen, also very easily circumvented. You'd have to reject anything that looks high in entropy (so much for git clone and sending PDFs) and adding a few null bytes to avoid that trigger is also peanuts. These appliances are snakeoil as far as I've seen. But then I very rarely see our customers use this sort of stuff, and when I do it's usually trivial to circumvent (as I invariably have to to do my work). Now the repository you linked doesn't use regexes, it uses "a cutting edge pre-trained deep learning model, used to efficiently identify sensitive data". Cool. But I don't see any stats from real world traffic, and I also don't see anyone adding custom python code onto their mitm box to match this against gigabits of traffic. Is this a product that is relevant here, or more of a tech demo that works on example files and could theoretically be adapted? Either way, since it's irrelevant to what the author did, I'm not even sure if this is just spam.
- arbirk 6y agoThis observation os way too casual imo: "We noticed a 3 month gap about 5 month ago, and it corresponded with the guy moving the kernel build from a Linux laptop to a new Windows laptop with a VirtualBox VM in it for compiling the kernel. It looks as if it took the attackers three months to gain access back into the box and into the VM build." If the attackers have access to brute force OS engineers / sysadmins work pc's then that should probably be the headline. The rest is just about not being found
- CameronNemo 6y agoMaybe if you are a business oriented person. But reading through the analysis, I felt like the researcher seriously enjoyed the hunt and the "not being found" part.
- deleted 6y ago[deleted]
- Syzygies 6y agoWow. They don't pay you enough.
- abalashov 6y agoAll I learned is (a reminder that) I actually have no idea how computers work. :-)
- djmips 6y ago"I think they let some intern fresh out of college write that one." - I think it was intentional; They probably had a tool to generate that code.