5 ms·
It feels like there are some missing dots and connections here: I see how a concurrency or memory safety bug can accidentally exfil a private key into a debuggi
by tptacek 3y ago
It feels like there are some missing dots and connections here: I see how a concurrency or memory safety bug can accidentally exfil a private key into a debugging artifact, easily, but presumably the attacker here had to know about the crash, and the layout of the crash dump, and also have been ready and waiting in Microsoft's corporate network? Those seem like big questions. "Assume breach" is a good network defense strategy, but you don't literally just accept the notion that you're breached.
- shadowgovt 3y ago> but presumably the attacker here had to know about the crash, and the layout of the crash dump If I were an advanced persistent threat attacker working for China who had compromised Microsoft's internal network via employee credentials (and I'm not), the first thing I'd do is figure out where they keep the crash logs and quietly exfil them, alongside the debugging symbols. Often, these are not stored securely enough relative to their actual value. Having spent some time at a FAANG, every single new hire, with the exception of those who have worked in finance or corporate regulation, assumes you can just glue crash data onto the bugtracker (that's what bugtrackers are for, tracking bugs, which includes reproducing them, right?). You have to detrain them of that and you have to have a vault for things like crashdumps that is so easy to use that people don't get lazy and start circumventing your protections because their job is to fix bugs and you've made their job harder. With a compromised engineer's account, we can assume the attacker at least has access to the bugtracker and probably the ability to acquire or generate debug symbols for a binary. All that's left then is to wait for one engineer to get sloppy and paste a crashdump as an attachment on a bug, then slurp it before someone notices and deletes it (assuming they do; even at my big scary "We really care about user privacy" corp, individual engineers were loathe to make a bug harder to understand by stripping crashlogs off of it unless someone in security came in and whipped them. Proper internal opsec can really slow down development here).
- Eduard 3y ago>... you have to have a vault for things like crashdumps that is so easy to use that people don't get lazy... Let's assume a crash dump can be megabytes up to gigabytes big. How could a vault handle this securely? the moment it is copied from the vault to the developer's computer, you introduce data remanence (undelete from file system). keeping such coredump purely in RAM makes it accessible on a compromised developer machine (GNU Debugger), and if the developer machine crashes, its coredump contains/wraps the sensitive coredump. A vault that doesn't allow direct/full coredump download, but allows queries (think "SQL queries against a vault REST API") could still be queried for e.g. "select * from coredump where string like '%secret_key%'". So without more insight, a coredump vault sounds like security theater which tremendously makes it more difficult for intended purposes.
- bri3d 3y agoWhile your points are all valid theoretically, keeping stuff off of developer filesystems can still help a lot practically. This attacker probably (it's unclear, since the write-up doesn't tell us) scanned compromised machines for key material using some kind of dragnet scanning tool. If the data wasn't on the compromised filesystem, they wouldn't have found it. Even though perhaps in theory they could have sat on a machine with debug access (depending on the nature of the compromise, this is even a stretch - reading another process's RAM usually requires much higher privilege than filesystem access) and obtained a core dump from RAM. Security is always a tension between the theoretical and the practical and I think "putting crash dumps in an easy-to-use place that isn't a developer's Downloads folder" isn't a bad idea.
- nijave 3y agoEphemeral compute/VM/debug environment with restricted access. Tear down the environment after the debugging is done. Keeping the crash dumps in a vault presumably allows more permission/control that an internal issue tracker (usually anyone can access the issue tracker). At least a vault can apply RBAC or even time based policies so these things aren't laying around forever.
- sfink 3y agoEverything is imperfect, but where I work crashdumps are uploaded straight to a secure vault and then deleted from the origin system. The dumps are processed, and insensitive data is extracted and published with relatively lenient access controls. Sensitive data, such as raw memory dumps, require a higher tier of permissions. In order to be eligible for that higher tier, your developer machine has to be more locked down than that of people who are not in the secure group. (You also need to have a reason to need more access.) Given that stack traces, crash addresses, and most register contents are considered to be security insensitive, most people don't really need access to the raw dumps. It's far from perfect, but it would be unfair to call it "security theater". It seems like a pretty decent balance in practice. Admittedly, we have the slight advantage of several hundred millions installs, so the actual bugs that are causing crashes are likely to happen quite a few times and statistical analysis will often provide better clues than diving deep into an individual crash dump.
- saagarjha 3y agoWhy does an attacker need symbols?
- shadowgovt 3y agoThey're not needed, but they will speed up comprehension of the contents of the program. ... but they're definitely not strictly necessary.
- Eduard 3y agoyour statement: > but presumably the attacker here had to know about the crash, and the layout of the crash dump another statement from the article: > Our credential scanning methods did not detect its presence (this issue has been corrected). The article does not give any timeline when things happened. Imagine the following timeline: - hacker gets coredump in 2021, doesn't know that it contains valuable credentials. - For data retention policy reasons, Microsoft deletes their copy of the coredump — but hacker just keeps it. - Microsoft updates its credential scanning methods. - Microsoft runs updated credential software over their reduced archive (retention policy) of coredumps. As that particular coredump doesn't exist anymore at Microsoft, they are not aware of the issue. - hacker get scanner update. - hacker runs updated credential scanner software over their archive of coredumps. Jackpot.
- Muromec 3y agoOh, yeah, the amount of times I see HAR file with things in it just floating in JIRA is not fun.
- formerly_proven 3y agoAnother article from Microsoft in this affair that barely (if indeed) answers more questions than it raises.
- gnfargbl 3y agoThe article says that the employee compromise happened some time after the crash dump had been moved to the corporate network. It says that MS don't have evidence of exfil, but my reading is that they do have some evidence of the compromise. The article also says that Microsoft's credential scanning tools failed to find the key, and that issue has now been corrected. This makes me think that the key was detectable by scanning. Overall, my reading of this is that the engineer moved the dump containing the key into their account at some point, and it just sat there for a time. At a later point, the attacker compromised the account and pulled all available files. They then scanned for keys (with better tooling than MS had; maybe it needed something more sophisticated than looking for BEGIN PRIVATE KEY), and hit the jackpot.
- mistrial9 3y agoit brings a lot of questions to the table about what employee knew what, and when.. A real question is - under a "zero trust" environment, how many motivated insiders have they accumulated with their IT employment and contracting.
- roymj88 3y agoWas having the same thought.. Also anyone know what happened with the employee?
- skywhopper 3y agoThis article says the lack of exfil evidence is "because of log retention policies", ie, they deleted the logs since the exfil happened.
- tgv 3y ago> and hit the jackpot. And how often do you hit the jackpot? For larger lotteries, it's less than once in a million. So that leads to two equally unpleasant alternatives: 1. The attacker was informed where to find the key. 2. The attackers have compromised a large part of Microsoft engineering and routinely scan all their files.
- 3y ago
- stilist 3y agoYeah: 'After April 2021, when the key was leaked to the corporate environment in the crash dump, the Storm-0558 actor was able to successfully compromise a Microsoft engineer’s corporate account. This account had access to the debugging environment containing the crash dump which incorrectly contained the key.' So either the attacker was already in the network and happened to find the dump while doing some kind of scanning that wasn't detected, or they knew to go after this specific person's account.
- nicknow 3y agoOr they knew/discovered that there was a repository of crash dumps - likely a widely known piece of information - and just grabbed as much as they could. Nothing in the write-up indicates any connection between the compromised engineer and this particular crash dump, other than they had access.
- tedunangst 3y agoIf you infiltrate a system and see sshd.core in the filesystem, you're not going to snarf that up?
- rwmj 3y agoI just checked the source and openssh doesn't appear to set madvise(MADV_DONTDUMP) anywhere :-( That seems like an oversight? For comparison openssl has a set of "secure malloc" functions (for keys etc) which uses MADV_DONTDUMP amongst other mitigations.
- garblegarble 3y agoInterestingly, it looks like ssh-agent disables core dumps[1], but I don't see similar usage for sshd 1: https://github.com/openssh/openssh-portable/blob/694150ad92765574ff82a18f4e86322bd3231e68/ssh-agent.c#L2248 https://github.com/openssh/openssh-portable/blob/694150ad927...
- Lt_Riza_Hawkeye 3y agosshd runs as root, so the core dumps would be readable as root-only, no? If you have root access already you could dump it even while it's still running with ptrace anyways
- garblegarble 3y ago>sshd runs as root, so the core dumps would be readable as root-only, no Yes, although the article we're discussing shows that you can't rely on that, the dump could be subsequently moved to a developer machine for investigation, and unencrypted key material left in could be compromised that way... defense in depth would make sense here.
- tedunangst 3y agoOn OpenBSD, you'd be looking for MAP_CONCEAL, though it's not used in many places, either.
- moyix 3y agoI believe there are somewhat standard tools for scanning memory dumps for cryptographic material, which have been around since the cold boot attack era. And I can imagine attackers opportunistically looking for crash dumps with that in mind. But it does seem like an awfully lucky (for the attacker) sequence of events...
- tlb 3y agoSince they can supposedly redact the key from crash dumps, the key must have some identifiable footprint in memory. The redact_key_from_crash_dump function would be a good place to look for juicy bugs. Any case it misses tells you what to look for.