6 ms·

Sinkholed: A DNS Horror Story (2019)

127 points by susam 3y ago

cryptonector 3y ago
> Unfortunately, their operation inadvertently flagged my domain name as one of the domain names to be sinkholed because it matched the pattern of command and control (C2) domain names generated by a malware family named Nymaim, one of the malware families hosted on Avalanche. Although, they had validity checks to avoid sinkholing false-positives, my domain name unfortunately slipped through those checks. [...] That's incredible. That "[the domainname] matched the pattern of command and control (C2) domain names generated by a malware family named Nymaim" was enough to get it sinkholed is nuts. There should be a fair bit of manual checks here before applying this sort of death penalty. Take my username. I actually got targeted by my bank some years ago for a similar reason: "crypto" surely means you're doing crypto-currencies, right?, so you must register as a money services business (MSB)! Uh, no, nothing of the sort, and thankfully I was able to disabuse them of the notion that I had anything to do with crypto-currencies. (Indeed, I'm vehemently opposed to proof-of-work currencies, and as for proof-of-stake, why not just do double-spend detection and leave it at that?)
- TheHappyOddish 3y ago
  I've used a domain with "uber" in it for over 20 years, and since the ride sharing service started I have quite a few issues pop up. At least people know how to spell it now.
  - NoZebra120vClip 3y ago
    Yeah, how do you spell it? Über, with an umlaut like the original German word? Or do you spell it Ueber, where the "ue" stands in for an umlaut in character sets which are incapable of reproducing the glyph? Or do you spell it Uber like the American ride-sharing service, ignoring niceties of pronunciation and trading those for convenience by people who only speak English and don't know German phonology?
    - klabb3 3y ago
      As someone who has a name containing a non-ascii letter living in the US, I greatly prefer simply removing the dots or apostrophe or whatever and just use the ascii version. Those transliterations like ü->ue or ø->oe cause much more problems than they solve. Yes, there are lots of letters for where this doesn't work at all, but for the ones that do, the principle of least surprise is better than phonetical correctness, imo. I pick my battles with Anglo- and US-centrism and this one isn’t worth fighting for, imo. “Whatever the letter looks closest to” has a much higher chance of a consistent interpretation. This isn’t even the fault of anglocentrism – it’s an issue between any languages with different alphabets. Heck, I don’t know Cyrillic, but at least with say Vietnamese i can go from ê->e without even turning on my brain. EDIT: oh, and reconstructing a word or sentence with missing umlauts etc is very easy to do, at least in my language. You can infer by context if there are ambiguities in 99% of cases.
    - realitythreek 3y ago
      The word Uber has existed in English speaking countries for much longer than the ride sharing service. Using that spelling doesn’t affiliate you with the company. The opposite actually as the company used a common word.
      - fragmede 3y ago
        Arguably, Uber is different from uber, meaning outstanding or supreme, and the former has only existed since 2011 due to a complaint from San Francisco taxis causing a rename from Ubercab.
- mannykannot 3y ago
  My impression was that, when botnet sinkholing was first practiced, a lot of care was taken over avoiding collateral damage, but if so, then the standards seem to have decayed. This might have been a consequence of frustration - effective action seeming to be just out of reach - but I have also seen the fallacy that goes "we are the good guys, so therefore what we do is for the good." See also "normalization of deviance." This is why I suspect that, if the AI Apocalypse is technically possible, it is inevitable.
  - Spooky23 3y ago
    Like anything else, when it’s new, the A-team people are working on it. When changes slow and products or services move to steady state, care and feeding moves to dollar focused app teams.
cobertos 3y ago
I would really like to know what sort of traffic patterns that caused the detection. How would you accidentally mistake susam.in for a C2 server? Where are they sniffing traffic at to determine this?
- m3047 3y ago
  This and the "how the fuck does this happen?" comment... There are two technical components which are salient to the Avalanche story here which are DNS related. First is <<fast flux>>. A rough sketch of this looks like servers behind reverse proxy servers, and DNS servers. Both the proxy and DNS server components are intended to cycle extremely fast, rendering IP address blocking impractical and tracking difficult. We'd often find that the actual mothership servers used named virtual hosting and by sending different Host: headers we could tickle different phish. Second is <<DGA>> or a Domain Generation Algorithm, the point of which is to enable autonomous discovery of Command and Control servers by implants. In practice you can think of a DGA as TOTP for domain names. So at any given time the algorithm offers a bag of possible C2 server domain names (which constantly changes) and the implants rattles doorknobs until it hears "notary sojack" from one of them. (There can be lots of these, thousands or tens of thousands at any time.) I think DGA is the concern here. You may think there's some traffic sniffing going on, but that's only the case for observed implant activity. Let's look at some implant and its C2 from a traffic standpoint. The implant is going around asking "notary sojack? notary sojack?" to every Tom, Dick and Carol and even to cats and garbage cans. The C2 looks like a garbage can but has an incongruous collection of fans who are also observed muttering "notary sojack" at random objects. It was possible in most cases to reverse engineer the DGA and seeds with confidence. DNS zones would be populated with the flavor of the day (and maybe the days prior and after) and utilized by caching resolver operators as response policy zones (RPZs) or e.g. Bro lookup tables. The notion being if something in your org (using your resolvers) is doing DNS lookups for a bunch of these, you might want to take a look and see if it's rattling doorknobs and muttering "notary sojack". That's the idea. Technically, somebody could block on it but there is the risk of false positives. Your vendor might vett the list in some fashion; or not. I don't recall domains being seized as commonplace in my time. I could make up hypotheticals as to what might cause this to occur but I've got no evidence or experience supporting one over another. It could've been a mistake. In any case, after a few days the DGA moves on and the domain is useless; that comports with the domain being returned to the owner. (I'm not under any noncompetes at this time.)
LoganDark 3y ago
How the fuck does some random-ass company have the authority to just yank domains away instantly like this, without consent or confirmation? That's what I want to know.
TZubiri 3y ago
This is why you don't use meme TLDs.
teddyh 3y ago
So, nothing to do with the actual DNS, but instead with the domain name registrar.
- deleted 3y ago
  [deleted]
- inopinatus 3y ago
  systems include people
- klysm 3y ago
  Nothing to do with DNS? Just something to do with the organization that controls some DNS records
  - teddyh 3y ago
    At least, not a technical problem, and not even with the zone for their domain, but with the zone of the .in domain.
KirillPanov 3y ago
Reminder: you can't really own a domain name. Use public keys instead. Like i2p and tor do.
- gerdesj 3y ago
  What do you mean: "you can't really own a domain name." That seems to be like saying you can't own a house! Unless you run a registrar, you have to engage the services of a registrar to register your domain with the internet infrastructure. In the UK, the Land Registry is where you register your claim for a particular plot of land and property ownership and where your mortgage provider registers an interest against your property until it is paid off. I forget the precise legal terms. The internets does have a reasonably thorough concept of domain ownership and there is a decent set of precedents. eg MacDonalds wrested control of macdonalds.co.uk from a Scottish shyster who has the surname "MacDonald". He did not have a Ltd company in his name but Maccy D does.
  - defaultcompany 3y ago
    The difference is that the postal service will still forward your mail for you if you sell or lose your house. There is no way to make that happen with a domain name. A person who registers your expired domain name not only has no obligation to forward your email, they can also just read it and potentially get access to any services you set up using that email address as authentication.
  - tomatocracy 3y ago
    Just as an aside - the UK land registry isn't a perfect system at all, even where land is actually registered. For example - I own a flat where the land registry scan of the original long lease which legally documents what I own is incomplete. They don't keep original documents any more since they scanned everything and got rid of the originals years ago to save storage costs. Neither I or the freeholder have original copies either. Theoretically they are supposed to pay to correct this if I complain - but part of what's missing is a plan showing the extent of what I own, which opens up a potential dispute with the freeholder as to what land the lease was originally granted over and I don't really want to have a legal fight over it (and I'm not sure the land registry would cover any legal costs of a dispute). It's not a day to day issue but there's some risk there for the future.
e63f67dd-065b 3y ago
This is actually terrifying to me. If my domain goes, so goes my email -- everything is @myname.com, so I won't receive any email. If my email is gone then I just instantly lost access to a big chunk of the internet services I use, and my online identity might as well be dead. Sure I still have my work email but that's no consolation for all the services registered to my domain email. I'm not sure what the right course of action here is to mitigate such a risk. Add backup emails on a different domain registered with a different registrar to services that support such a thing? Stop worrying because it's rare enough that it's a waste of time? Is this what happens to people that get banned on gmail?
- qingcharles 3y ago
  I've recently been through this and it is a total nightmare. I lost my phone number too, so there was no way to use that validation channel. And many systems have no human support now, so you are super screwed. And then things break for no reason at all and panic the shit out of me. My Namecheap TOTP stopped working today for no reason at all. So I had to go through hours of support to verify my identity, but the whole way through I was thinking "What if I can't verify?"
- nucleardog 3y ago
  It's a rabbit hole to go down, for sure. I spent a while one week going through all my online identity, services I use, etc and putting together basically a dependency tree. What kicked this off was an overlapping but probably more unusual concern. Basically, I was worried that I had things _too_ secure. If I lost certain access, I'd be _screwed_. It's great that this account needs 45 factor 6 dimensional holographic password authentication to log in, but what happens if I lose something or get bonked on the head and forget something or... how can I recover access, but also set this up in a way where the backdoor I leave is _not_ one that's easily accessible to others. Anyway, long story short is I have a separate ccTLD domain from my country that is _exclusively_ for use as the root of my identity/recovery. Everything related to it is in a separate account. It charges to a credit card that's not used for anything else. The only thing it does is receives email and dumps them into object storage so I can periodically review. (I don't want them forwarded elsewhere in case the email is something like a password recovery email.) The recovery solution for this is the ccTLD's dispute resolution policy, and finally my (local) courts. As sexy as that Cocos Islands or Indian Ocean Territory or vanity TLD is, I have a lot more options more easily available to me with my local ccTLD administrator and local courts. I'm pretty much relying on the court's ability to accurately verify my identity as the lock on the back door.

127 points by susam 3y ago

> Unfortunately, their operation inadvertently flagged my domain name as one of the domain names to be sinkholed because it matched the pattern of command and control (C2) domain names generated by a malware family named Nymaim, one of the malware families hosted on Avalanche. Although, they had validity checks to avoid sinkholing false-positives, my domain name unfortunately slipped through those checks. [...] That's incredible. That "[the domainname] matched the pattern of command and control (C2) domain names generated by a malware family named Nymaim" was enough to get it sinkholed is nuts. There should be a fair bit of manual checks here before applying this sort of death penalty. Take my username. I actually got targeted by my bank some years ago for a similar reason: "crypto" surely means you're doing crypto-currencies, right?, so you must register as a money services business (MSB)! Uh, no, nothing of the sort, and thankfully I was able to disabuse them of the notion that I had anything to do with crypto-currencies. (Indeed, I'm vehemently opposed to proof-of-work currencies, and as for proof-of-stake, why not just do double-spend detection and leave it at that?)

TheHappyOddish 3y ago
I've used a domain with "uber" in it for over 20 years, and since the ride sharing service started I have quite a few issues pop up. At least people know how to spell it now.
- NoZebra120vClip 3y ago
  Yeah, how do you spell it? Über, with an umlaut like the original German word? Or do you spell it Ueber, where the "ue" stands in for an umlaut in character sets which are incapable of reproducing the glyph? Or do you spell it Uber like the American ride-sharing service, ignoring niceties of pronunciation and trading those for convenience by people who only speak English and don't know German phonology?
  - klabb3 3y ago
    As someone who has a name containing a non-ascii letter living in the US, I greatly prefer simply removing the dots or apostrophe or whatever and just use the ascii version. Those transliterations like ü->ue or ø->oe cause much more problems than they solve. Yes, there are lots of letters for where this doesn't work at all, but for the ones that do, the principle of least surprise is better than phonetical correctness, imo. I pick my battles with Anglo- and US-centrism and this one isn’t worth fighting for, imo. “Whatever the letter looks closest to” has a much higher chance of a consistent interpretation. This isn’t even the fault of anglocentrism – it’s an issue between any languages with different alphabets. Heck, I don’t know Cyrillic, but at least with say Vietnamese i can go from ê->e without even turning on my brain. EDIT: oh, and reconstructing a word or sentence with missing umlauts etc is very easy to do, at least in my language. You can infer by context if there are ambiguities in 99% of cases.
  - realitythreek 3y ago
    The word Uber has existed in English speaking countries for much longer than the ride sharing service. Using that spelling doesn’t affiliate you with the company. The opposite actually as the company used a common word.
    - fragmede 3y ago
      Arguably, Uber is different from uber, meaning outstanding or supreme, and the former has only existed since 2011 due to a complaint from San Francisco taxis causing a rename from Ubercab.
mannykannot 3y ago
My impression was that, when botnet sinkholing was first practiced, a lot of care was taken over avoiding collateral damage, but if so, then the standards seem to have decayed. This might have been a consequence of frustration - effective action seeming to be just out of reach - but I have also seen the fallacy that goes "we are the good guys, so therefore what we do is for the good." See also "normalization of deviance." This is why I suspect that, if the AI Apocalypse is technically possible, it is inevitable.
- Spooky23 3y ago
  Like anything else, when it’s new, the A-team people are working on it. When changes slow and products or services move to steady state, care and feeding moves to dollar focused app teams.

cobertos 3y ago

I would really like to know what sort of traffic patterns that caused the detection. How would you accidentally mistake susam.in for a C2 server? Where are they sniffing traffic at to determine this?

m3047 3y ago
This and the "how the fuck does this happen?" comment... There are two technical components which are salient to the Avalanche story here which are DNS related. First is <<fast flux>>. A rough sketch of this looks like servers behind reverse proxy servers, and DNS servers. Both the proxy and DNS server components are intended to cycle extremely fast, rendering IP address blocking impractical and tracking difficult. We'd often find that the actual mothership servers used named virtual hosting and by sending different Host: headers we could tickle different phish. Second is <<DGA>> or a Domain Generation Algorithm, the point of which is to enable autonomous discovery of Command and Control servers by implants. In practice you can think of a DGA as TOTP for domain names. So at any given time the algorithm offers a bag of possible C2 server domain names (which constantly changes) and the implants rattles doorknobs until it hears "notary sojack" from one of them. (There can be lots of these, thousands or tens of thousands at any time.) I think DGA is the concern here. You may think there's some traffic sniffing going on, but that's only the case for observed implant activity. Let's look at some implant and its C2 from a traffic standpoint. The implant is going around asking "notary sojack? notary sojack?" to every Tom, Dick and Carol and even to cats and garbage cans. The C2 looks like a garbage can but has an incongruous collection of fans who are also observed muttering "notary sojack" at random objects. It was possible in most cases to reverse engineer the DGA and seeds with confidence. DNS zones would be populated with the flavor of the day (and maybe the days prior and after) and utilized by caching resolver operators as response policy zones (RPZs) or e.g. Bro lookup tables. The notion being if something in your org (using your resolvers) is doing DNS lookups for a bunch of these, you might want to take a look and see if it's rattling doorknobs and muttering "notary sojack". That's the idea. Technically, somebody could block on it but there is the risk of false positives. Your vendor might vett the list in some fashion; or not. I don't recall domains being seized as commonplace in my time. I could make up hypotheticals as to what might cause this to occur but I've got no evidence or experience supporting one over another. It could've been a mistake. In any case, after a few days the DGA moves on and the domain is useless; that comports with the domain being returned to the owner. (I'm not under any noncompetes at this time.)

LoganDark 3y ago

How the fuck does some random-ass company have the authority to just yank domains away instantly like this, without consent or confirmation? That's what I want to know.

TZubiri 3y ago

This is why you don't use meme TLDs.

teddyh 3y ago

So, nothing to do with the actual DNS, but instead with the domain name registrar.

deleted 3y ago
[deleted]
inopinatus 3y ago
systems include people
klysm 3y ago
Nothing to do with DNS? Just something to do with the organization that controls some DNS records
- teddyh 3y ago
  At least, not a technical problem, and not even with the zone for their domain, but with the zone of the .in domain.

KirillPanov 3y ago

Reminder: you can't really own a domain name. Use public keys instead. Like i2p and tor do.

gerdesj 3y ago
What do you mean: "you can't really own a domain name." That seems to be like saying you can't own a house! Unless you run a registrar, you have to engage the services of a registrar to register your domain with the internet infrastructure. In the UK, the Land Registry is where you register your claim for a particular plot of land and property ownership and where your mortgage provider registers an interest against your property until it is paid off. I forget the precise legal terms. The internets does have a reasonably thorough concept of domain ownership and there is a decent set of precedents. eg MacDonalds wrested control of macdonalds.co.uk from a Scottish shyster who has the surname "MacDonald". He did not have a Ltd company in his name but Maccy D does.
- defaultcompany 3y ago
  The difference is that the postal service will still forward your mail for you if you sell or lose your house. There is no way to make that happen with a domain name. A person who registers your expired domain name not only has no obligation to forward your email, they can also just read it and potentially get access to any services you set up using that email address as authentication.
- tomatocracy 3y ago
  Just as an aside - the UK land registry isn't a perfect system at all, even where land is actually registered. For example - I own a flat where the land registry scan of the original long lease which legally documents what I own is incomplete. They don't keep original documents any more since they scanned everything and got rid of the originals years ago to save storage costs. Neither I or the freeholder have original copies either. Theoretically they are supposed to pay to correct this if I complain - but part of what's missing is a plan showing the extent of what I own, which opens up a potential dispute with the freeholder as to what land the lease was originally granted over and I don't really want to have a legal fight over it (and I'm not sure the land registry would cover any legal costs of a dispute). It's not a day to day issue but there's some risk there for the future.

e63f67dd-065b 3y ago

This is actually terrifying to me. If my domain goes, so goes my email -- everything is @myname.com, so I won't receive any email. If my email is gone then I just instantly lost access to a big chunk of the internet services I use, and my online identity might as well be dead. Sure I still have my work email but that's no consolation for all the services registered to my domain email. I'm not sure what the right course of action here is to mitigate such a risk. Add backup emails on a different domain registered with a different registrar to services that support such a thing? Stop worrying because it's rare enough that it's a waste of time? Is this what happens to people that get banned on gmail?

qingcharles 3y ago
I've recently been through this and it is a total nightmare. I lost my phone number too, so there was no way to use that validation channel. And many systems have no human support now, so you are super screwed. And then things break for no reason at all and panic the shit out of me. My Namecheap TOTP stopped working today for no reason at all. So I had to go through hours of support to verify my identity, but the whole way through I was thinking "What if I can't verify?"
nucleardog 3y ago
It's a rabbit hole to go down, for sure. I spent a while one week going through all my online identity, services I use, etc and putting together basically a dependency tree. What kicked this off was an overlapping but probably more unusual concern. Basically, I was worried that I had things _too_ secure. If I lost certain access, I'd be _screwed_. It's great that this account needs 45 factor 6 dimensional holographic password authentication to log in, but what happens if I lose something or get bonked on the head and forget something or... how can I recover access, but also set this up in a way where the backdoor I leave is _not_ one that's easily accessible to others. Anyway, long story short is I have a separate ccTLD domain from my country that is _exclusively_ for use as the root of my identity/recovery. Everything related to it is in a separate account. It charges to a credit card that's not used for anything else. The only thing it does is receives email and dumps them into object storage so I can periodically review. (I don't want them forwarded elsewhere in case the email is something like a password recovery email.) The recovery solution for this is the ccTLD's dispute resolution policy, and finally my (local) courts. As sexy as that Cocos Islands or Indian Ocean Territory or vanity TLD is, I have a lot more options more easily available to me with my local ccTLD administrator and local courts. I'm pretty much relying on the court's ability to accurately verify my identity as the lock on the back door.