7 ms·

Show HN: File distribution over DNS: (ab)using DNS as a CDN

72 points by tonyg 3y ago

deleted 3y ago
[deleted]
WirelessGigabit 3y ago
> TL;DR. It works, more or less, so long as your resolver properly upgrades to DNS-over-TCP when it gets a truncated UDP response. Coincidence that MUSL just added support to DNS-over-TCP fallback? https://news.ycombinator.com/item?id=36933028 https://news.ycombinator.com/item?id=36933028
- tonyg 3y ago
  Heh, actually yes; the universe trying to tell me something, maybe?
tonyg 3y ago
(Has anyone tried the little demo? ... Does it work?)
jesprenj 3y ago
There's also iodine, a C program that tunnels IPv4 packets over DNS. Useful for bypassing captive portals on wifi, since DNS usually isn't restricted. https://github.com/yarrick/iodine https://github.com/yarrick/iodine Regarding cloudflare DNS over HTTPS: It could be that it tries to server data encoded as JSON, which is impossible in JSON. Some control characters and bytes 128-255 cannot be represented as JSON strings.
- OJFord 3y ago
  IME they block DNS other than their own (not even rewrite, just outright block). Not through experience trying to use iodine, but because I frequently have to drop my explicit DNS server in order to be able to reach the captive portal and connect legitimately.
  - smashed 3y ago
    Of course. A captive portal can also be more sneaky and mangle the packets to redirect them to their own DNS service, since it's not encrypted. But that's the beauty of iodine. It will still work because if the captive portal's name servers actually fully resolve requests, it will contact your upstream iodine controlled name servers and forward the response as-is, because that's just how DNS works. Of course it's also fairly easy to detect/block since your DNS usage will be completely abnormal.
  - jesprenj 3y ago
    Yeah, like a sibling comment replied, the idea of iodine is that you don't use your own resolver but the one that the captive portal provided. You need a domain name, xxx.example, then you can uplink data yyyyyy through the tunnel by querying yyyyyy.xxx.example and iodined, authoritative for xxx.example, will receive this query by the captive portal's resolver. To downlink data, you read responses (txt, cname or null records) that iodined will send for your queries. This probably fills up the cache of the captive portal's resolver (:
    - OJFord 3y ago
      Oh I see. I'm surprised if they upstream DNS requests though, it doesn't serve any legitimate purpose? The whole point is that they respond 172.16.0.1 or whatever for the CP to everything? (Until MAC address known or however they track.)
      - jesprenj 3y ago
        Sometimes they require 3rd party stuff, hotlinked images, js on frontpage, etc. I was on a ferry ship with paid sattelite wifi internet once and they only faked DNS responses for gstatic.com, google, etc. But they allowed my domain (DNS queries to it) and stripe.com (because that's how you paid) -- so they only faked responses for a small set of popular domains. IP access to the internet was, apart from stripe, blocked. And because stripe uses FastlyCDN, they allowed all of fastlycdn network ranges (not useful for iodine), so I could browse reddit (they also use fastly) using stripe's CDN (only html, without i.redd.it image hosting, that's elsewhere). AFAIR Fastly doesn't have a free plan, so I couldn't tunnel at higher speeds (via HTTP) but only via DNS. But yeah, sometimes they just outright respond to any query with their IP and you are out of luck.
derN3rd 3y ago
Wouldn't it be simpler and easier for clients to implement when they would use the TXT to store a magnet link and host the file via webtorrents? Maybe such a solution already exists, but I couldn't find it
- imoverclocked 3y ago
  It's not far from OpenAFS which uses SRV records to point to fileservers. It's not magnet/torrent but it's certainly DNS discovery for data on a different protocol.
- tonyg 3y ago
  Well, I'm not sure adding a webtorrent implementation counts as simpler for clients than just TXT record retrieval.
- whalesalad 3y ago
  Nope. How could you get any easier than plain DNS? That would require a torrent client and bittorrent protocol. The beauty of this hack is that it exists on top of the ubiquitous DNS system.
- dannyobrien 3y ago
  I think the obvious match here is how IPFS can use a dnslink TXT file to point to the content-addressable top of an IPFS-distributed static web https://docs.ipfs.tech/concepts/dnslink/ https://docs.ipfs.tech/concepts/dnslink/ . This is how a lot of dweb/web3 websites maintain and advertise a non-http mirror. It works out of the box for sites served using https://distributed.press/ https://distributed.press/ (non-commercial, open source) or https://fleek.co/ https://fleek.co/ (hosted). If you use Brave, or have the IPFS browser extension, you can access sites like https://ffdweb.org/ https://ffdweb.org/ as an option, or preferentially.
arjvik 3y ago
NSCDN? You missed the chance to name it CDNS!
victorbjorklund 3y ago
DNS is wild. I really need to dig down and better understand it.
- kalupa 3y ago
  I see what you did there ...
- rpastuszak 3y ago
  wait, who is wild?
woleium 3y ago
Don't do this. DNS works and continues to work because we don't abuse it.
- RockRobotRock 3y ago
  Pushing limits is how we get innovative technology. It's interesting, if nothing else.
- cookiengineer 3y ago
  Tell that to hackers doing DNS exfil since 10+ years. Modern ransomware is even able to abuse/imitate apple's airdrop and other DNS-SD services because they're let through by most configurations.
1vuio0pswjnm7 3y ago
"Hmm. Promising. Unfortunately, it doesn't seem to like the TXT records having binary data in them (bug? certainly TXT records are allowed to hold binary data, see here and then here), so it might not Just Work." Aside: IMHO, dnstxt from djbdns is easier for requesting TXT records than dig; it's a much smaller, simpler program. tinydns from djbdns can store any data in TXT records, i.e., arbitratry bytes specified by octal. Perhaps other authoritative servers can also do this today. At the time djbdns was released AFAIK it was the only one. "TXT (``text'') record for fqdn. tinydns-data creates a TXT record for fqdn containing the string s. You may use octal \nnn codes to include arbitrary bytes inside s; for example, \072 is a colon." https://cr.yp.to/djbdns/tinydns-data.html https://cr.yp.to/djbdns/tinydns-data.html Thus one could, e.g., store mini-web pages in TXT records. I experimented with this about 15 years ago.
m3047 3y ago
A lot of our current tech stacks are topheavy. I'm all for re-examining foundational internet technologies and objectively examining how stuff can be pushed down the stack as it were; I'm arguably one of the people destroying DNS if that's your mindset, seems like a good key-value store and "data diode" to me. I think email (SMTP) still hasn't been fully exploited as a job scheduling tool (events... mini batches... SMTP). Bearing in mind, I don't expect people to burden the public infrastructure with these kinds of activities. I think if you're going to use the DNS for telemetry you need to scale your deployment to handle at least hundreds of requests a second; this isn't hard with DNS, it's just not the mindset of DNS operators intending to use the DNS as it always has been used. There are also some blatantly political considerations which need to be swept away to make it truly seamless. The gnashing of teeth is obviously suspect when you consider the trajectory of HTTP: from web pages to XHR, SOAP and DoH. However questionable you might find the architectural choices, HTTP continues to work in spite of the abuse. Likewise, the demise of DNS is unlikely.
- dvasdekis 3y ago
  Have a think then about modern encryption and consensus on top of Usenet, which is the SMTP-compatible message queue of your dreams (in terms of available capacity)
  - m3047 3y ago
    I have thought about it, and some "modern" encryption but maybe not what you're thinking of.
rpastuszak 3y ago
I like that, I use Twitter/X as my CDN: https://sonnet.io/projects#:~:text=Laconic!%20(a%20Twitter%20CDN) https://sonnet.io/projects#:~:text=Laconic!%20(a%20Twitter%2...
- quickthrower2 3y ago
  CDN? or REST DB? Potato potato I guess
  - rpastuszak 3y ago
    Somewhere between S3 and Glacier depending how you look at it. > Potato potato I guess I'd go with potato. I actually own https://potato.horse https://potato.horse and now I'm starting to think that I should point cdn.potato.horse to that twitter thread!
unixfox 3y ago
Somewhat related article: https://blog.benjojo.co.uk/post/dns-filesystem-true-cloud-storage-dnsfs https://blog.benjojo.co.uk/post/dns-filesystem-true-cloud-st... (DNSFS. Store your files in others DNS resolver caches - 2018)
- tonyg 3y ago
  Very interesting and relevant! Thank you!
RockRobotRock 3y ago
This is definitely a harder drive.
- askiiart 3y ago
  Context: https://youtu.be/JcJSW7Rprio https://youtu.be/JcJSW7Rprio
tbiehn 3y ago
Yet another DNS file delivery thing - wrote a few years ago to get files into restrictive networks and results out: https://dualuse.io/blog/dnspump/ https://dualuse.io/blog/dnspump/
- tonyg 3y ago
  Cool, thanks for sharing!

72 points by tonyg 3y ago

deleted 3y ago

[deleted]

> TL;DR. It works, more or less, so long as your resolver properly upgrades to DNS-over-TCP when it gets a truncated UDP response. Coincidence that MUSL just added support to DNS-over-TCP fallback? https://news.ycombinator.com/item?id=36933028 https://news.ycombinator.com/item?id=36933028

tonyg 3y ago
Heh, actually yes; the universe trying to tell me something, maybe?

tonyg 3y ago

(Has anyone tried the little demo? ... Does it work?)

jesprenj 3y ago

There's also iodine, a C program that tunnels IPv4 packets over DNS. Useful for bypassing captive portals on wifi, since DNS usually isn't restricted. https://github.com/yarrick/iodine https://github.com/yarrick/iodine Regarding cloudflare DNS over HTTPS: It could be that it tries to server data encoded as JSON, which is impossible in JSON. Some control characters and bytes 128-255 cannot be represented as JSON strings.

OJFord 3y ago
IME they block DNS other than their own (not even rewrite, just outright block). Not through experience trying to use iodine, but because I frequently have to drop my explicit DNS server in order to be able to reach the captive portal and connect legitimately.
- smashed 3y ago
  Of course. A captive portal can also be more sneaky and mangle the packets to redirect them to their own DNS service, since it's not encrypted. But that's the beauty of iodine. It will still work because if the captive portal's name servers actually fully resolve requests, it will contact your upstream iodine controlled name servers and forward the response as-is, because that's just how DNS works. Of course it's also fairly easy to detect/block since your DNS usage will be completely abnormal.
- jesprenj 3y ago
  Yeah, like a sibling comment replied, the idea of iodine is that you don't use your own resolver but the one that the captive portal provided. You need a domain name, xxx.example, then you can uplink data yyyyyy through the tunnel by querying yyyyyy.xxx.example and iodined, authoritative for xxx.example, will receive this query by the captive portal's resolver. To downlink data, you read responses (txt, cname or null records) that iodined will send for your queries. This probably fills up the cache of the captive portal's resolver (:
  - OJFord 3y ago
    Oh I see. I'm surprised if they upstream DNS requests though, it doesn't serve any legitimate purpose? The whole point is that they respond 172.16.0.1 or whatever for the CP to everything? (Until MAC address known or however they track.)
    - jesprenj 3y ago
      Sometimes they require 3rd party stuff, hotlinked images, js on frontpage, etc. I was on a ferry ship with paid sattelite wifi internet once and they only faked DNS responses for gstatic.com, google, etc. But they allowed my domain (DNS queries to it) and stripe.com (because that's how you paid) -- so they only faked responses for a small set of popular domains. IP access to the internet was, apart from stripe, blocked. And because stripe uses FastlyCDN, they allowed all of fastlycdn network ranges (not useful for iodine), so I could browse reddit (they also use fastly) using stripe's CDN (only html, without i.redd.it image hosting, that's elsewhere). AFAIR Fastly doesn't have a free plan, so I couldn't tunnel at higher speeds (via HTTP) but only via DNS. But yeah, sometimes they just outright respond to any query with their IP and you are out of luck.

derN3rd 3y ago

Wouldn't it be simpler and easier for clients to implement when they would use the TXT to store a magnet link and host the file via webtorrents? Maybe such a solution already exists, but I couldn't find it

imoverclocked 3y ago
It's not far from OpenAFS which uses SRV records to point to fileservers. It's not magnet/torrent but it's certainly DNS discovery for data on a different protocol.
tonyg 3y ago
Well, I'm not sure adding a webtorrent implementation counts as simpler for clients than just TXT record retrieval.
whalesalad 3y ago
Nope. How could you get any easier than plain DNS? That would require a torrent client and bittorrent protocol. The beauty of this hack is that it exists on top of the ubiquitous DNS system.
dannyobrien 3y ago
I think the obvious match here is how IPFS can use a dnslink TXT file to point to the content-addressable top of an IPFS-distributed static web https://docs.ipfs.tech/concepts/dnslink/ https://docs.ipfs.tech/concepts/dnslink/ . This is how a lot of dweb/web3 websites maintain and advertise a non-http mirror. It works out of the box for sites served using https://distributed.press/ https://distributed.press/ (non-commercial, open source) or https://fleek.co/ https://fleek.co/ (hosted). If you use Brave, or have the IPFS browser extension, you can access sites like https://ffdweb.org/ https://ffdweb.org/ as an option, or preferentially.

arjvik 3y ago

NSCDN? You missed the chance to name it CDNS!

victorbjorklund 3y ago

DNS is wild. I really need to dig down and better understand it.

kalupa 3y ago
I see what you did there ...
rpastuszak 3y ago
wait, who is wild?

woleium 3y ago

Don't do this. DNS works and continues to work because we don't abuse it.

RockRobotRock 3y ago
Pushing limits is how we get innovative technology. It's interesting, if nothing else.
cookiengineer 3y ago
Tell that to hackers doing DNS exfil since 10+ years. Modern ransomware is even able to abuse/imitate apple's airdrop and other DNS-SD services because they're let through by most configurations.

1vuio0pswjnm7 3y ago

"Hmm. Promising. Unfortunately, it doesn't seem to like the TXT records having binary data in them (bug? certainly TXT records are allowed to hold binary data, see here and then here), so it might not Just Work." Aside: IMHO, dnstxt from djbdns is easier for requesting TXT records than dig; it's a much smaller, simpler program. tinydns from djbdns can store any data in TXT records, i.e., arbitratry bytes specified by octal. Perhaps other authoritative servers can also do this today. At the time djbdns was released AFAIK it was the only one. "TXT (``text'') record for fqdn. tinydns-data creates a TXT record for fqdn containing the string s. You may use octal \nnn codes to include arbitrary bytes inside s; for example, \072 is a colon." https://cr.yp.to/djbdns/tinydns-data.html https://cr.yp.to/djbdns/tinydns-data.html Thus one could, e.g., store mini-web pages in TXT records. I experimented with this about 15 years ago.

m3047 3y ago

A lot of our current tech stacks are topheavy. I'm all for re-examining foundational internet technologies and objectively examining how stuff can be pushed down the stack as it were; I'm arguably one of the people destroying DNS if that's your mindset, seems like a good key-value store and "data diode" to me. I think email (SMTP) still hasn't been fully exploited as a job scheduling tool (events... mini batches... SMTP). Bearing in mind, I don't expect people to burden the public infrastructure with these kinds of activities. I think if you're going to use the DNS for telemetry you need to scale your deployment to handle at least hundreds of requests a second; this isn't hard with DNS, it's just not the mindset of DNS operators intending to use the DNS as it always has been used. There are also some blatantly political considerations which need to be swept away to make it truly seamless. The gnashing of teeth is obviously suspect when you consider the trajectory of HTTP: from web pages to XHR, SOAP and DoH. However questionable you might find the architectural choices, HTTP continues to work in spite of the abuse. Likewise, the demise of DNS is unlikely.

dvasdekis 3y ago
Have a think then about modern encryption and consensus on top of Usenet, which is the SMTP-compatible message queue of your dreams (in terms of available capacity)
- m3047 3y ago
  I have thought about it, and some "modern" encryption but maybe not what you're thinking of.

rpastuszak 3y ago

I like that, I use Twitter/X as my CDN: https://sonnet.io/projects#:~:text=Laconic!%20(a%20Twitter%20CDN) https://sonnet.io/projects#:~:text=Laconic!%20(a%20Twitter%2...

quickthrower2 3y ago
CDN? or REST DB? Potato potato I guess
- rpastuszak 3y ago
  Somewhere between S3 and Glacier depending how you look at it. > Potato potato I guess I'd go with potato. I actually own https://potato.horse https://potato.horse and now I'm starting to think that I should point cdn.potato.horse to that twitter thread!

unixfox 3y ago

Somewhat related article: https://blog.benjojo.co.uk/post/dns-filesystem-true-cloud-storage-dnsfs https://blog.benjojo.co.uk/post/dns-filesystem-true-cloud-st... (DNSFS. Store your files in others DNS resolver caches - 2018)

tonyg 3y ago
Very interesting and relevant! Thank you!

RockRobotRock 3y ago

This is definitely a harder drive.

askiiart 3y ago
Context: https://youtu.be/JcJSW7Rprio https://youtu.be/JcJSW7Rprio

tbiehn 3y ago

Yet another DNS file delivery thing - wrote a few years ago to get files into restrictive networks and results out: https://dualuse.io/blog/dnspump/ https://dualuse.io/blog/dnspump/

tonyg 3y ago
Cool, thanks for sharing!