7 ms·

Domain Shadowing: Leveraging CDNs for Robust Blocking-Resistant Communications

52 points by Sami_Lehtinen 6y ago

thitcanh 6y ago
Just as a side-note since their example uses CloudFlare: I think that host-rewriting is disabled on CloudFlare as a security or spam-avoidance feature. If you want to have CF change the host to something else, like censored.com, this domain must also be part of your controlled domains, which means CF can’t be used for this purpose. This at least was the case last time I looked into this (12 months ago) It’s also possible that this feature is enabled for paying users, which I wasn’t.
- luckylion 6y ago
  You can do it with a Worker, it'll do completely independent requests to the backend and you can then rewrite them (e.g. change headers, content) and relay them to the client.
  - VWWHFSfQ 6y ago
    I found this to be the case on CloudFront as well. As far as I can tell, it's not possible to direct the request to a different, custom origin based on the Host header. Only the URL path. The recommendation is to use a Lambda@Edge worker to do it. I find this very annoying.
  - bastawhiz 6y ago
    I did just this, to create a version of Github with scrambled text. https://guthib.mattbasta.workers.dev/ https://guthib.mattbasta.workers.dev/
- futhey 6y ago
  Some low-cost CDNs allow this, like BelugaCDN and BunnyCDN, probably others, with a web GUI for setup.
cortesoft 6y ago
I am not authorized to view this page?
- schoen 6y ago
  Maybe your network operator doesn't want you to read about techniques for reading things that network operators don't want you to read.
arkadiyt 6y ago
This also has interesting implications for website operators who use ip allowlists to ensure they "only" receive incoming traffic from their own CDN account. For instance: - if you have corp-domain.com on Cloudflare with http basic auth enabled at the edge, which then proxies to your origin server which has an allowlist of Cloudflare ips, I can make my own account and configure it to point at your origin server, granting me access - if you depend on the cloudflare WAF to block certain attacks to your origin, I can bypass the WAF by configuring a cloudflare worker in my own account to send traffic to your origin - etc
- tgsovlerkhgsel 6y ago
  Cloudflare also supports Authenticated Origin Pull, where the request from Cloudflare to the origin server will be authenticated with a client certificate. IIRC the same client cert was used for all customers, so the same method allowed you to bypass WAF/access rules etc. even if the victim had authenticated origin pull enabled, as long as the victim didn't validate the host header (which they would't if they used the default setup from the official setup guide). Cloudflare knew about this for many months, possibly a year or multiple years, and I'm not sure if it's fixed or not. I'm not sure CF allows host-rewriting in this scenario. Edit: At least the authenticated origin pull setup guide is unchanged. Not going to verify the entire chain again.
- EE84M3i 6y ago
  Allowing only IPs over your vendor of choice has never been secure. You need to additionally ensure the request is hitting your configuration, not someone else's.
RcouF1uZ4gsC 6y ago
> The user registers a random domain as the "shadow" domain, for example: shadow.com. We assume the censor won't block this newly registered domain. Can’t the censor get a list of all registered domains and check if they end up serving the same content as a censored site. This seems like something that a dedicated censored should be able to pretty easily. In addition, don’t most domain registrars require some sort of payment and thus some type of identifying information? For those ones, the censoring authority can use that to track down who registered this shadow domain. The censoring authority can also block domains from registrars that allow anonymous/pseudonymous registration.
- ignoramous 6y ago
  Most censors employ the cheapest possible firewall solution for their blockades. And those, more often than not, involve merely blocking DNS requests. DoH (and DoT to a lesser extent) mitigate censorship to an extent. Censoring HTTP/S connections by inspecting SNI (server name identification) field is prevalent, but it is defeated by clients like GreenTunnel and GetIntra.Org that split the SNI carrying packet in to two. A stateful (read: expensive) firewall can defeat this technique, but not many use it. And anyway, in this game of cat and mouse, Lantern and Psiphon, the two VPNs, have been faring admirably well.
- luckylion 6y ago
  > Can’t the censor get a list of all registered domains and check if they end up serving the same content as a censored site. This seems like something that a dedicated censored should be able to pretty easily. For some TLDs, yes (e.g. .com), but others (e.g. Germany's .de) don't make their zonefiles available.
ignoramous 6y ago
Domain Shadowing is neat but it seems to involve a bit of DNS work from the censored website, too. With edge compute, I would instead prefer to run a http reverse proxy over Cloudflare Workers or Lambda@Edge to the censored servers. In fact, with 100K requests in their free tier, Workers would likely work for every user that can create a Cloudflare account. I can see it work nicely for text-heavy, non-interactive websites just fine. I run a F/OSS DoH resolver, which uses Workers to IP-front itself. Not many censors block Cloudflare's anycast IPs as the collateral damage that occurs from blocking other web properties (behind those shared IPs) isn't worth it for some. As for domain-shadowing / domain-fronting (which Signal once employed pretty effectively), the imminent ECH (encrypted client hello) standard is going to make the very many censors that rely on SNI inspection obsolete for web properties that implement it anyway. I'm also most excited about Tor's support for Snowflake (WebRTC bridges): https://snowflake.torproject.org/ https://snowflake.torproject.org/. Literally, anyone on an uncensored network can run a Tor proxy-node from within their web browser to which other clients from behind censors can connect to. WebRTC isn't usually blocked either due to the ubiquity of web video and audio chatrooms.
- 1vuio0pswjnm7 6y ago
  If the DoH server is hosted on Cloudflare we can use ESNI to send DoH queries. Below is simple example using ESNI-enabled openssl.^1 Cloudflare ESNI works nicely for me. Very fast. YMMV. 1. https://github.com/sftcd/openssl/tree/master/esnistuff https://github.com/sftcd/openssl/tree/master/esnistuff # requirements: openssl-esni-2019-08-30 drill sed grep od tr # get an ESNI key ( printf 'GET /dns-query?name=_esni.www.cloudflare.com&type=TXT&ct=application/dns-json HTTP/1.1\r\n'; printf 'Host: cloudflare-dns.com\r\n'; printf 'Connection: close\r\n'; printf '\r\n'; ) |openssl s_client -4 -noservername -ign_eof -connect 104.16.249.249:443 2>/dev/null \ |grep -o "data\":\"[^}]*"|sed -n s'/.\{9\}//;s/=.*/=/;wESNIRR' # make a DNS request for the name "doh.crypto.sx" over HTTPS using GET method read k < ESNIRR; h=doh.crypto.sx; ( printf 'GET /dns-query?dns=AAABAAABAAAAAAAAA2RvaAZjcnlwdG8Cc3gAAAEAAQ HTTP/1.1\r\n'; printf 'Host: doh.crypto.sx\r\n'; printf 'Connection: close\r\n'; printf 'Accept: application/dns-message\r\n'; printf '\r\n' )|LD_LIBRARY_PATH=./openssl-esni-2019-08-30 ./openssl-esni-2019-08-30/apps/openssl \ s_client -4 -tls1_3 -noservername -ign_eof -connect 104.16.249.249:443 -esni $h -esnirr $k 2>/dev/null > pkt od -An -tx1 -vw3000000 < pkt|tr -d '\40'|sed 's/00008[15]8[03]000[01]00/\ &/g'|drill -i/dev/stdin 2>/dev/null; rm pkt ESNIRR; unset h;
colmmacc 6y ago
O.k. before I comment I have to disclose that I sometimes work on blocking domain fronting abuse (here's a blog post from me - https://aws.amazon.com/blogs/security/enhanced-domain-protections-for-amazon-cloudfront-requests/ https://aws.amazon.com/blogs/security/enhanced-domain-protec...) so I have a bit of an interest here. This is an interesting technique because it starts with the user and not the domain owner. The user creates a single-use domain to mask their own access to some website. Clever! but I'd be really wary about recommending this to someone who is trying to hide access from an oppressive regime. Websites often load all sorts of out of domain resources, and unless you're rewriting all of the HTML (and somehow all dynamically generated HTML) there's still going to be a lot of it. That Facebook example probably went and loaded a bunch of fbcdn.com resources. Seems worth a mention in the privacy section, especially for a post on the Tor project.
- water8 6y ago
  This is why JSP still exists

52 points by Sami_Lehtinen 6y ago

Just as a side-note since their example uses CloudFlare: I think that host-rewriting is disabled on CloudFlare as a security or spam-avoidance feature. If you want to have CF change the host to something else, like censored.com, this domain must also be part of your controlled domains, which means CF can’t be used for this purpose. This at least was the case last time I looked into this (12 months ago) It’s also possible that this feature is enabled for paying users, which I wasn’t.

luckylion 6y ago
You can do it with a Worker, it'll do completely independent requests to the backend and you can then rewrite them (e.g. change headers, content) and relay them to the client.
- VWWHFSfQ 6y ago
  I found this to be the case on CloudFront as well. As far as I can tell, it's not possible to direct the request to a different, custom origin based on the Host header. Only the URL path. The recommendation is to use a Lambda@Edge worker to do it. I find this very annoying.
- bastawhiz 6y ago
  I did just this, to create a version of Github with scrambled text. https://guthib.mattbasta.workers.dev/ https://guthib.mattbasta.workers.dev/
futhey 6y ago
Some low-cost CDNs allow this, like BelugaCDN and BunnyCDN, probably others, with a web GUI for setup.

cortesoft 6y ago

I am not authorized to view this page?

schoen 6y ago
Maybe your network operator doesn't want you to read about techniques for reading things that network operators don't want you to read.

arkadiyt 6y ago

This also has interesting implications for website operators who use ip allowlists to ensure they "only" receive incoming traffic from their own CDN account. For instance: - if you have corp-domain.com on Cloudflare with http basic auth enabled at the edge, which then proxies to your origin server which has an allowlist of Cloudflare ips, I can make my own account and configure it to point at your origin server, granting me access - if you depend on the cloudflare WAF to block certain attacks to your origin, I can bypass the WAF by configuring a cloudflare worker in my own account to send traffic to your origin - etc

tgsovlerkhgsel 6y ago
Cloudflare also supports Authenticated Origin Pull, where the request from Cloudflare to the origin server will be authenticated with a client certificate. IIRC the same client cert was used for all customers, so the same method allowed you to bypass WAF/access rules etc. even if the victim had authenticated origin pull enabled, as long as the victim didn't validate the host header (which they would't if they used the default setup from the official setup guide). Cloudflare knew about this for many months, possibly a year or multiple years, and I'm not sure if it's fixed or not. I'm not sure CF allows host-rewriting in this scenario. Edit: At least the authenticated origin pull setup guide is unchanged. Not going to verify the entire chain again.
EE84M3i 6y ago
Allowing only IPs over your vendor of choice has never been secure. You need to additionally ensure the request is hitting your configuration, not someone else's.

RcouF1uZ4gsC 6y ago

> The user registers a random domain as the "shadow" domain, for example: shadow.com. We assume the censor won't block this newly registered domain. Can’t the censor get a list of all registered domains and check if they end up serving the same content as a censored site. This seems like something that a dedicated censored should be able to pretty easily. In addition, don’t most domain registrars require some sort of payment and thus some type of identifying information? For those ones, the censoring authority can use that to track down who registered this shadow domain. The censoring authority can also block domains from registrars that allow anonymous/pseudonymous registration.

ignoramous 6y ago
Most censors employ the cheapest possible firewall solution for their blockades. And those, more often than not, involve merely blocking DNS requests. DoH (and DoT to a lesser extent) mitigate censorship to an extent. Censoring HTTP/S connections by inspecting SNI (server name identification) field is prevalent, but it is defeated by clients like GreenTunnel and GetIntra.Org that split the SNI carrying packet in to two. A stateful (read: expensive) firewall can defeat this technique, but not many use it. And anyway, in this game of cat and mouse, Lantern and Psiphon, the two VPNs, have been faring admirably well.
luckylion 6y ago
> Can’t the censor get a list of all registered domains and check if they end up serving the same content as a censored site. This seems like something that a dedicated censored should be able to pretty easily. For some TLDs, yes (e.g. .com), but others (e.g. Germany's .de) don't make their zonefiles available.

ignoramous 6y ago

Domain Shadowing is neat but it seems to involve a bit of DNS work from the censored website, too. With edge compute, I would instead prefer to run a http reverse proxy over Cloudflare Workers or Lambda@Edge to the censored servers. In fact, with 100K requests in their free tier, Workers would likely work for every user that can create a Cloudflare account. I can see it work nicely for text-heavy, non-interactive websites just fine. I run a F/OSS DoH resolver, which uses Workers to IP-front itself. Not many censors block Cloudflare's anycast IPs as the collateral damage that occurs from blocking other web properties (behind those shared IPs) isn't worth it for some. As for domain-shadowing / domain-fronting (which Signal once employed pretty effectively), the imminent ECH (encrypted client hello) standard is going to make the very many censors that rely on SNI inspection obsolete for web properties that implement it anyway. I'm also most excited about Tor's support for Snowflake (WebRTC bridges): https://snowflake.torproject.org/ https://snowflake.torproject.org/. Literally, anyone on an uncensored network can run a Tor proxy-node from within their web browser to which other clients from behind censors can connect to. WebRTC isn't usually blocked either due to the ubiquity of web video and audio chatrooms.

1vuio0pswjnm7 6y ago
If the DoH server is hosted on Cloudflare we can use ESNI to send DoH queries. Below is simple example using ESNI-enabled openssl.^1 Cloudflare ESNI works nicely for me. Very fast. YMMV. 1. https://github.com/sftcd/openssl/tree/master/esnistuff https://github.com/sftcd/openssl/tree/master/esnistuff # requirements: openssl-esni-2019-08-30 drill sed grep od tr # get an ESNI key ( printf 'GET /dns-query?name=_esni.www.cloudflare.com&type=TXT&ct=application/dns-json HTTP/1.1\r\n'; printf 'Host: cloudflare-dns.com\r\n'; printf 'Connection: close\r\n'; printf '\r\n'; ) |openssl s_client -4 -noservername -ign_eof -connect 104.16.249.249:443 2>/dev/null \ |grep -o "data\":\"[^}]*"|sed -n s'/.\{9\}//;s/=.*/=/;wESNIRR' # make a DNS request for the name "doh.crypto.sx" over HTTPS using GET method read k < ESNIRR; h=doh.crypto.sx; ( printf 'GET /dns-query?dns=AAABAAABAAAAAAAAA2RvaAZjcnlwdG8Cc3gAAAEAAQ HTTP/1.1\r\n'; printf 'Host: doh.crypto.sx\r\n'; printf 'Connection: close\r\n'; printf 'Accept: application/dns-message\r\n'; printf '\r\n' )|LD_LIBRARY_PATH=./openssl-esni-2019-08-30 ./openssl-esni-2019-08-30/apps/openssl \ s_client -4 -tls1_3 -noservername -ign_eof -connect 104.16.249.249:443 -esni $h -esnirr $k 2>/dev/null > pkt od -An -tx1 -vw3000000 < pkt|tr -d '\40'|sed 's/00008[15]8[03]000[01]00/\ &/g'|drill -i/dev/stdin 2>/dev/null; rm pkt ESNIRR; unset h;

colmmacc 6y ago

O.k. before I comment I have to disclose that I sometimes work on blocking domain fronting abuse (here's a blog post from me - https://aws.amazon.com/blogs/security/enhanced-domain-protections-for-amazon-cloudfront-requests/ https://aws.amazon.com/blogs/security/enhanced-domain-protec...) so I have a bit of an interest here. This is an interesting technique because it starts with the user and not the domain owner. The user creates a single-use domain to mask their own access to some website. Clever! but I'd be really wary about recommending this to someone who is trying to hide access from an oppressive regime. Websites often load all sorts of out of domain resources, and unless you're rewriting all of the HTML (and somehow all dynamically generated HTML) there's still going to be a lot of it. That Facebook example probably went and loaded a bunch of fbcdn.com resources. Seems worth a mention in the privacy section, especially for a post on the Tor project.

water8 6y ago
This is why JSP still exists