4 ms·

It's been many years, and I am still angry and disappointed by Cloudflare's decision to block domain fronting and drop Lantern as a customer. Lantern was one of

It's been many years, and I am still angry and disappointed by Cloudflare's decision to block domain fronting and drop Lantern as a customer. Lantern was one of the most effective Great Firewall bypass proxies at the time, and Cloudflare was expanding in China. (I was at Cloudflare at the time, but I don't have private information on the deliberation. I strongly considered quitting over it, maybe I should have, but I was junior back then.) The CEO even came on HN to try to frame it as an abuse mitigation, accusing Lantern of exploiting Cloudflare and arguing that they were not a customer. That was obviously false because you need to have a Cloudflare zone configured for domain fronting to work. They were a customer as much as the targeted hate websites they strenuously defend. https://news.ycombinator.com/item?id=9234367 https://news.ycombinator.com/item?id=9234367 Companies show their color in selecting who they will stand up for.

jgrahamc 4y ago
The insinuation is that somehow we had to stop domain fronting because “China”. That’s false. What actually happened was we started to get reports from customers that we were serving content from unrelated sites under their domains. Wasn’t happening a lot but for a reverse proxy that’s terrifying and about as bad as it can get. And what made it even weirder was that the domain fronting was connecting to other proxy servers outside Cloudflare and so we literally had situation where a Cloudflare customer's website suddenly served up google.com (not a customer). The issue was domain fronting where SNI didn’t match Host causing us to handle the traffic incorrectly sometimes (infrequently but not zero). Since standard use of the Internet doesn’t need SNI and Host to not match we blocked the use of domain fronting very fast to ensure the integrity of our service. Lantern was domain fronting tickling this bug and causing our customers trouble. Hence we dropped domain fronting. I remember this well because I actually debugged it myself and reported the problem in Jira. You weren’t involved and inventing a story about “China” is dead wrong.
- FiloSottile 4y ago
  As I said above, I was not involved in the deliberation, so I don't know the reason domain fronting was blocked. I clearly remember it being an explicit decision though, not a bug mitigation, and Matthew's explanation on HN makes no mention of a technical issue. I do remember one terrifying bug that Lantern was tickling which caused responses to cross streams, and I was involved in debugging that, but it was not due to the Host/SNI mismatch. It just happened around the same time that domain fronting was blocked. (I am going to respect my confidentiality agreement here, but if you want I can share what I remember here or in private.)
  - jgrahamc 4y ago
    I clearly remember it being an explicit decision though, not a bug mitigation To be clear, that's not correct we did do this to mitigate the bug. We were facing the bug that I described (the cross stream thing) showing up when Lantern was used. It was causing disruption to our service and customers were writing in. We were trying to understand what was happening and needed to stop it. One of the things we did to stop it was disable domain fronting. As we were seeing the customer reports we didn't know if this was an OpenSSL bug, something in NGINX, something in our code, but we did know that Lantern was somehow causing it and they were doing domain fronting which wasn't the standard use of our service and so we dropped it.
    - FiloSottile 4y ago
      Maybe domain fronting was initially disabled as an unsuccessful attempt to fix that bug, that's possible and as I said I was not involved in that decision. Still, if that's the case, there was a policy decision afterwards to leave it disabled, because disabling it did not fix the bug, as you seem to agree. (Again, not elaborating on the bug publicly without permission, but I remember it turned out to have nothing to do with the SNI.) My point is that disabling domain fronting (or leaving it disabled after finding the bug's root cause) was a policy decision, not something necessary to mitigate the bug or prevent it from re-occurring.
      - hungryforeggs 4y ago
        tptacek 4y ago
        Just for the record, there are very few people on the Internet I would trust more about this kind of stuff than Filippo.
        hungryforeggs 4y ago
- mapgrep 4y ago
  …he says, as his company eagerly does business with Xi in China. No organization I am part of will ever do business with you if I can stop it. You fought for Turing but I suppose the Uyghur concentration camps mean nothing to you. They are not British so how could their lives be worth fighting for when there is money to be made (for yourself) without regard to morality or any sense of decency. But good job doing Cloudflare PR.
- mindslight 4y ago
  Could you not make domain fronting only work when the inside Host contains a specific component like start-frontin.example.com ? Then it will never work for a naive browser wanting to reach www.customer.com, but will work for non-browser clients deliberately trying to front.
adamfisk 4y ago
Fascinating, Filippo. We stayed silent on it at the time primarily because we were keeping a low profile particularly as more and more Chinese were using Lantern, but there was also back channel pressure through various contacts, to be honest related to the pending Cloudflare expansion in China. There was also a prelude to all of this that I think made things stickier and bizarrely personal. Prince and I share a mutual friend who introduced us just a few weeks prior. Prince said he supported what we were doing, but asked that I not talk about it publicly, presumably because of the pending China deal. The problem was that literally moments after our friend had introduced us via email, and before he made that request, I had a call with the WSJ where I talked about precisely this. I did everything I could to walk back the article, but Prince didn't buy it and seemed to go ballistic over it. After the WSJ piece, we pulled back from talking more publicly in general. Oh, I forgot! We also partly stayed silent because they didn't actually shut down what we were doing at all =). They matched the SNI to the Host header, sure, but they missed a little detail: we weren't using SNI. Hehe. Lantern worked for another six months or so, and then, through a similarly bizarre sequence of events, we essentially tipped them/you off to what was happening. We remained a customer throughout, and we're a customer to this day. Either way, though, Cloudflare does great work, and everyone has their faults, so I'm generally sympathetic over the whole thing with the one caveat that I am truly unclear how much ultimately did relate to China, most clearly in terms of any public support for these internet freedom techniques. Oh, and I've wanted you to work on Lantern forever btw. Oooh actually if you're not aware of it, the uTLS Go TLS fork is a hugely impactful project that's in widespread use (I would guess maybe 50 million monthly active users rely on it in censored regions via various projects) but needs updating - https://github.com/refraction-networking/utls https://github.com/refraction-networking/utls Oh, and if you think we were effective in China then, you should see what we're doing in Russia and especially Iran now!
adamfisk 4y ago
Oh it's also worth noting that Cloudflare is actually more aggressive in blocking domain fronting than almost anyone else. Lots of folks match the SNI to the Host header, but Cloudflare takes it a step further and also makes sure that TLS connections without SNI have a Host header that's scoped to the IP/server they're actually visiting. That means you can't, for example (not that we would ever, ever do this hehehe), scan the whole Cloudflare IP space for IPs to front through without SNI.

jgrahamc 4y ago

The insinuation is that somehow we had to stop domain fronting because “China”. That’s false. What actually happened was we started to get reports from customers that we were serving content from unrelated sites under their domains. Wasn’t happening a lot but for a reverse proxy that’s terrifying and about as bad as it can get. And what made it even weirder was that the domain fronting was connecting to other proxy servers outside Cloudflare and so we literally had situation where a Cloudflare customer's website suddenly served up google.com (not a customer). The issue was domain fronting where SNI didn’t match Host causing us to handle the traffic incorrectly sometimes (infrequently but not zero). Since standard use of the Internet doesn’t need SNI and Host to not match we blocked the use of domain fronting very fast to ensure the integrity of our service. Lantern was domain fronting tickling this bug and causing our customers trouble. Hence we dropped domain fronting. I remember this well because I actually debugged it myself and reported the problem in Jira. You weren’t involved and inventing a story about “China” is dead wrong.

FiloSottile 4y ago
As I said above, I was not involved in the deliberation, so I don't know the reason domain fronting was blocked. I clearly remember it being an explicit decision though, not a bug mitigation, and Matthew's explanation on HN makes no mention of a technical issue. I do remember one terrifying bug that Lantern was tickling which caused responses to cross streams, and I was involved in debugging that, but it was not due to the Host/SNI mismatch. It just happened around the same time that domain fronting was blocked. (I am going to respect my confidentiality agreement here, but if you want I can share what I remember here or in private.)
- jgrahamc 4y ago
  I clearly remember it being an explicit decision though, not a bug mitigation To be clear, that's not correct we did do this to mitigate the bug. We were facing the bug that I described (the cross stream thing) showing up when Lantern was used. It was causing disruption to our service and customers were writing in. We were trying to understand what was happening and needed to stop it. One of the things we did to stop it was disable domain fronting. As we were seeing the customer reports we didn't know if this was an OpenSSL bug, something in NGINX, something in our code, but we did know that Lantern was somehow causing it and they were doing domain fronting which wasn't the standard use of our service and so we dropped it.
  - FiloSottile 4y ago
    Maybe domain fronting was initially disabled as an unsuccessful attempt to fix that bug, that's possible and as I said I was not involved in that decision. Still, if that's the case, there was a policy decision afterwards to leave it disabled, because disabling it did not fix the bug, as you seem to agree. (Again, not elaborating on the bug publicly without permission, but I remember it turned out to have nothing to do with the SNI.) My point is that disabling domain fronting (or leaving it disabled after finding the bug's root cause) was a policy decision, not something necessary to mitigate the bug or prevent it from re-occurring.
    - hungryforeggs 4y ago
      - tptacek 4y ago
        Just for the record, there are very few people on the Internet I would trust more about this kind of stuff than Filippo.
        hungryforeggs 4y ago
mapgrep 4y ago
…he says, as his company eagerly does business with Xi in China. No organization I am part of will ever do business with you if I can stop it. You fought for Turing but I suppose the Uyghur concentration camps mean nothing to you. They are not British so how could their lives be worth fighting for when there is money to be made (for yourself) without regard to morality or any sense of decency. But good job doing Cloudflare PR.
mindslight 4y ago
Could you not make domain fronting only work when the inside Host contains a specific component like start-frontin.example.com ? Then it will never work for a naive browser wanting to reach www.customer.com, but will work for non-browser clients deliberately trying to front.

adamfisk 4y ago

Fascinating, Filippo. We stayed silent on it at the time primarily because we were keeping a low profile particularly as more and more Chinese were using Lantern, but there was also back channel pressure through various contacts, to be honest related to the pending Cloudflare expansion in China. There was also a prelude to all of this that I think made things stickier and bizarrely personal. Prince and I share a mutual friend who introduced us just a few weeks prior. Prince said he supported what we were doing, but asked that I not talk about it publicly, presumably because of the pending China deal. The problem was that literally moments after our friend had introduced us via email, and before he made that request, I had a call with the WSJ where I talked about precisely this. I did everything I could to walk back the article, but Prince didn't buy it and seemed to go ballistic over it. After the WSJ piece, we pulled back from talking more publicly in general. Oh, I forgot! We also partly stayed silent because they didn't actually shut down what we were doing at all =). They matched the SNI to the Host header, sure, but they missed a little detail: we weren't using SNI. Hehe. Lantern worked for another six months or so, and then, through a similarly bizarre sequence of events, we essentially tipped them/you off to what was happening. We remained a customer throughout, and we're a customer to this day. Either way, though, Cloudflare does great work, and everyone has their faults, so I'm generally sympathetic over the whole thing with the one caveat that I am truly unclear how much ultimately did relate to China, most clearly in terms of any public support for these internet freedom techniques. Oh, and I've wanted you to work on Lantern forever btw. Oooh actually if you're not aware of it, the uTLS Go TLS fork is a hugely impactful project that's in widespread use (I would guess maybe 50 million monthly active users rely on it in censored regions via various projects) but needs updating - https://github.com/refraction-networking/utls https://github.com/refraction-networking/utls Oh, and if you think we were effective in China then, you should see what we're doing in Russia and especially Iran now!

adamfisk 4y ago

Oh it's also worth noting that Cloudflare is actually more aggressive in blocking domain fronting than almost anyone else. Lots of folks match the SNI to the Host header, but Cloudflare takes it a step further and also makes sure that TLS connections without SNI have a Host header that's scoped to the IP/server they're actually visiting. That means you can't, for example (not that we would ever, ever do this hehehe), scan the whole Cloudflare IP space for IPs to front through without SNI.