5 ms·

The insinuation is that somehow we had to stop domain fronting because “China”. That’s false. What actually happened was we started to get reports from custome

The insinuation is that somehow we had to stop domain fronting because “China”. That’s false. What actually happened was we started to get reports from customers that we were serving content from unrelated sites under their domains. Wasn’t happening a lot but for a reverse proxy that’s terrifying and about as bad as it can get. And what made it even weirder was that the domain fronting was connecting to other proxy servers outside Cloudflare and so we literally had situation where a Cloudflare customer's website suddenly served up google.com (not a customer). The issue was domain fronting where SNI didn’t match Host causing us to handle the traffic incorrectly sometimes (infrequently but not zero). Since standard use of the Internet doesn’t need SNI and Host to not match we blocked the use of domain fronting very fast to ensure the integrity of our service. Lantern was domain fronting tickling this bug and causing our customers trouble. Hence we dropped domain fronting. I remember this well because I actually debugged it myself and reported the problem in Jira. You weren’t involved and inventing a story about “China” is dead wrong.

FiloSottile 4y ago
As I said above, I was not involved in the deliberation, so I don't know the reason domain fronting was blocked. I clearly remember it being an explicit decision though, not a bug mitigation, and Matthew's explanation on HN makes no mention of a technical issue. I do remember one terrifying bug that Lantern was tickling which caused responses to cross streams, and I was involved in debugging that, but it was not due to the Host/SNI mismatch. It just happened around the same time that domain fronting was blocked. (I am going to respect my confidentiality agreement here, but if you want I can share what I remember here or in private.)
- jgrahamc 4y ago
  I clearly remember it being an explicit decision though, not a bug mitigation To be clear, that's not correct we did do this to mitigate the bug. We were facing the bug that I described (the cross stream thing) showing up when Lantern was used. It was causing disruption to our service and customers were writing in. We were trying to understand what was happening and needed to stop it. One of the things we did to stop it was disable domain fronting. As we were seeing the customer reports we didn't know if this was an OpenSSL bug, something in NGINX, something in our code, but we did know that Lantern was somehow causing it and they were doing domain fronting which wasn't the standard use of our service and so we dropped it.
  - FiloSottile 4y ago
    Maybe domain fronting was initially disabled as an unsuccessful attempt to fix that bug, that's possible and as I said I was not involved in that decision. Still, if that's the case, there was a policy decision afterwards to leave it disabled, because disabling it did not fix the bug, as you seem to agree. (Again, not elaborating on the bug publicly without permission, but I remember it turned out to have nothing to do with the SNI.) My point is that disabling domain fronting (or leaving it disabled after finding the bug's root cause) was a policy decision, not something necessary to mitigate the bug or prevent it from re-occurring.
    - hungryforeggs 4y ago
      - tptacek 4y ago
        Just for the record, there are very few people on the Internet I would trust more about this kind of stuff than Filippo.
        hungryforeggs 4y ago
        tptacek 4y ago
        I'm extraordinarily comfortable with how my comments here reflect on my judgement.
    - eastdakota 4y ago
      - dang 4y ago
        Commenters aren't allowed to attack others like this on HN, regardless of how wrong the other person is or it feels like they are. If you'd please review and follow the site guidelines, we'd appreciate it: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html.
mapgrep 4y ago
…he says, as his company eagerly does business with Xi in China. No organization I am part of will ever do business with you if I can stop it. You fought for Turing but I suppose the Uyghur concentration camps mean nothing to you. They are not British so how could their lives be worth fighting for when there is money to be made (for yourself) without regard to morality or any sense of decency. But good job doing Cloudflare PR.
mindslight 4y ago
Could you not make domain fronting only work when the inside Host contains a specific component like start-frontin.example.com ? Then it will never work for a naive browser wanting to reach www.customer.com, but will work for non-browser clients deliberately trying to front.

FiloSottile 4y ago

As I said above, I was not involved in the deliberation, so I don't know the reason domain fronting was blocked. I clearly remember it being an explicit decision though, not a bug mitigation, and Matthew's explanation on HN makes no mention of a technical issue. I do remember one terrifying bug that Lantern was tickling which caused responses to cross streams, and I was involved in debugging that, but it was not due to the Host/SNI mismatch. It just happened around the same time that domain fronting was blocked. (I am going to respect my confidentiality agreement here, but if you want I can share what I remember here or in private.)

jgrahamc 4y ago
I clearly remember it being an explicit decision though, not a bug mitigation To be clear, that's not correct we did do this to mitigate the bug. We were facing the bug that I described (the cross stream thing) showing up when Lantern was used. It was causing disruption to our service and customers were writing in. We were trying to understand what was happening and needed to stop it. One of the things we did to stop it was disable domain fronting. As we were seeing the customer reports we didn't know if this was an OpenSSL bug, something in NGINX, something in our code, but we did know that Lantern was somehow causing it and they were doing domain fronting which wasn't the standard use of our service and so we dropped it.
- FiloSottile 4y ago
  Maybe domain fronting was initially disabled as an unsuccessful attempt to fix that bug, that's possible and as I said I was not involved in that decision. Still, if that's the case, there was a policy decision afterwards to leave it disabled, because disabling it did not fix the bug, as you seem to agree. (Again, not elaborating on the bug publicly without permission, but I remember it turned out to have nothing to do with the SNI.) My point is that disabling domain fronting (or leaving it disabled after finding the bug's root cause) was a policy decision, not something necessary to mitigate the bug or prevent it from re-occurring.
  - hungryforeggs 4y ago
    - tptacek 4y ago
      Just for the record, there are very few people on the Internet I would trust more about this kind of stuff than Filippo.
      - hungryforeggs 4y ago
        tptacek 4y ago
        I'm extraordinarily comfortable with how my comments here reflect on my judgement.
  - eastdakota 4y ago
    - dang 4y ago
      Commenters aren't allowed to attack others like this on HN, regardless of how wrong the other person is or it feels like they are. If you'd please review and follow the site guidelines, we'd appreciate it: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html.

mapgrep 4y ago

…he says, as his company eagerly does business with Xi in China. No organization I am part of will ever do business with you if I can stop it. You fought for Turing but I suppose the Uyghur concentration camps mean nothing to you. They are not British so how could their lives be worth fighting for when there is money to be made (for yourself) without regard to morality or any sense of decency. But good job doing Cloudflare PR.

mindslight 4y ago

Could you not make domain fronting only work when the inside Host contains a specific component like start-frontin.example.com ? Then it will never work for a naive browser wanting to reach www.customer.com, but will work for non-browser clients deliberately trying to front.