4 ms·

I thought this was a great writeup, but one significant issue I see unanswered is why did all OCSP responses fall out of cache at the same time? In a cache with

I thought this was a great writeup, but one significant issue I see unanswered is why did all OCSP responses fall out of cache at the same time? In a cache with an even distribution of expiration times they should have seen a gradual increase in traffic as responses steadily fell out of cache. Adding jitter when setting cache duration for the CDNshould help even out the rate at which responses fall out of cache. In addition, monitoring cache rate and measuring request rate at the CDN should have been big indicators that it wasn't a DDoS. Lastly, is this kind of upstream throttling with no customer communication common? That seems like a big failing on the ISPs side.

zeta0134 9y ago
I work at a hosting company, and I can attest to throttling at the ISP side being somewhat common. My own company will throttle or null route customers almost immediately if enough traffic comes through that it starts to affect other customers behind the same switches. We try to notify customers as quickly as possible (emails go out within minutes) but there are a lot of cases where the emails end up in unmonitored inboxes and customers don't realize they're down until their clients complain to them about it. In any case, it sounds like there may be a bit of a logic bug clientside if a service downtime causes all of their clients to generate so much traffic that it looks like DDoS. The clients should be throttling themselves to prevent overloading a downed upstream service, why didn't that happen here? That's worth investigating, though much more difficult to fix at this point with that many copies of the client in the wild. EDIT: I just thought to actually look up OCSP, I assumed it was the mechanism something like certbot was using to renew certificated. This is built into browsers? Yeah, nevermind on this whole paragraph then.
- mholt 9y ago
  > This is built into browsers? Yup, unfortunately, browsers often have to do revocation (and SCT) checks because traditional servers like Apache and nginx don't staple OCSP responses by default (and even if they are configured to do so, the implementations in these servers are not robust against outages). Firefox checks OCSP for DV certs, but that will be disabled in a near-future release.
  - wbl 9y ago
    Even better: OCSP failures cannot result in failing closed because of middleboxen. So all this traffic wasn't needed anyway.
  - tialaramex 9y ago
    Your reference to SCT here seems weird. [Anybody following at home: SCTs are signed timestamps proving a particular Certificate Transparency log server logged this certificate's details at a moment in time. These log servers are public proof of what certificates should exist. The cryptography behind the log servers makes it impossible for them to lie about what they know beyond a certain horizon, policy today sets that horizon at 24 hours] Today the only thing any browser (Chrome) does with SCT by default is to verify whether valid (properly signed) SCTs are provided for certain certificates. This doesn't result in any additional connections, although if OCSP is needed already the SCTs might optionally be included with OCSP. Eventually browsers will be able to automatically report information to detect any discrepancies (e.g. where a log is telling different things to different people), but today that's something which exists only in prototype, not as a default feature in production web browsers ordinary people use every day.

zeta0134 9y ago

I work at a hosting company, and I can attest to throttling at the ISP side being somewhat common. My own company will throttle or null route customers almost immediately if enough traffic comes through that it starts to affect other customers behind the same switches. We try to notify customers as quickly as possible (emails go out within minutes) but there are a lot of cases where the emails end up in unmonitored inboxes and customers don't realize they're down until their clients complain to them about it. In any case, it sounds like there may be a bit of a logic bug clientside if a service downtime causes all of their clients to generate so much traffic that it looks like DDoS. The clients should be throttling themselves to prevent overloading a downed upstream service, why didn't that happen here? That's worth investigating, though much more difficult to fix at this point with that many copies of the client in the wild. EDIT: I just thought to actually look up OCSP, I assumed it was the mechanism something like certbot was using to renew certificated. This is built into browsers? Yeah, nevermind on this whole paragraph then.

mholt 9y ago
> This is built into browsers? Yup, unfortunately, browsers often have to do revocation (and SCT) checks because traditional servers like Apache and nginx don't staple OCSP responses by default (and even if they are configured to do so, the implementations in these servers are not robust against outages). Firefox checks OCSP for DV certs, but that will be disabled in a near-future release.
- wbl 9y ago
  Even better: OCSP failures cannot result in failing closed because of middleboxen. So all this traffic wasn't needed anyway.
- tialaramex 9y ago
  Your reference to SCT here seems weird. [Anybody following at home: SCTs are signed timestamps proving a particular Certificate Transparency log server logged this certificate's details at a moment in time. These log servers are public proof of what certificates should exist. The cryptography behind the log servers makes it impossible for them to lie about what they know beyond a certain horizon, policy today sets that horizon at 24 hours] Today the only thing any browser (Chrome) does with SCT by default is to verify whether valid (properly signed) SCTs are provided for certain certificates. This doesn't result in any additional connections, although if OCSP is needed already the SCTs might optionally be included with OCSP. Eventually browsers will be able to automatically report information to detect any discrepancies (e.g. where a log is telling different things to different people), but today that's something which exists only in prototype, not as a default feature in production web browsers ordinary people use every day.