4 ms·
I don't understand that logic, wouldn't people interacting with the website normally also generate the same amount if not more DNS requests?
by curiousgal 5y ago
I don't understand that logic, wouldn't people interacting with the website normally also generate the same amount if not more DNS requests?
- bt1a 5y agoThere's a lot of caching involved in the chain of requests that would alleviate this request volume if things were working.
- eklbt 5y agoMy best guess is that after n many attempts to access the provided IP, the local DNS cache deletes the entry causing a miss. Then the cycle continues.
- pwagland 5y agoNo, since the positive response will normally be cached for "some time" dependant on a number of factors. The negative response on the other hand often won't get cached, again, dependent on settings.
- keithnoizu 5y agoYep, I always make it a point to cache cache-misses in my code.
- ars 5y agoSo then when I'm on some kind of blocked WiFi and nothing resolves, and I switch to a properly working WiFi your code will continue to fail? It's not so simple to cache misses - you don't know if it's a real miss or some kind of error. For example if Facebook cached the miss, then even when they are back up nothing would connect.
- paledot 5y agoIt's remarkable the effect even short TTL caching can have given enough traffic. I recall once caching a value that was being accessed on every page load with a TTL of 1s resulting in a >99% reduction in query volume, and that's nowhere near Facebook/internet backbone scale.
- keithnoizu 5y agoyep, prepriming the cache rather than passively allowing it be rebuilt by request/queries can also result in some nice improvements and depending on replication delay across database servers avoid some unexpected query results reaching the end user. In the past I was the architect of a top 2000 alexa ranked social networking site, data synchronization delays were insane under certain load patterns high single low double digit second write propagation delays.
- keithnoizu 5y agoYes. I handle around a million requests per minute. I exponentially increase the cache period after subsequent misses to avoid an outage ddos the whole system. This tends to be beneficial regardless of the root cause. edit this is especially useful for handling search/query misses as a query with no results is going to scan any relevant indexes etc. until it is clear no match exists meaning a no results query may take up more cycles than a hit.
- keithnoizu 5y agoI'm talking back-end not in app data caching. I would also cache misses there as well but with less aggressive ttl.
- rkeene2 5y agoNegative responses are cachable with the appropriate time to live from the Start of Authority record for the zone.
- jcims 5y agoI know you're just replying to the parent statement but unfortunately in this case the SOA went down with the ship. None of the (admittedly few) clients I've tested are caching the lack of a response for facebook.com's SOA or address records.
- masklinn 5y agoNormally the request resolves then gets cached locally, on the edge, by the ISP, … DNS is cached to a ridiculous levels. But if the request does not resolve there’s no caching, the next request goes through the entire thing and hits the server again.
- smegcicle 5y agosomething about caching
- _joel 5y agoIt will have been cached at closer to the edge, but once the TTL expires, so does the cache. That means all the DNS requests that would have been served via local caches end up hitting the upstream DNS servers. For a site like Facebook that will be creating an asbolute deluge of requests. Andecdotal but the whole of the internet feels sluggish atm.
- yakubin 5y agoAnecdotally, my personal website feels faster than normally. Gandi DNS.
- kentonv 5y agoIt's disappointingly common for cloud-backed apps and device firmware to go into a hot retry loop on any kind of network failure. A lot of engineers just haven't heard of exponential backoff, to say nothing of being able to implement and test it properly for a scenario that almost never happens. Even if you assume Facebook's own apps have reasonable failure logic, there's all kinds of third-party apps and devices integrating with their API that probably get it wrong. Surprise botnet!