5 ms·

Under "things I tried", she lists: > clearing my Cloudflare cache lots of times (this would temporarily fix the problem, but it would just crop up again later)

Under "things I tried", she lists: > clearing my Cloudflare cache lots of times (this would temporarily fix the problem, but it would just crop up again later) > upgrading to a new ‘realm’ on my webhost, in the hopes that there was a bad Apache server or something that I could move away from > Making sure <!DOCTYPE html> was at the beginning of all my HTML in case that helped browsers figure it out it was HTML (it didn’t) > Switching away from nearlyfreespeech’s “free beta bandwidth” program > emailing Cloudflare’s support to see if they knew anything about this > making a lot of curl requests to my webhost directly to see if I could reproduce it (I couldn’t) The last point in this list makes it seem like the issue could probably be with Cloudflare. Considering that she spends an additional $5 per month for the Cloudflare Workers solution now, a similar thing I would've tried would be to turn off Cloudflare DNS/caching on the site for a few weeks (or longer) and observe (this can be done by turning off caching on the Cloudflare console or by changing the DNS servers on NearlyFreeSpeech.net from Cloudflare to what NFSN provides). This would add a minor cost every month, though a lot lesser than $5, IMO. If the problem shows up, then it's certainly something on NFSN. If not, then it's certainly Cloudflare.

kentonv 8y ago
It could be that some particular request header (maybe an unusual user-agent) causes the web server to fail to send a content-type, for some reason. But if the response was nevertheless marked cacheable, then Cloudflare may have cached it and served it to other users who didn't send the unusual header. This problem would seem to go away if you disabled Cloudflare caching, but would still technically be a problem with the web server, not Cloudflare... When thinking about cost, don't forget to factor in developer time. As a rule of thumb, your time as a developer is worth $1 per minute. (Disclosure: I'm the tech lead of Cloudflare Workers.)
- user5994461 8y ago
  You need to set "Vary: User-Agent" when the response varies. It's very important.
  - manigandham 8y ago
    Cloudflare CDN does not use the `Vary` header and only caches by the URL, unless you have an enterprise plan.
    - user5994461 8y ago
      Good to know. That breaks a lot of stuff. Good thing we always had the enterprise plan at work.
      - kentonv 8y ago
        `Vary` is tricky because it potentially forces the cache to do many more disk reads per lookup, hurting performance. Without `Vary`, there's at most one cache entry per cache key, and logic is simple: load that entry, verify it hasn't expired, and then return. With `Vary`, there could be many entries matching any cache key, and you can't even compute a hash bucket upfront because you have to examine the `Vary` header on the cached response to find out which headers matter. You either have to do a linear scan of all entries for that URL, or you need to maintain some sort of fancy index, and even then it's going to be slower than without `Vary`. FWIW, with Cloudflare Workers you can implement logic that is (almost) strictly superior to Vary, and doesn't have these performance challenges. Instead of varying on a header, you can write code to vary on arbitrary properties of the request. For example, `Vary: User-Agent` would normally partition the cache not only for every browser, but for every minor version of every browser and every OS version, ruining your cache hit rate. With a Worker, you can parse the User-Agent and decide the buckets in your own way in code, then compute a custom cache key (or just add a query parameter to the URL) based on that. (Workers run "in front of" cache, so you get to modify the request before cache lookup occurs.) The only down side compared to the header is that your code has to know what to vary on at request time, whereas the `Vary` header can be determined at response time. But if you're writing code specific to some application or web site, usually this is not that hard to determine based on URL alone. (Disclosure: I'm the tech lead of Cloudflare Workers.)
        byroot 8y ago
- Sir_Cmpwn 8y ago
  It's not pure cost. The knowledge and experience gained from chasing the bug all the way down is worth something.
- newscracker 8y ago
  The reason I mentioned cost here is because NFSN, the web host that Julia said she uses for this blog, is a pay-as-you-go service that works out to be a lot cheaper for low volume use compared to oversold hosts that promise unlimited storage/bandwidth (or very large numbers that in reality would trigger abuse prevention mechanisms). Of course, NFSN is also run without any nonsense marketing, and that’s a plus to support. So I’d assumed that cost may have been a reason she chose NFSN (though her situation might’ve changed since the choice and there wasn’t enough time to move to another host).
chubot 8y ago
Yeah I would definitely try either turning off CloudFlare, or switching to some different CDN that provides a CloudFlare-like service. My gut says its CloudFlare, because what I learned from CloudBleed is that they've engineered a really complex system to get good performance from a bad architecture, and complex code is prone to failure. https://en.wikipedia.org/wiki/Cloudbleed https://en.wikipedia.org/wiki/Cloudbleed It might also be on the hosting end, but hosting setups are somewhat "commoditized" now, and probably less tuned for performance, so I would expect fewer exotic, unreproducible bugs. Personally I just use Dreamhost, and my blog has been on HN many times, including the #1 spot, and it's worked fine. You don't need a lot of technology to serve static content these days. Computers are fast.

kentonv 8y ago

It could be that some particular request header (maybe an unusual user-agent) causes the web server to fail to send a content-type, for some reason. But if the response was nevertheless marked cacheable, then Cloudflare may have cached it and served it to other users who didn't send the unusual header. This problem would seem to go away if you disabled Cloudflare caching, but would still technically be a problem with the web server, not Cloudflare... When thinking about cost, don't forget to factor in developer time. As a rule of thumb, your time as a developer is worth $1 per minute. (Disclosure: I'm the tech lead of Cloudflare Workers.)

user5994461 8y ago
You need to set "Vary: User-Agent" when the response varies. It's very important.
- manigandham 8y ago
  Cloudflare CDN does not use the `Vary` header and only caches by the URL, unless you have an enterprise plan.
  - user5994461 8y ago
    Good to know. That breaks a lot of stuff. Good thing we always had the enterprise plan at work.
    - kentonv 8y ago
      `Vary` is tricky because it potentially forces the cache to do many more disk reads per lookup, hurting performance. Without `Vary`, there's at most one cache entry per cache key, and logic is simple: load that entry, verify it hasn't expired, and then return. With `Vary`, there could be many entries matching any cache key, and you can't even compute a hash bucket upfront because you have to examine the `Vary` header on the cached response to find out which headers matter. You either have to do a linear scan of all entries for that URL, or you need to maintain some sort of fancy index, and even then it's going to be slower than without `Vary`. FWIW, with Cloudflare Workers you can implement logic that is (almost) strictly superior to Vary, and doesn't have these performance challenges. Instead of varying on a header, you can write code to vary on arbitrary properties of the request. For example, `Vary: User-Agent` would normally partition the cache not only for every browser, but for every minor version of every browser and every OS version, ruining your cache hit rate. With a Worker, you can parse the User-Agent and decide the buckets in your own way in code, then compute a custom cache key (or just add a query parameter to the URL) based on that. (Workers run "in front of" cache, so you get to modify the request before cache lookup occurs.) The only down side compared to the header is that your code has to know what to vary on at request time, whereas the `Vary` header can be determined at response time. But if you're writing code specific to some application or web site, usually this is not that hard to determine based on URL alone. (Disclosure: I'm the tech lead of Cloudflare Workers.)
      - byroot 8y ago
Sir_Cmpwn 8y ago
It's not pure cost. The knowledge and experience gained from chasing the bug all the way down is worth something.
newscracker 8y ago
The reason I mentioned cost here is because NFSN, the web host that Julia said she uses for this blog, is a pay-as-you-go service that works out to be a lot cheaper for low volume use compared to oversold hosts that promise unlimited storage/bandwidth (or very large numbers that in reality would trigger abuse prevention mechanisms). Of course, NFSN is also run without any nonsense marketing, and that’s a plus to support. So I’d assumed that cost may have been a reason she chose NFSN (though her situation might’ve changed since the choice and there wasn’t enough time to move to another host).

chubot 8y ago

Yeah I would definitely try either turning off CloudFlare, or switching to some different CDN that provides a CloudFlare-like service. My gut says its CloudFlare, because what I learned from CloudBleed is that they've engineered a really complex system to get good performance from a bad architecture, and complex code is prone to failure. https://en.wikipedia.org/wiki/Cloudbleed https://en.wikipedia.org/wiki/Cloudbleed It might also be on the hosting end, but hosting setups are somewhat "commoditized" now, and probably less tuned for performance, so I would expect fewer exotic, unreproducible bugs. Personally I just use Dreamhost, and my blog has been on HN many times, including the #1 spot, and it's worked fine. You don't need a lot of technology to serve static content these days. Computers are fast.