5 ms·

No. Cloudflare offers different levels of protection. One level is ‘prevent DDoS.’ Another level is ‘prevent bots from accessing the site at all.’ Not all bots

No. Cloudflare offers different levels of protection. One level is ‘prevent DDoS.’ Another level is ‘prevent bots from accessing the site at all.’ Not all bots are part of a DDoS. The problem is that many website owners turn on the second setting, because ‘bots are bad,’ without realizing that this means that some of their users are going to have to fill out Captchas. (Comment written from memory, I may have details wrong.)

marginalia_nu 3y ago
Sometimes it's a lesser evil. Clouflare blocks about 1.6 million bot search queries per day on my search engine. Simply could not operate it without this inconvenience.
- KomoD 3y ago
  1.6 million out of how many total?
  - marginalia_nu 3y ago
    50k legitimate queries / day on a slow day. A HN hug of death is maybe 100-150k/day.
    - yamtaddle 3y ago
      I think those who haven't operated a publicly-visible server on the open Internet in some time might be surprised at just how shark-infested these waters are now. It's, like, mostly sharks.
- justinclift 3y ago
  > I'm currently looking for hosting for a large term frequency data file that is necessary for several of the search engine's core functions. Did you get that sorted out? Asking because we (sqlitebrowser.org, dbhub.io) have a bunch of Hetzner dedicated servers that are nowhere near fully utilised. Could probably figure something decent out using those, as Hetzner doesn't charge for bandwidth.
  - marginalia_nu 3y ago
    Yeah that's solved itself, I eventually got a cheap VPS @ downloads.marginalia.nu for providing these files. It solves the immediate problem of hosting the data that can't go in git. How much space have you got by the way?
    - justinclift 3y ago
      Checking just now, there's at least 1/2 TB spare on all except one machine. For these boxes, once they're set up they tend to not grow all that much in disk space.
    - justinclift 3y ago
      On a different note, is IPv6 supposed to be enabled for "downloads.marginalia.nu"?

marginalia_nu 3y ago

Sometimes it's a lesser evil. Clouflare blocks about 1.6 million bot search queries per day on my search engine. Simply could not operate it without this inconvenience.

KomoD 3y ago
1.6 million out of how many total?
- marginalia_nu 3y ago
  50k legitimate queries / day on a slow day. A HN hug of death is maybe 100-150k/day.
  - yamtaddle 3y ago
    I think those who haven't operated a publicly-visible server on the open Internet in some time might be surprised at just how shark-infested these waters are now. It's, like, mostly sharks.
justinclift 3y ago
> I'm currently looking for hosting for a large term frequency data file that is necessary for several of the search engine's core functions. Did you get that sorted out? Asking because we (sqlitebrowser.org, dbhub.io) have a bunch of Hetzner dedicated servers that are nowhere near fully utilised. Could probably figure something decent out using those, as Hetzner doesn't charge for bandwidth.
- marginalia_nu 3y ago
  Yeah that's solved itself, I eventually got a cheap VPS @ downloads.marginalia.nu for providing these files. It solves the immediate problem of hosting the data that can't go in git. How much space have you got by the way?
  - justinclift 3y ago
    Checking just now, there's at least 1/2 TB spare on all except one machine. For these boxes, once they're set up they tend to not grow all that much in disk space.
  - justinclift 3y ago
    On a different note, is IPv6 supposed to be enabled for "downloads.marginalia.nu"?