4 ms·

> Something is deeply wrong with such an adversarial ecosystem. If sites don't want to be found and indexed why go to any effort to include them? On the other h

> Something is deeply wrong with such an adversarial ecosystem. If sites don't want to be found and indexed why go to any effort to include them? On the other hand there are millions of small sites out there keen to be found. I work on a small - medium ecommerce website and my code just... sucks. I kind of don't want to admit it but it is true. When there is some Chinese search engine that tries to crawl all the product detail pages during the day (presumably at night for them?), it slows down the site to a crawl. I mean technically I should have the pages set up so they can't pierce through the cloudflare cache but it is easier to just ask cloudflare to challenge user (captcha?) if there are more than n (I think currently set to something small like ten) requests per second from any single source. I don't understand all the business decisions but yeah, I'd suspect the biggest reason is we simply have poor codebases and can't spend too much time fixing this while we have so many backlog items from marketing to work on...

ALittleLight 4y ago
Why are page loads so slow or demanding? I can't imagine how a web crawler could be DoS'ing you if it's in good faith. What is the TPS? What caching are you doing? What's your stack like?
- Gh0stRAT 4y ago
  Not GP, but from having run a small/niche search engine that got hammered by a crawler in the past: Webserver was a single VM running a Java + Spring webserver in Tomcat, connecting to an overworked Solr cluster to do the actual faceted searching. Caches kept most page loads for organic traffic within respectable bounds, but the crawler destroyed our cache hit rate when it was scraping our site and at one point did exhaust a concurrent connection limit of some kind because there were so many slow/timing-out requests in progress at the same time.
  - ALittleLight 4y ago
    I would expect that a small to medium e-commerce site would cache all their pages.
    - oogali 4y ago
      That approach depends on what's in the pages. Does the served HTML contain everything: the product, related products, comments, reviews, etc? If so, caching the entire page might be counterproductive. But if the page is designed in such a way where it's broken up into fragments, then caching heavily is your friend. Those fragments can either be loaded client-side (Javascript) or server-side (ESI, SSR, etc) -- but page fragments are key to increasing how much and how long you can cache. And with this, you've added a layer of complexity that a small to medium e-commerce site may not be able to fulfill with their in-house talent.
- deleted 4y ago
  [deleted]
deleted 4y ago
[deleted]

ALittleLight 4y ago

Why are page loads so slow or demanding? I can't imagine how a web crawler could be DoS'ing you if it's in good faith. What is the TPS? What caching are you doing? What's your stack like?

Gh0stRAT 4y ago
Not GP, but from having run a small/niche search engine that got hammered by a crawler in the past: Webserver was a single VM running a Java + Spring webserver in Tomcat, connecting to an overworked Solr cluster to do the actual faceted searching. Caches kept most page loads for organic traffic within respectable bounds, but the crawler destroyed our cache hit rate when it was scraping our site and at one point did exhaust a concurrent connection limit of some kind because there were so many slow/timing-out requests in progress at the same time.
- ALittleLight 4y ago
  I would expect that a small to medium e-commerce site would cache all their pages.
  - oogali 4y ago
    That approach depends on what's in the pages. Does the served HTML contain everything: the product, related products, comments, reviews, etc? If so, caching the entire page might be counterproductive. But if the page is designed in such a way where it's broken up into fragments, then caching heavily is your friend. Those fragments can either be loaded client-side (Javascript) or server-side (ESI, SSR, etc) -- but page fragments are key to increasing how much and how long you can cache. And with this, you've added a layer of complexity that a small to medium e-commerce site may not be able to fulfill with their in-house talent.
deleted 4y ago
[deleted]

deleted 4y ago

[deleted]