10 ms·

> but that is detectable so you end up renting IP's from mobile number farms or trying to build your own 4G network. Something is deeply wrong with such an a

> but that is detectable so you end up renting IP's from mobile number farms or trying to build your own 4G network. Something is deeply wrong with such an adversarial ecosystem. If sites don't want to be found and indexed why go to any effort to include them? On the other hand there are millions of small sites out there keen to be found. The established idea of a "search engine" seems stuck, limited and based on some 90's technology that worked on a 90's web that no longer exists. Surely after 30 years we can build some kind of content discovery layer on top of what's out there?

Xeoncross 4y ago
https://blogsurf.io/ https://blogsurf.io/ is an example of a small search engine that just stuck to a directory of known blogs instead of indexing the big sites or randomly crawling the web and ending up with mostly gibberish pages from all the spam sites.
- mannyistyping 4y ago
  thank you for sharing this! I read through the site's about and I really enjoy how the creator wanted to stick to a specific area for quality over quantity.
jeffbee 4y ago
I think that is not what they mean. I think what they meant is the site will detect your headless robot and serve it good content, while serving spam and malware to everyone else. The crawlers need their own distributed networks of unrelated addresses to prevent or detect this behavior.
thanksgiving 4y ago
> Something is deeply wrong with such an adversarial ecosystem. If sites don't want to be found and indexed why go to any effort to include them? On the other hand there are millions of small sites out there keen to be found. I work on a small - medium ecommerce website and my code just... sucks. I kind of don't want to admit it but it is true. When there is some Chinese search engine that tries to crawl all the product detail pages during the day (presumably at night for them?), it slows down the site to a crawl. I mean technically I should have the pages set up so they can't pierce through the cloudflare cache but it is easier to just ask cloudflare to challenge user (captcha?) if there are more than n (I think currently set to something small like ten) requests per second from any single source. I don't understand all the business decisions but yeah, I'd suspect the biggest reason is we simply have poor codebases and can't spend too much time fixing this while we have so many backlog items from marketing to work on...
- ALittleLight 4y ago
  Why are page loads so slow or demanding? I can't imagine how a web crawler could be DoS'ing you if it's in good faith. What is the TPS? What caching are you doing? What's your stack like?
  - Gh0stRAT 4y ago
    Not GP, but from having run a small/niche search engine that got hammered by a crawler in the past: Webserver was a single VM running a Java + Spring webserver in Tomcat, connecting to an overworked Solr cluster to do the actual faceted searching. Caches kept most page loads for organic traffic within respectable bounds, but the crawler destroyed our cache hit rate when it was scraping our site and at one point did exhaust a concurrent connection limit of some kind because there were so many slow/timing-out requests in progress at the same time.
    - ALittleLight 4y ago
      I would expect that a small to medium e-commerce site would cache all their pages.
      - oogali 4y ago
        That approach depends on what's in the pages. Does the served HTML contain everything: the product, related products, comments, reviews, etc? If so, caching the entire page might be counterproductive. But if the page is designed in such a way where it's broken up into fragments, then caching heavily is your friend. Those fragments can either be loaded client-side (Javascript) or server-side (ESI, SSR, etc) -- but page fragments are key to increasing how much and how long you can cache. And with this, you've added a layer of complexity that a small to medium e-commerce site may not be able to fulfill with their in-house talent.
  - deleted 4y ago
    [deleted]
- deleted 4y ago
  [deleted]
jonhohle 4y ago
Maybe we need a categorized, hand curated directory of sites that users can submit their own sites to for inclusion and categorization. Maybe like an open directory. Perhaps Mozilla could operate it, or maybe Yahoo!
- noduerme 4y ago
  I know, right? Imagine if you went to the front page of Yahoo! and it was like a curated directory of websites. Like... a portal. It could look something like this: https://web.archive.org/web/20000302042007/http://www1.yahoo.com/ https://web.archive.org/web/20000302042007/http://www1.yahoo...
  - mysterydip 4y ago
    I would love to see a combo of a search engine and a portal. Basically at any step drilling down into links for categories and subcategories, I could type in a search bar and every site at that level and below would be searched.
    - marginalia_nu 4y ago
      I've been sort of considering going this path with my own search engine, but I feel whenever there is any sort of interactivity, there is the serious problem of spam and abuse; I'm not quite sure how to solve that. Feels like accepting user content no longer flies on the web in 2022 due to the effectiveness of bot spam and lack of viable countermeasures :-/
      - mysterydip 4y ago
        I think non-crowdsourced manual curation (by you/your trusted volunteers) would be a feature. Sure, search would have more limited results, but really with how much cruft there is I don't see that as a bad thing. Start with the "known good/useful" domains and branch out from there.
        4y ago
altdataseller 4y ago
>> If sites don't want to be found and indexed why go to any effort to include them? On the other hand there are millions of small sites out there keen to be found. Then they should treat all bots equally and block Google as well. If they block Google as well, then yes, we should leave them alone. Why give unfair treatment to Google? That's anti-competitive behavior and it just prevents new search engines from being created.
- nonrandomstring 4y ago
  I think I understand, combined with jeffbee's answer, that these sites are behaving selectively according to who you are. So we're back to "No Blacks or Irish" on the 2022 Internet? What do you think they have against smaller search engines? I can't quite fathom the motives.
  - wolfgang42 4y ago
    There are a lot of crawlers out there, and many of them are ill-behaved. When GoogleBot crawls your site, you get more visitors. When FizzBuzzBot/0.1.3 comes along, you’re more likely to get an overloaded server, weird URLs in your crash logs, spam, or any other manner of mess. Small search engines getting blocked is just collateral damage from websites trying to solve this problem with a blunt ban-hammer.
    - bryanrasmussen 4y ago
      This is probably because, as I have learned here on HN, 1 in 4 programmers can't write a decent FizzBuzzBot and should never be hired.
  - autoexec 4y ago
    > So we're back to "No Blacks or Irish" on the 2022 Internet? I think this is the inevitable result of surveillance capitalism. Once you can reliably be identified as holding certain views or meeting some arbitrary criteria pages can be dynamically altered to restrict what you have access to. A racist social platform can look completely benign to anyone who isn't part of the in-group, forums for teens can automatically lock out anyone suspected of being over a certain age, companies can show products or services only to people within a certain class, etc. We already restrict a lot of access to content based on guesses about location. Youtube doesn't even let you see that certain content on their platform exists on their site unless you're logged in from an IP in specific counties (as opposed to showing you the videos exist and throwing up an error saying it's not available for you when you try to view it) and netflix blocks VPNs just to try to keep a bunch of their content away from the wrong kinds of people. These are the kinds of barriers the internet should have freed us from, but instead it's being used to put up more gates and to force people into them.
amelius 4y ago
> Something is deeply wrong with such an adversarial ecosystem. If sites don't want to be found and indexed why go to any effort to include them? I think it is not about being found. It is more about being copied. These sites are afraid their content is stolen, so they only allow Google to crawl them.
noncoml 4y ago
They don’t want to be indexed unless you are Google
throwaway81523 4y ago
> If sites don't want to be found and indexed why go to any effort to include them? The news sites etc. want to be indexed by Google, and they whitelist Google so they show up in search results. They don't want to be crawled by random aggregators and scrapers which don't bring them ad views and subscriptions. Indexing for me and not for thee. Other places like Slack channels are private or semi-private and shouldn't be indexed, I agree. There was some kind of scheme a while back (before Freenode went down the tubes in a completely different and unrelated way) to index Freenode. People rightfully freaked out about it and the scheme was shot down.
anotherrandom 4y ago
Sounds like federation might be making a comeback eventually

Xeoncross 4y ago

https://blogsurf.io/ https://blogsurf.io/ is an example of a small search engine that just stuck to a directory of known blogs instead of indexing the big sites or randomly crawling the web and ending up with mostly gibberish pages from all the spam sites.

mannyistyping 4y ago
thank you for sharing this! I read through the site's about and I really enjoy how the creator wanted to stick to a specific area for quality over quantity.

jeffbee 4y ago

I think that is not what they mean. I think what they meant is the site will detect your headless robot and serve it good content, while serving spam and malware to everyone else. The crawlers need their own distributed networks of unrelated addresses to prevent or detect this behavior.

thanksgiving 4y ago

> Something is deeply wrong with such an adversarial ecosystem. If sites don't want to be found and indexed why go to any effort to include them? On the other hand there are millions of small sites out there keen to be found. I work on a small - medium ecommerce website and my code just... sucks. I kind of don't want to admit it but it is true. When there is some Chinese search engine that tries to crawl all the product detail pages during the day (presumably at night for them?), it slows down the site to a crawl. I mean technically I should have the pages set up so they can't pierce through the cloudflare cache but it is easier to just ask cloudflare to challenge user (captcha?) if there are more than n (I think currently set to something small like ten) requests per second from any single source. I don't understand all the business decisions but yeah, I'd suspect the biggest reason is we simply have poor codebases and can't spend too much time fixing this while we have so many backlog items from marketing to work on...

ALittleLight 4y ago
Why are page loads so slow or demanding? I can't imagine how a web crawler could be DoS'ing you if it's in good faith. What is the TPS? What caching are you doing? What's your stack like?
- Gh0stRAT 4y ago
  Not GP, but from having run a small/niche search engine that got hammered by a crawler in the past: Webserver was a single VM running a Java + Spring webserver in Tomcat, connecting to an overworked Solr cluster to do the actual faceted searching. Caches kept most page loads for organic traffic within respectable bounds, but the crawler destroyed our cache hit rate when it was scraping our site and at one point did exhaust a concurrent connection limit of some kind because there were so many slow/timing-out requests in progress at the same time.
  - ALittleLight 4y ago
    I would expect that a small to medium e-commerce site would cache all their pages.
    - oogali 4y ago
      That approach depends on what's in the pages. Does the served HTML contain everything: the product, related products, comments, reviews, etc? If so, caching the entire page might be counterproductive. But if the page is designed in such a way where it's broken up into fragments, then caching heavily is your friend. Those fragments can either be loaded client-side (Javascript) or server-side (ESI, SSR, etc) -- but page fragments are key to increasing how much and how long you can cache. And with this, you've added a layer of complexity that a small to medium e-commerce site may not be able to fulfill with their in-house talent.
- deleted 4y ago
  [deleted]
deleted 4y ago
[deleted]

jonhohle 4y ago

Maybe we need a categorized, hand curated directory of sites that users can submit their own sites to for inclusion and categorization. Maybe like an open directory. Perhaps Mozilla could operate it, or maybe Yahoo!

noduerme 4y ago
I know, right? Imagine if you went to the front page of Yahoo! and it was like a curated directory of websites. Like... a portal. It could look something like this: https://web.archive.org/web/20000302042007/http://www1.yahoo.com/ https://web.archive.org/web/20000302042007/http://www1.yahoo...
- mysterydip 4y ago
  I would love to see a combo of a search engine and a portal. Basically at any step drilling down into links for categories and subcategories, I could type in a search bar and every site at that level and below would be searched.
  - marginalia_nu 4y ago
    I've been sort of considering going this path with my own search engine, but I feel whenever there is any sort of interactivity, there is the serious problem of spam and abuse; I'm not quite sure how to solve that. Feels like accepting user content no longer flies on the web in 2022 due to the effectiveness of bot spam and lack of viable countermeasures :-/
    - mysterydip 4y ago
      I think non-crowdsourced manual curation (by you/your trusted volunteers) would be a feature. Sure, search would have more limited results, but really with how much cruft there is I don't see that as a bad thing. Start with the "known good/useful" domains and branch out from there.
      - 4y ago

altdataseller 4y ago

>> If sites don't want to be found and indexed why go to any effort to include them? On the other hand there are millions of small sites out there keen to be found. Then they should treat all bots equally and block Google as well. If they block Google as well, then yes, we should leave them alone. Why give unfair treatment to Google? That's anti-competitive behavior and it just prevents new search engines from being created.

nonrandomstring 4y ago
I think I understand, combined with jeffbee's answer, that these sites are behaving selectively according to who you are. So we're back to "No Blacks or Irish" on the 2022 Internet? What do you think they have against smaller search engines? I can't quite fathom the motives.
- wolfgang42 4y ago
  There are a lot of crawlers out there, and many of them are ill-behaved. When GoogleBot crawls your site, you get more visitors. When FizzBuzzBot/0.1.3 comes along, you’re more likely to get an overloaded server, weird URLs in your crash logs, spam, or any other manner of mess. Small search engines getting blocked is just collateral damage from websites trying to solve this problem with a blunt ban-hammer.
  - bryanrasmussen 4y ago
    This is probably because, as I have learned here on HN, 1 in 4 programmers can't write a decent FizzBuzzBot and should never be hired.
- autoexec 4y ago
  > So we're back to "No Blacks or Irish" on the 2022 Internet? I think this is the inevitable result of surveillance capitalism. Once you can reliably be identified as holding certain views or meeting some arbitrary criteria pages can be dynamically altered to restrict what you have access to. A racist social platform can look completely benign to anyone who isn't part of the in-group, forums for teens can automatically lock out anyone suspected of being over a certain age, companies can show products or services only to people within a certain class, etc. We already restrict a lot of access to content based on guesses about location. Youtube doesn't even let you see that certain content on their platform exists on their site unless you're logged in from an IP in specific counties (as opposed to showing you the videos exist and throwing up an error saying it's not available for you when you try to view it) and netflix blocks VPNs just to try to keep a bunch of their content away from the wrong kinds of people. These are the kinds of barriers the internet should have freed us from, but instead it's being used to put up more gates and to force people into them.

amelius 4y ago

> Something is deeply wrong with such an adversarial ecosystem. If sites don't want to be found and indexed why go to any effort to include them? I think it is not about being found. It is more about being copied. These sites are afraid their content is stolen, so they only allow Google to crawl them.

noncoml 4y ago

They don’t want to be indexed unless you are Google

throwaway81523 4y ago

> If sites don't want to be found and indexed why go to any effort to include them? The news sites etc. want to be indexed by Google, and they whitelist Google so they show up in search results. They don't want to be crawled by random aggregators and scrapers which don't bring them ad views and subscriptions. Indexing for me and not for thee. Other places like Slack channels are private or semi-private and shouldn't be indexed, I agree. There was some kind of scheme a while back (before Freenode went down the tubes in a completely different and unrelated way) to index Freenode. People rightfully freaked out about it and the scheme was shot down.

anotherrandom 4y ago

Sounds like federation might be making a comeback eventually