7 ms·

I'd love a truly open source world class search engine. Curious how both the crawler and the search index / search is done by the likes of Google/Bing/DDG. Even

by nikivi 6y ago

I'd love a truly open source world class search engine. Curious how both the crawler and the search index / search is done by the likes of Google/Bing/DDG. Eventually someone will make an oss version of it that can compete. The beauty of such oss solution maybe the custom heuristics that can be created based off the crawled data.

ddorian43 6y ago
Good luck with that mate. Check out https://commoncrawl.org/ https://commoncrawl.org/
Mediterraneo10 6y ago
The challenges to OSS developers are numerous. First of all, many popular sites on the internet block crawlers other than Google and Bing, because only those ones seem to matter to their business, and any small upstart would be assumed to be a dodgy bot. Secondly, Google amasses the database it has only with vast data centers, incredible amounts of bandwidth, and power requirements unavailable to a startup.
- creese 6y ago
  How would anyone block a crawler? A crawler is just a headless browser.
  - tleb_ 6y ago
    robots.txt https://www.robotstxt.org/ https://www.robotstxt.org/ https://en.wikipedia.org/wiki/Robots_exclusion_standard https://en.wikipedia.org/wiki/Robots_exclusion_standard
    - Xylakant 6y ago
      Note that robots.txt is a hint to well-behaved crawlers, not blocking them in any regard. You can block crawlers if you can identify them, but reliably identifying them is hard.
      - tleb_ 6y ago
        We should probably classify the crawler identifying problem as impossible and move along. Less resources wasted and easier automation for everyone. Assuming a crawler is malicious is narrow-minded.
        ddorian43 6y ago
        https://developers.google.com/search/docs/advanced/verifying-googlebot https://developers.google.com/search/docs/advanced/verifying...
        Xylakant 6y ago
        This helps to verify that a bot that announces itself as google bot is indeed a google bot. It doesn’t help identify a bot that pretends to be a user/browser.
- aembleton 6y ago
  Just use the string 'googlebot' in your user agent. After all, Google uses 'Mozilla' in their Google Bot user agent String for similar reasons - because sites might expect it.
  - ddorian43 6y ago
    https://developers.google.com/search/docs/advanced/verifying-googlebot https://developers.google.com/search/docs/advanced/verifying...
    - aembleton 6y ago
      That's interesting. I know a few companies that don't run that check but instead just go off the user agent.
anotherdirtbag 6y ago
There's no need to compete. People who want things like this just do it themselves. Checkout YaCy https://github.com/yacy/yacy_search_server https://github.com/yacy/yacy_search_server

by nikivi 6y ago

ddorian43 6y ago

Good luck with that mate. Check out https://commoncrawl.org/ https://commoncrawl.org/

Mediterraneo10 6y ago

The challenges to OSS developers are numerous. First of all, many popular sites on the internet block crawlers other than Google and Bing, because only those ones seem to matter to their business, and any small upstart would be assumed to be a dodgy bot. Secondly, Google amasses the database it has only with vast data centers, incredible amounts of bandwidth, and power requirements unavailable to a startup.

creese 6y ago
How would anyone block a crawler? A crawler is just a headless browser.
- tleb_ 6y ago
  robots.txt https://www.robotstxt.org/ https://www.robotstxt.org/ https://en.wikipedia.org/wiki/Robots_exclusion_standard https://en.wikipedia.org/wiki/Robots_exclusion_standard
  - Xylakant 6y ago
    Note that robots.txt is a hint to well-behaved crawlers, not blocking them in any regard. You can block crawlers if you can identify them, but reliably identifying them is hard.
    - tleb_ 6y ago
      We should probably classify the crawler identifying problem as impossible and move along. Less resources wasted and easier automation for everyone. Assuming a crawler is malicious is narrow-minded.
      - ddorian43 6y ago
        https://developers.google.com/search/docs/advanced/verifying-googlebot https://developers.google.com/search/docs/advanced/verifying...
        Xylakant 6y ago
        This helps to verify that a bot that announces itself as google bot is indeed a google bot. It doesn’t help identify a bot that pretends to be a user/browser.
aembleton 6y ago
Just use the string 'googlebot' in your user agent. After all, Google uses 'Mozilla' in their Google Bot user agent String for similar reasons - because sites might expect it.
- ddorian43 6y ago
  https://developers.google.com/search/docs/advanced/verifying-googlebot https://developers.google.com/search/docs/advanced/verifying...
  - aembleton 6y ago
    That's interesting. I know a few companies that don't run that check but instead just go off the user agent.

anotherdirtbag 6y ago

There's no need to compete. People who want things like this just do it themselves. Checkout YaCy https://github.com/yacy/yacy_search_server https://github.com/yacy/yacy_search_server