3 ms·

This and the other engines seem to implement all the components of crawling, indexing, and searching strung together. Is there a reason for this? Wouldn't an op

This and the other engines seem to implement all the components of crawling, indexing, and searching strung together. Is there a reason for this? Wouldn't an option of, let's say, crawling + indexing made available separately, where others could built a search algorithm on top of, or just the crawling as a service made available. Are there stuff like these already available? Or is it just not a viable option?

ddorian43 3y ago
There is common crawl: https://github.com/commoncrawl https://github.com/commoncrawl
- marginalia_nu 3y ago
  Kinda sucks that it's stuck in AWS with no easy way of exfiltrating the data from the Amazon ecosystem. Last I tried I got like 100 Kb/s on their HTTP mirror. At that rate, the download would take 12 years.
  - ddorian43 3y ago
    I just tried 2 http examples from https://commoncrawl.org/get-started https://commoncrawl.org/get-started for an old dataset and the most recent one and got 110Mb/s (my full download bandwidth): wget https://data.commoncrawl.org/crawl-data/CC-MAIN-2018-17/segments/1524125937193.1/warc/CC-MAIN-20180420081400-20180420101400-00000.warc.gz wget https://data.commoncrawl.org/crawl-data/CC-MAIN-2023-23/segments/1685224643388.45/warc/CC-MAIN-20230527223515-20230528013515-00483.warc.gz
deleted 3y ago
[deleted]
marginalia_nu 3y ago
Crawling can be done collaboratively, but there's not a lot of point to doing this. Crawling is the cheap and easy part. As for the rest, in order to perform well, the indexer needs to be built specifically tailored to the what the search engine is doing. Often you're scrounging for places to cram in individual bits to encode some additional piece of information about the term. If a DBMS tries to support every use case, a search engine index does the opposite, it supports a singular use case and cuts every corner imaginable and then some to make that happen with as much resource frugality as possible.
- daoudc 3y ago
  Thanks, Marginalia search was a big inspiration for this project!

ddorian43 3y ago

There is common crawl: https://github.com/commoncrawl https://github.com/commoncrawl

marginalia_nu 3y ago
Kinda sucks that it's stuck in AWS with no easy way of exfiltrating the data from the Amazon ecosystem. Last I tried I got like 100 Kb/s on their HTTP mirror. At that rate, the download would take 12 years.
- ddorian43 3y ago
  I just tried 2 http examples from https://commoncrawl.org/get-started https://commoncrawl.org/get-started for an old dataset and the most recent one and got 110Mb/s (my full download bandwidth): wget https://data.commoncrawl.org/crawl-data/CC-MAIN-2018-17/segments/1524125937193.1/warc/CC-MAIN-20180420081400-20180420101400-00000.warc.gz wget https://data.commoncrawl.org/crawl-data/CC-MAIN-2023-23/segments/1685224643388.45/warc/CC-MAIN-20230527223515-20230528013515-00483.warc.gz

deleted 3y ago

[deleted]

marginalia_nu 3y ago

Crawling can be done collaboratively, but there's not a lot of point to doing this. Crawling is the cheap and easy part. As for the rest, in order to perform well, the indexer needs to be built specifically tailored to the what the search engine is doing. Often you're scrounging for places to cram in individual bits to encode some additional piece of information about the term. If a DBMS tries to support every use case, a search engine index does the opposite, it supports a singular use case and cuts every corner imaginable and then some to make that happen with as much resource frugality as possible.

daoudc 3y ago
Thanks, Marginalia search was a big inspiration for this project!