3 ms·

How do you even build a search index today when websites barely link to each other? Nowadays the bulk of linking goes to ecommerce sites (amazon) from content

How do you even build a search index today when websites barely link to each other? Nowadays the bulk of linking goes to ecommerce sites (amazon) from content farms (reddit) and all those sites are submitted directly to Google. I don't think crawlable internet exists anymore.

gostsamo 1y ago
You are making a broad generalization and even it is based on the assumption that the page-ranking algorithm is the only possible way to do it.
- scotty79 1y ago
  I make no assumption beyond the one that if you want to index a page you need to know its address and if the author of the website is not going to give it to you because you are not Google and no known website links to it, then you have no way of finding it out. You can't build anything independently. You have to go to Google.
loa_in_ 1y ago
It isn't impossible nowadays to enumerate domain names using DNS data and score them based on the content they serve. Isn't that what we really want as users? Scoring based not on proxies for relevance like referral count, but on viewable content?
- guillem_lefait 1y ago
  It's possible indeed as I'm doing it for another reason (monitoring sovereigty). You can ask Icann [0] access to gTld domain list files (if you have a legitimate reason to do so). Once access you are granted access to a gTld, you can download a compressed csv file with a line per couple <domain, nameserver>. [0] https://czds.icann.org/home https://czds.icann.org/home
NitpickLawyer 1y ago
> How do you even build a search index today You can start with seeds like common crawl, and go from there. You can also get DNS records from various providers. Then there's SSL cert logs that you can crawl. Plenty of sources, if you have funding (search by itself without ads sponsoring it might be a net loss, except some niche uses like kagi?)

3 ms·

How do you even build a search index today when websites barely link to each other? Nowadays the bulk of linking goes to ecommerce sites (amazon) from content

by scotty79 1y ago

gostsamo 1y ago
You are making a broad generalization and even it is based on the assumption that the page-ranking algorithm is the only possible way to do it.
- scotty79 1y ago
  I make no assumption beyond the one that if you want to index a page you need to know its address and if the author of the website is not going to give it to you because you are not Google and no known website links to it, then you have no way of finding it out. You can't build anything independently. You have to go to Google.
loa_in_ 1y ago
It isn't impossible nowadays to enumerate domain names using DNS data and score them based on the content they serve. Isn't that what we really want as users? Scoring based not on proxies for relevance like referral count, but on viewable content?
- guillem_lefait 1y ago
  It's possible indeed as I'm doing it for another reason (monitoring sovereigty). You can ask Icann [0] access to gTld domain list files (if you have a legitimate reason to do so). Once access you are granted access to a gTld, you can download a compressed csv file with a line per couple <domain, nameserver>. [0] https://czds.icann.org/home https://czds.icann.org/home
NitpickLawyer 1y ago
> How do you even build a search index today You can start with seeds like common crawl, and go from there. You can also get DNS records from various providers. Then there's SSL cert logs that you can crawl. Plenty of sources, if you have funding (search by itself without ads sponsoring it might be a net loss, except some niche uses like kagi?)