4 ms·
maybe it's time to centralize crawlers and search data, making it publicly available to anyone willing to process it. Not only would that reduce crawler traffi
by sigmonsays 2y ago
maybe it's time to centralize crawlers and search data, making it publicly available to anyone willing to process it.
Not only would that reduce crawler traffic on websites to a single crawler entity it would make page data available for any indexer.
The idea that one company owns this data is kind of silly, it should be a coalition or a group of companies working together...
- RunningDroid 2y agoI think a better design would be to define a spec for search data so sites that implement it could generate a ".well-known/search_data.zst" (or whatever) and people would only need to crawl the site to check compliance with the spec.