4 ms·
I'm beginning to think that there's no right answer, because the ideal crawling rules depend a lot on what type on content you're seeking, which obviously can't
by rozap 4y ago
I'm beginning to think that there's no right answer, because the ideal crawling rules depend a lot on what type on content you're seeking, which obviously can't be known up front.
I'm working on a federated search tool. I've built a few different sectors (I've called them "realms") that I care about (programming, automotive, racing, fediverse) and found that I need pretty different heuristics depending. Example: car forums circa ~2005 are a treasure trove of valuable information, but 17 year old posts about programming are (in general) less interesting. That informs a lot about how URLs need or do not need to be re-crawled.
- marginalia_nu 4y agoYeah, probably. Could also be the answer isn't more complicated than to offer up the ability to control the search a bit more. Especially with forums it's easy to figure out the post time since there's only some half-dozen forum softwares. Maybe just add the option to filter by year or something would go a long way. I do think offering different filters like that is probably a good option to having the search engine try to mind-read what you want based on spying on your historical queries and machine learning haruspicy.
- rozap 4y agoYea, I don't think the idea has any real value. It kind of falls in between a bookmarking tool and a search engine. But forming search engines from trees of other search engines is kind of cool, and I have stumbled onto knowledge that I wouldn't have otherwise found. I'm not sure what the balance is between mind reading and offering too many knobs to a user is, though.