2 ms·
> I wonder if / when Marginalia will get onboard and implement support for it, too. I created an issue for it as a reminder. Probably not gonna implement supp
by marginalia_nu 4y ago
> I wonder if / when Marginalia will get onboard and implement support for it, too.
I created an issue for it as a reminder. Probably not gonna implement support for this type of active crawling hints in the short term, because a lot of my model is based on full-site crawls at 8 week intervals. While something like this would help with identifying new content, it doesn't do much to identify when links go dead, so you still need to crawl passively.
Although, on some level I'm a bit uneasy, I have a hunch this may be a bit of a vulnerability. In general giving websites the tools to control the crawling process beyond like robots.txt and so on seems a bit sketch. Maybe it's possible to build checks and balances to prevent that though.
Would take some work to support real-time updates. Not impossible, but I've got a lot of work to do with regard to search result accuracy that I feel is more important.
I've been looking at using RSS feeds as a signal for when to re-index sites before, not doing that now but the general idea works I think