3 ms·
This is a novel concept. I wonder if / when Marginalia will get onboard and implement support for it, too. It would be cool to be able to push the update sign
by metadat 4y ago
This is a novel concept. I wonder if / when Marginalia will get onboard and implement support for it, too.
It would be cool to be able to push the update signal to a bunch of search engines when I publish a new page (even if all of my websites get virtually no traffic and don't even come up with the appropriate, highly unique keyword combos in Google, Bing, or Marginalia - they don't even have any ads or anything terrible, perhaps terribly boring or SEO unoptimized, haha).
I wonder if there could be a market for something which collects website change information and offers an API to query for new pages / updated pages over the past X time interval across Y set of [interesting] websites. I could see this being useful, sort of like RSS but 100% general purpose.
I'd call it something like Invertdex.
- marginalia_nu 4y ago> I wonder if / when Marginalia will get onboard and implement support for it, too. I created an issue for it as a reminder. Probably not gonna implement support for this type of active crawling hints in the short term, because a lot of my model is based on full-site crawls at 8 week intervals. While something like this would help with identifying new content, it doesn't do much to identify when links go dead, so you still need to crawl passively. Although, on some level I'm a bit uneasy, I have a hunch this may be a bit of a vulnerability. In general giving websites the tools to control the crawling process beyond like robots.txt and so on seems a bit sketch. Maybe it's possible to build checks and balances to prevent that though. Would take some work to support real-time updates. Not impossible, but I've got a lot of work to do with regard to search result accuracy that I feel is more important. I've been looking at using RSS feeds as a signal for when to re-index sites before, not doing that now but the general idea works I think