4 ms·
Congratulations, I always enjoy new search engines W.r.t. "and updated intermittently," I wanted to draw your attention to the HN realtime API: https://github.
by mdaniel 4y ago
Congratulations, I always enjoy new search engines
W.r.t. "and updated intermittently," I wanted to draw your attention to the HN realtime API: https://github.com/HackerNews/API#live-data https://github.com/HackerNews/API#live-data and also that S.O. offers Atom Feeds: https://stackoverflow.com/feeds/ https://stackoverflow.com/feeds/ (I'd guess the rest do, too, but I didn't verify)
I am a huge proponent of taking advantage of any update features that a site offers, because otherwise the "how about now?" of re-crawling is wasteful to both parties.
- wolfgang42 4y agoMy current architecture is built around batch ingestion[1], and doesn’t (yet) have a way to do incremental updates. This is great for getting coverage—there are a lot of long-tail results in my search engine but not in Google!—but it does mean there’s more lag and the results aren’t instantly up-to-date. [1]: e.g. for StackOverflow, I download an XML dump of the entire site once a quarter: https://search.feep.dev/blog/post/2021-09-04-stackexchange https://search.feep.dev/blog/post/2021-09-04-stackexchange