3 ms·
Crawling noob here: what's a good strategy to crawl these websites? Reddit is huge, how does one efficiently crawl it? Search API for Github links?
by ronyfadel 5y ago
Crawling noob here: what's a good strategy to crawl these websites?
Reddit is huge, how does one efficiently crawl it? Search API for Github links?
- stanislavb 5y agoI've built a custom engine to achieve that (Elixir/Phoenix). It uses a simplified version of the Aho–Corasick algorithm https://en.wikipedia.org/wiki/Aho%E2%80%93Corasick_algorithm https://en.wikipedia.org/wiki/Aho%E2%80%93Corasick_algorithm to match all mentions of subscribed keywords/URLs. I use the same engine for SaaSHub, too.