3 ms·
Some days I wonder what would stop me writing my own web crawler and building my own damn search engine from scratch. I can't ever come up with a hard reason it
by bob1029 5y ago
Some days I wonder what would stop me writing my own web crawler and building my own damn search engine from scratch. I can't ever come up with a hard reason it wouldnt work in 2022, other than the dynamic javascript blogspam that may be more challenging to trawl for meaningful content these days. Perhaps this is a win-win: The shit I can't crawl I wouldnt want to read anyways.
Compete with google? Who gives a shit. I just want to be able to hit a full text index that points back to URLs. I don't need instant PhD-tier answers for life questions.
- acatton 5y ago(Disclaimer: I work for a big company which makes a browser, an OS for smartphones and a search engine. This is my own opinion, and does not reflect my employer, and so on, and so on…) > Some days I wonder what would stop me writing my own web crawler and building my own damn search engine from scratch. A few things: * Storing the index. For example if I search for "Sushi", google says there is 1.3 Billion results. That's already 19.3 GiB just to store the relationship between "sushi" and its results. (If you use UUIDs) And you need to do that for most words in the english dictionnary. And I'm assuming you're not even storing what you crawl for debugging/re-indexing. * Website admins might throttle you. For example, discourse — a website which runs most discussion for Rust, Ocaml, and many other projects — blocked the bingbot at some point[1] due to its crawling aggressivity. Cloudflare, which is the front-facing caching service for almost half of the internet now, is notoriously anti-crawling. (It will quickly display captchas) * grep over the web will yield you bad results. (most of the time) You want a search engine which at least groups words semantically. If I search for horses, I'm usually also interested in mares and ponies. [1] https://meta.discourse.org/t/bingbot-is-no-longer-default-throttled/84620 https://meta.discourse.org/t/bingbot-is-no-longer-default-th...