4 ms·

Awesome project! How are you able to keep the site running after HN kiss of death? What is your stack, elastic search or something simper? How did you crawl so

by soheil 5y ago

Awesome project! How are you able to keep the site running after HN kiss of death? What is your stack, elastic search or something simper? How did you crawl so many websites for a project this size? Did you use any APIs like duck duck go or data from other search engines? Are you still incorporating something like PageRank to ensure good results are prioritized or is it just the text-based-ness factor?

marginalia_nu 5y ago
> How are you able to keep the site running after HN kiss of death? I originally targeted a Raspberry Pi4-cluster. It was only able to deal with about 200k pages at that stage, but it did shape the design in a way that makes very thrifty use of the available hardware. My day job is also developing this sort of highly performance java applications, I guess it helps. > What is your stack, elastic search or something simper? It's a custom index engine I built for this. I do use mariadb for some ancillary data and to support the crawler, but it's only doing trivial queries. > How did you crawl so many websites for a project this size? It's not that hard. Like it seems like it would be, and there certainly is an insane number of edge cases, but if you just keep tinkering you can easily crawl dozens of pages per second even on modest hardware (of course distributed across different domains). > Did you use any APIs like duck duck go or data from other search engines? Nope, it's all me. > Are you still incorporating something like PageRank to ensure good results are prioritized or is it just the text-based-ness factor? I'm using a somewhat convoluted algorithm that takes into consideration the text-based-ness of the page, but also how many incoming links the domain has, but it's a weighted value that factors in the text-based-ness of the origin domains. It would be interesting to try a page rank-style approach, but my thinking is that because it's the algorithm, it's also the algorithm everyone is trying to game.

by soheil 5y ago

marginalia_nu 5y ago

> How are you able to keep the site running after HN kiss of death? I originally targeted a Raspberry Pi4-cluster. It was only able to deal with about 200k pages at that stage, but it did shape the design in a way that makes very thrifty use of the available hardware. My day job is also developing this sort of highly performance java applications, I guess it helps. > What is your stack, elastic search or something simper? It's a custom index engine I built for this. I do use mariadb for some ancillary data and to support the crawler, but it's only doing trivial queries. > How did you crawl so many websites for a project this size? It's not that hard. Like it seems like it would be, and there certainly is an insane number of edge cases, but if you just keep tinkering you can easily crawl dozens of pages per second even on modest hardware (of course distributed across different domains). > Did you use any APIs like duck duck go or data from other search engines? Nope, it's all me. > Are you still incorporating something like PageRank to ensure good results are prioritized or is it just the text-based-ness factor? I'm using a somewhat convoluted algorithm that takes into consideration the text-based-ness of the page, but also how many incoming links the domain has, but it's a weighted value that factors in the text-based-ness of the origin domains. It would be interesting to try a page rank-style approach, but my thinking is that because it's the algorithm, it's also the algorithm everyone is trying to game.