3 ms·
Yes. Gigablast is run by one person. Assuming you roll your own hardware and co-lo it or such you can get by with minimal hardware. The catch being once you hit
by boyter 5y ago
Yes. Gigablast is run by one person. Assuming you roll your own hardware and co-lo it or such you can get by with minimal hardware. The catch being once you hit scale your servers become hammered. It’s one reason why I have been investigating using aws lambda to hold and search the index. Solves the initial scale problem.
Honestly though the big issue is crawling. Not only is it a bandwidth monster many sites are hostile to non google bots, along with cdn’s and cloudflare.
- marginalia_nu 5y ago> Honestly though the big issue is crawling. Not only is it a bandwidth monster many sites are hostile to non google bots, along with cdn’s and cloudflare. I do my own crawling and don't agree with any of these statements. Bandwidth is not a bottleneck, and blocking is mostly only a problem if your bot is too aggressive.
- boyter 5y agoCould be due to the long form content you index. I found those sort of sites tend to have less reluctance on 3rd parties. Also possible you are more talented than I with your crawler writing.
- ColinHayhurst 5y agoCrawling is a challenge but not as much as mentioned. We have had a respectful bot since 2004, so the problems for us are more about awareness than hostility. Indexing is the bigger challenge. An efficient fast service for an index of billions of pages is a different level to that for millions of pages.
- boyter 5y agoI have always found crawling harder personally. At least with indexing you can switch to batch processing, or split the index into real-time and stale portions to give that fresh feel. Just my personal opinion though. I have never had the chance to index billions of pages. Several hundred million is about as far as I have gone.