4 ms·
I built my own, it's easier than you might think: https://greppr.org/ https://greppr.org/
by saltysalt 1y ago
I built my own, it's easier than you might think: https://greppr.org/ https://greppr.org/
- uticus 1y agowould be interested in directions on how to do this myself. like, take existing bookmarks, make a crawler. nice work, btw
- saltysalt 1y agoThanks! If I have time, I should write a how-to blog.
- commandersaki 1y agoI've not written a crawler before, but did something similar. I needed to mirror websites ala `wget -r` and there doesn't seem to be a tool or library that aside from wget that does it, so I translated the wget -r algorithm by reading the source, as best as I could, in Go. It's not parallelised or anything as that looked complicated, but was handy when integrating it into a backend project that needed that functionality. Was a fun learning experience and I found it a bit of a complex project due to interpreting the links in HTML, so I imagine doing a crawler is even more difficult. Also found Go HTML parser not that great.
- mbirth 1y agoThere’s also Marginalia: https://marginalia-search.com/ https://marginalia-search.com/
- saltysalt 1y agoVery nice!
- busymom0 1y agoWhat's the cost and architecture of running such a site and crawling all those pages?
- saltysalt 1y agoA single bare-metal server!
- busymom0 1y agoOh wow. Do you need a proxy or something to be able to crawl so many pages? Can I email you with some technical questions? Thanks.
- saltysalt 1y agoNo proxy yet, but I am considering one as many sites are re-directing my crawler based on its IP, which is causing indexing issues. The hardest part BY FAR is the crawler: initially I was using Apache Nutch but it got slower and slower as the index grew, so I replaced it with my own crawler that I wrote in PHP (comfortable for me) and made that multi-threaded using Supervisor. The second hardest part was the amount of security I had to build in to prevent bots running spam searches and hogging my infra. I'll try to write a blog soon and post it here.
- busymom0 1y agoDo you have multiple IPs? I am trying to build something which needs just the published at and updated at date fields for thousands of links and I am afraid my IP will get blocked quickly.
- saltysalt 1y agoJust one IP for now. You are right to worry about being blocked from crawling however, it has happened to me already on a few sites. The key things to help mitigate against this are: 1. Always identify your crawler via a consistent user-agent string, that explains its a web search crawler and not a generic web browser. 2. Always obey the directives in robots.txt. 3. Make sure your crawler is not too aggressive (low frequency of requests). (updated for formatting)
- xigoi 1y agoIt seems to me that building a curated search engine is much easier than a general-purpose search engine.
- saltysalt 1y agoYou can achieve both in my opinion.
- machiaweliczny 1y agoNice, what are you using as storage?
- saltysalt 1y agoLucene on local disc.