4 ms·

I tried to set up YaCy [1] at home to index a few of may favorite smaller websites, so I could quickly search just them. That turned out to be a bad idea. Some

by l72 6y ago

I tried to set up YaCy [1] at home to index a few of may favorite smaller websites, so I could quickly search just them. That turned out to be a bad idea. Some ended up blocking my home IP address and others reported me to my ISP. None of these sites were that large, and I wasn't continuously crawling them... [1] https://yacy.net/ https://yacy.net/

jedimastert 6y ago
How often were you searching?
- l72 6y ago
  I was regularly searching, but I was rarely indexing any of the sites. I struggled to even get an initial index of many of the sites, due to being blocked or being reported.
  - jedimastert 6y ago
    If you were getting blocked on an initial load, you were either hit with rate limiting or an unrecognized user agent
slenk 6y ago
I have been running my own Searx instance in AWS for a while and have not gotten blocked yet anywhere
- lgats 6y ago
  Wow, I've never heard of Searx - https://searx.space/ https://searx.space/ It's privacy friendly proxied search results, a lot like the Nitter project for Twitter. https://github.com/zedeus/nitter/wiki/Instances https://github.com/zedeus/nitter/wiki/Instances