> Citation needed, because if you do this, you'll also cut yourself off every search engine in existence. Most major search engine has dedicated blocks of addr

> Citation needed, because if you do this, you'll also cut yourself off every search engine in existence. Most major search engine has dedicated blocks of addresses and uses unique user-agents. If you just literally sent wget or curl requests, you will be identified as a "bad" crawler almost immediately.