I wrote a search engine in school. Since I was naive/lazy/wanted a large corpus it didn't follow robots.txt. It ended up DOSing multiple sites, including Duke's

I wrote a search engine in school. Since I was naive/lazy/wanted a large corpus it didn't follow robots.txt. It ended up DOSing multiple sites, including Duke's law school during class sign up. I don't think those sites wanted to be featured on my search engine.