In the 90s I spent a lot of time on my website and I loved learning how web crawlers worked. I started using a robots.txt file without really understanding it.

In the 90s I spent a lot of time on my website and I loved learning how web crawlers worked. I started using a robots.txt file without really understanding it. I ended up blocking everything thinking it would make my site faster for visitors because--crawlers might crawl the site all the time. After I graduated from college I lost access to my website which was hosted on the Computer Science department's web servers. I wish I hadn't used that robots.txt file. I would love to find the pages I made that compared interfold vs. exterfold staple strength, or the site I made with a ranch theme with a cowboy that had humorous advice....I don't have any content in archive.org because it honored the robots.txt file. ...sigh...wish I had backed up my stuff.