Browsers are merely one sort of "user agent"; it was envisaged that all sorts of agents, mostly automated, would be crawling the Web, e.g. fetching news, findin

Browsers are merely one sort of "user agent"; it was envisaged that all sorts of agents, mostly automated, would be crawling the Web, e.g. fetching news, finding the cheapest price for some widget, etc. Unfortunately, only browsers and search indexers seem to have caught on (and many sites are actively hostile to anything else)