A 2016 work project I did requires to crawl 100 million web pages from around 100k different web sites each night. There was no existing infrastructure what so

A 2016 work project I did requires to crawl 100 million web pages from around 100k different web sites each night. There was no existing infrastructure what so ever. I was told that it is a big data project and internet scale processing power is required (and actually budgeted for it). Built a Python crawler to pull data from the Internet using EC2 spot instances, as they are dirty cheap. After just a few days happy coding, I eventually got to a place where I can reliably (spot instances are not that reliable) pull the data, compress them and download them to our own data centre for processing (policy rather than design decision) for under $50 with the majority spent on transferring the final compressed output. Big data? 100 million small files a run is a typical small data project, big data means trillions of files a day, or something like exabytes scale data volume.