Used to work at a company that produced 20 gigs of analytics every day which is probably the biggest data I'll ever work on. My junior project was writing some

Used to work at a company that produced 20 gigs of analytics every day which is probably the biggest data I'll ever work on. My junior project was writing some data crunching jobs that did aggregations batched and in real time, and store the result in parquet blobs in azure. My boss was smart enough to have stakeholder meetings where they regularly discussed what to keep and what to throw away, and with some smart algorithms we were able to compress all that data down into like 200MB per day. We loaded the last 2 months into an sql server and the last 2 years further aggregated into another, and the whole company used the data in excel to do queries on it in reasonable time. The big data is rotting away on tape storage in case they ever need it in the future. My boss got a lot of stuff right and I learned a lot, though I only realized that in hindsight. Dude was a bad manager but he knew his data.