it’s not. stuff logs into s3. learn to mine them in parallel using lambda or ec2 spot. grow tech or teams as needed for scale. never egress data and never pers

it’s not. stuff logs into s3. learn to mine them in parallel using lambda or ec2 spot. grow tech or teams as needed for scale. never egress data and never persist data outside of the cheapest s3 tiers. expire data on some sane schedule. data processing is fun, interesting, and valuable. it is core to understanding your systems. if you can’t do this well, there is probably a lot more you can’t do well either. in that case, life is going to be very expensive. it’s ok to not do this well yet! spend some portion of your week doing this and you will improve quickly.