Also we need to talk about what should be logged and where. There seem to be two schools of thought, just enough to tell something is wrong but not what it is

Also we need to talk about what should be logged and where. There seem to be two schools of thought, just enough to tell something is wrong but not what it is - OR - you get to drink from the firehose. And most orgs go from the first to the second. As to where, well thats at the hardware/vm/container level, and mirror and extend what it does. Nothing worse than 20 different ideas of how to log and rotate and trying to figure out who did what, when where and why. If you can't match a log entry to a running environment... well. I weep quietly inside when some or all of this goes through one, or several S3 buckets for no good reason.