The company I work for uses Splunk to ingest about 70TB/day. Our services send to fluentd running on each instance which aggregate and flush to a Kinesis strea

The company I work for uses Splunk to ingest about 70TB/day. Our services send to fluentd running on each instance which aggregate and flush to a Kinesis stream in AWS with KCL workers responsible for putting it through a separate pipeline that allocates the logs to specific indexes depending on the service(s) they come from as well as applying ACLs on a per-index basis.