During my time at Wanderu we set up Filebeat on all app machines, forwarding json logs to Kafka, and then on to ELK and Pipelinedb for analysis/alerting/session

During my time at Wanderu we set up Filebeat on all app machines, forwarding json logs to Kafka, and then on to ELK and Pipelinedb for analysis/alerting/session killing/ etc. It worked very well, as there are almost always application edge cases that should not be blocked immediately with Cloudflare or at a FW level. ELK was used for exploration and/or setting alerts on thresholds that we had found/defined beforehand. Pipelinedb was used to run continuous aggregates on a stream, augment (pop json log lines off a stream, enrich them with Maxmind, etc) logs in "realtime", and immediately surface malicious behavior. We'd then programmatically kill sessions or add Cloudflare rules upstream, based on the behavior that surfaced in a Pipelinedb continuous aggregate. Wanderu: https://www.wanderu.com/ https://www.wanderu.com/ Filebeat: https://www.elastic.co/products/beats/filebeat https://www.elastic.co/products/beats/filebeat PipelinedB: https://www.pipelinedb.com/ https://www.pipelinedb.com/