At Netsil (microservice observability startup), API analytics and stream-processor was built using a dialect of Datalog. Netsil was acquired by Nutanix in 2018

At Netsil (microservice observability startup), API analytics and stream-processor was built using a dialect of Datalog. Netsil was acquired by Nutanix in 2018 and the technology was incorporated into a couple of product lines. The Datalog engine at Netsil is pretty sophisticated, capable of fine-grained parallelism (built on task-stealing architecture) - Datalog programs are chained together in a pipeline, such that the output of one program is an input to another. These programs (stages in the pipeline) can be run in serial, parallel or context_ordered modes. In context_ordered mode the tuples are sent to the execution engine ordered by their context_keys -- e.g. captured packets from the same flow are processed serially and in parallel across flows. In addition, it incorporates SQLite embedded DB for materialized tables and state. The raw events are aggregated using Datalog rules & tables and SQL queries are auto-generated from high-level Datalog rules. We also extended Datalog with the concept of "Accumulators", stateful objects in rule head that emit a tuple when certain conditions are met - e.g. we build Windowed Counters, TCP re-assembly from network packets etc. using Accumulators. In addition to all this, one could distribute Datalog programs across machines -- Tuples serialize to protobuf and sent across machines over ZeroMQ etc. We used distributed features by sending raw metrics from the Host based Agents on customer's machines to the cloud service where the Datalog based stream processor joins the incoming metrics/events tuples with metadata collected from external sources (e.g. K8S tags). The Datalog engine also synchronized some tables with Zookeeper for distributing config updates in real-time such as metadata tables that were used in stream-joins. E.g. each time a new K8S pod was detected in customer environment, an update tuple was sent to cloud, which updated the Zookeeper backed materialized table thereby synchronizing with all other stream-processor pods in near-realtime.