5 ms·

Looks neat, but in what way is this serverless? It's a pretty complex diagram: https://github.com/matanolabs/matano/blob/main/website/src/assets/diagram.png h

Looks neat, but in what way is this serverless? It's a pretty complex diagram: https://github.com/matanolabs/matano/blob/main/website/src/assets/diagram.png https://github.com/matanolabs/matano/blob/main/website/src/a...

wizwit999 4y ago
There's no servers to maintain in the entire architecture, we heavily use Lambda and even use MSK serverless for Kafka.
- georgyo 4y ago
  This sounds truely nightmarish and costly. Even moderate volumes of data are going to add up very quickly cost wise. I see the term zero-ops. But maintaining and debugging this pipeline is going to require some ops, even if you are not managing VMs.
  - shaeqahmed 4y ago
    Using and maintaining Matano is a fraction of the cost compared to popular non-serverless alternatives like ELK or Spunk. Matano is specifically designed for petabyte-scale security analytics use-cases that don't fit in a traditional SIEM. The serverless data ingestion pipeline means you don't need to over-provision for ingestion (Logstash and Splunk Forwarders are notorious for related costs / ops in high scale use-cases) in the write path. For reads, since Matano queries Iceberg tables backed by highly-compressed parquet files on object storage you won't pay anything close to what you would for a database or search engine based SIEM.
    - mdaniel 4y ago
      > For reads, since Matano queries Iceberg tables backed by highly-compressed parquet files on object storage you won't pay anything close to what you would for a database or search engine based SIEM Where do you show an example of querying anything? There's an empty "detector" in the examples directory, which I guess gets called once per row of this 20MiB/s alleged elsewhere? Anyway, I find comparing this to Splunk to be a bit premature
      - FridgeSeal 4y ago
        Tools like Spark, Trino, etc can be pointed at parquet/iceberg/etc files in S3, and they'll let you issue SQL queries against the files directly. Means it integrates out of the box with whatever data tooling is being used in your org already.
thegagne 4y ago
I am pretty sure they are saying is that the matano portion of it, which does the security log processing, deploys in a serverless fashion (lambda I assume?). This means as a dev you don’t need to maintain a server, or even container image, you just deploy the code, which is less maintenance overhead and more scalable. The diagram just shows how it interacts with the other components of the log pipeline.
- wizwit999 4y ago
  Thanks for explaining! Yep, that's what we mean. We use serverless services like S3, Lambda, Athena, MSK Serverless, Firehose so you don't have to maintain any servers.
deleted 4y ago
[deleted]

wizwit999 4y ago

There's no servers to maintain in the entire architecture, we heavily use Lambda and even use MSK serverless for Kafka.

georgyo 4y ago
This sounds truely nightmarish and costly. Even moderate volumes of data are going to add up very quickly cost wise. I see the term zero-ops. But maintaining and debugging this pipeline is going to require some ops, even if you are not managing VMs.
- shaeqahmed 4y ago
  Using and maintaining Matano is a fraction of the cost compared to popular non-serverless alternatives like ELK or Spunk. Matano is specifically designed for petabyte-scale security analytics use-cases that don't fit in a traditional SIEM. The serverless data ingestion pipeline means you don't need to over-provision for ingestion (Logstash and Splunk Forwarders are notorious for related costs / ops in high scale use-cases) in the write path. For reads, since Matano queries Iceberg tables backed by highly-compressed parquet files on object storage you won't pay anything close to what you would for a database or search engine based SIEM.
  - mdaniel 4y ago
    > For reads, since Matano queries Iceberg tables backed by highly-compressed parquet files on object storage you won't pay anything close to what you would for a database or search engine based SIEM Where do you show an example of querying anything? There's an empty "detector" in the examples directory, which I guess gets called once per row of this 20MiB/s alleged elsewhere? Anyway, I find comparing this to Splunk to be a bit premature
    - FridgeSeal 4y ago
      Tools like Spark, Trino, etc can be pointed at parquet/iceberg/etc files in S3, and they'll let you issue SQL queries against the files directly. Means it integrates out of the box with whatever data tooling is being used in your org already.

thegagne 4y ago

I am pretty sure they are saying is that the matano portion of it, which does the security log processing, deploys in a serverless fashion (lambda I assume?). This means as a dev you don’t need to maintain a server, or even container image, you just deploy the code, which is less maintenance overhead and more scalable. The diagram just shows how it interacts with the other components of the log pipeline.

wizwit999 4y ago
Thanks for explaining! Yep, that's what we mean. We use serverless services like S3, Lambda, Athena, MSK Serverless, Firehose so you don't have to maintain any servers.

deleted 4y ago

[deleted]