shaeqahmed

5 ms·

1.
by shaeqahmed 4y ago
Matano is completely serverless and stores all data in ZSTD compressed parquet files in dirt-cheap object storage, allowing you to bring your own analytics stack for queries on large amounts of data for things like investigations and threat
2.
by shaeqahmed 4y ago
Yep, SIEM is just a superset of Log Management as it needs to do things like alerting + correlation + detection etc. in addition to ingesting logs to be considered a SIEM. It is a common use case to send application logs along with security
3.
by shaeqahmed 4y ago
Thank you for typing up a long detailed response. I think a lot of the points and concerns you bring up are valid, and we are mostly agreed upon. In Matano however, we see Python as a viable component in security operations for narrowly tra
4.
by shaeqahmed 4y ago
Long term, I believe Python (along with good ol' SQL for correlation) is the best language to model the kind of attacker behaviours companies are dealing with in the cloud and a lot of the difficulties with it are not inherent but arou
5.
by shaeqahmed 4y ago
The code is written in high performance multi-threaded Rust and uses the [1] Arrow compute framework. We also batch events and target about 32MB of event data per lambda invocations. As a result it can process tens of thousands of events pe
6.
by shaeqahmed 4y ago
We launched before Amazon Security Lake :) Amazon Security Lake's main value prop is that it is a single place where AWS / partner security logs can be stored and sent to downstream vendors. As such, Amazon only writes OCSF normal
7.
by shaeqahmed 4y ago
Some big differences: - Matano has realtime Python + SQL detections as code with advanced correlation support. Chronicle uses inflexible YARA-like detection rules iirc - Matano supports Sigma detections by automatically transpiling them to
8.
by shaeqahmed 4y ago
Thank you! We definitely believe in open source and don't need AGPL. Sending you love as you deal with that Splunk instance. P.S. feel free to open some issues for any log sources you'd like to see supported in Matano
9.
by shaeqahmed 4y ago
Thank you! Yes with AppTrail we wanted to solve the pain points around SaaS audit logs but since it was a product that needed to be sold and integrated into B2B startups rather than the enterprises that felt the pain points and needed audit
10.
by shaeqahmed 4y ago
We are working on a solution for GCP and Azure :) GCP recently announced Iceberg support with BigLake and support for federation across multi-cloud lakes so it would be perfect use cases. If you are interested in using Matano for GCP, feel
11.
by shaeqahmed 4y ago
I completely agree with you and the need for a fully integrated solution with great visualizations without hosting additional tools that aren't purpose built! Unfortunately there are very few SIEMs that get this right today.. Here&#x2
12.
by shaeqahmed 4y ago
Many enterprises using Splunk are already being forced to purchase products like Cribl to route some of their data to a data lake because writing it all to Splunk is just way too expensive at that scale 1-100TB+/day (7 figures $). But
13.
by shaeqahmed 4y ago
We think building a more efficient solution using data lakes is a win-win because it unlocks additional use cases for customers and allows them to analyze larger datasets within the same budget. Solutions that offer a magnitude of order bet
14.
by shaeqahmed 4y ago
Nope, Matano is named after one of the deepest lakes in the world - Lake Matano of Indonesia ;)
15.
by shaeqahmed 4y ago
I'm on the fence about this one. When I heard it was announced, and that it was created by none other than the creators of the amazing AWS CDK, I was really excited by what could be possible. Having worked on complex infra automation u
16.
by shaeqahmed 4y ago
okay but what about biometric auth via fingerprint - passwordless & zero context switches (provided your device supports this)
17.
by shaeqahmed 4y ago
Yes, they would be handled automatically. Data ingestion is supported through S3 or Kafka, where files are picked up and ETL'D into structured Iceberg tables conforming to an ECS-like schema. Feel free to join our Discord, happy to wa
18.
by shaeqahmed 4y ago
Matano is designed specifically for petabyte-scale security log analytics use cases, so performance and costs are a top priority. Our data pipeline borrows from Vector's Rust based data transformation language [0] for maximal performan
19.
by shaeqahmed 4y ago
Using and maintaining Matano is a fraction of the cost compared to popular non-serverless alternatives like ELK or Spunk. Matano is specifically designed for petabyte-scale security analytics use-cases that don't fit in a traditional S
20.
by shaeqahmed 4y ago
It is similar, although Snowflake is more of a query engine whereas we are a cloud security data platform built on an open data model (Apache Iceberg). We help you ingest and normalize data from common security sources into a data lake and
21.
by shaeqahmed 4y ago
Its a data lake in which you store security logs. That includes Cloud/SaaS audit logs, network security logs (Zeek, Suricata, Snort), VPN/Firewall logs, and more.
22.
Show HN: Open-source serverless security lake powered by Rust + Apache Iceberg(github.com)
63 points by shaeqahmed 4y ago|48 comments
23.
by shaeqahmed 4y ago
Cool product and pricing model > Cloud Log Lake That's the first time I'm hearing a Clickhouse backend described as a lake. Care to explain?
24.
by shaeqahmed 5y ago
> Customers also use Convoy as a broker for inter-service communication. Very cool product! But this part threw me off a bit, wouldn't a distributed message broker like Kafka be a much better option for this use-case?
25.
Fresher Data Lake on AWS S3(robinhood.engineering)
1 points by shaeqahmed 5y ago|0 comments
26.
by shaeqahmed 5y ago
A CDC alone wouldn't. You would need to track and store that metadata in an additional database along with transaction ID, so it can be joined with the raw change stream to form a complete audit log.
27.
by shaeqahmed 5y ago
Looks cool, I've tested out some of these React-inspired Rust UI frameworks and just can't seem to get over the convoluted macros with no syntax highlighting. Why hasn't anyone taken an approach to create something like .rsx