5 ms·

Introducing Parseable, simple, efficient, cloud native log storage and query system. As Developers, SRE, DevOps look to replace Elastic from their logging stac

Introducing Parseable, simple, efficient, cloud native log storage and query system. As Developers, SRE, DevOps look to replace Elastic from their logging stack - we believe the log storage of future won't be another search and indexing engine in a new language. Parseable leverages Apache Arrow, Parquet and widely available object storage platforms for efficiency, cost effectiveness and performance.

wmanley 4y ago
Are you using S3 Select[1]? [1]: https://aws.amazon.com/blogs/aws/s3-glacier-select/ https://aws.amazon.com/blogs/aws/s3-glacier-select/
wmanley 4y ago
I really like the idea of storing logs in parquet format in S3. It's something I've wanted to play with for a long time, but never have had a use-case. I'm also a big fan of Rust. I imagine you've seen apenwarr's blog post on logging: https://apenwarr.ca/log/20190216 https://apenwarr.ca/log/20190216 With that in mind I'd like to read more about the architecture. Specifically: 1. Do you have separate components for ingest and querying that can be scaled independently? 2. How dumb is the log ingest path?
abledon 4y ago
logged into https://demo.parseable.io/index.html https://demo.parseable.io/index.html with parseable/parseable as username/password... no data is showing for frontend/k6test feed
- devupio 4y ago
  Apologies, can you try the `frontend` stream and change the duration to last 7 days or so.
  - abledon 4y ago
    results! thanks
e12e 4y ago
How does it compare with loki/qryn("née cloki/clickhouse loki)? https://grafana.com/oss/loki/ https://grafana.com/oss/loki/ https://github.com/metrico/qryn https://github.com/metrico/qryn Aside - with so much to choose from wrt logs and metrics - I'd assume "prometheus for logs" implied "pulling" log data (rather than push). While maybe "Victoria metrics for logs" might imply "easier to operate, efficient, fewer moving parts".
- qxip 4y ago
  at qryn we're past the logs-only phase and currently we're concentrating on the polyglot aspects - we already support ingestion and querying of logs, metrics and telemetry using various inputs (loki, influx, elastic, etc) and supporting logql, promql and tempo for searching. Less moving parts, more power.
- mdaniel 4y ago
  I hadn't heard of qryn before, but it seems that the AGPL has become the go-to license of choice for these kind of tools. It makes me sad, but for clarity: it just makes me sad, the authors are entitled to use whatever license they want
  - atomicnumber3 4y ago
    As someone with an AGPL sticker on my laptop - why does that make you sad?
    - ensignavenger 4y ago
      I always find it confusing what I can and can't do with the entire GPL family of licenses, specifically when something is considered a "derivative" work or not. The AGPL network provision makes this an even bigger issue. It also introduces a whole new layer of uncertainty. I also have seen a trend over the years of companies who adopt AGPL not caring about the open source community around the code. This is not specific to the AGPL, and not every user of it does so, but I see it correlating more often. Many companies refuse to use AGPL licensed code... some claim their lawyers are stupid and don't know what they are doing... but it can still prevent adoption of AGPL licensed products.

wmanley 4y ago

Are you using S3 Select[1]? [1]: https://aws.amazon.com/blogs/aws/s3-glacier-select/ https://aws.amazon.com/blogs/aws/s3-glacier-select/

wmanley 4y ago

I really like the idea of storing logs in parquet format in S3. It's something I've wanted to play with for a long time, but never have had a use-case. I'm also a big fan of Rust. I imagine you've seen apenwarr's blog post on logging: https://apenwarr.ca/log/20190216 https://apenwarr.ca/log/20190216 With that in mind I'd like to read more about the architecture. Specifically: 1. Do you have separate components for ingest and querying that can be scaled independently? 2. How dumb is the log ingest path?

abledon 4y ago

logged into https://demo.parseable.io/index.html https://demo.parseable.io/index.html with parseable/parseable as username/password... no data is showing for frontend/k6test feed

devupio 4y ago
Apologies, can you try the `frontend` stream and change the duration to last 7 days or so.
- abledon 4y ago
  results! thanks

e12e 4y ago

How does it compare with loki/qryn("née cloki/clickhouse loki)? https://grafana.com/oss/loki/ https://grafana.com/oss/loki/ https://github.com/metrico/qryn https://github.com/metrico/qryn Aside - with so much to choose from wrt logs and metrics - I'd assume "prometheus for logs" implied "pulling" log data (rather than push). While maybe "Victoria metrics for logs" might imply "easier to operate, efficient, fewer moving parts".

qxip 4y ago
at qryn we're past the logs-only phase and currently we're concentrating on the polyglot aspects - we already support ingestion and querying of logs, metrics and telemetry using various inputs (loki, influx, elastic, etc) and supporting logql, promql and tempo for searching. Less moving parts, more power.
mdaniel 4y ago
I hadn't heard of qryn before, but it seems that the AGPL has become the go-to license of choice for these kind of tools. It makes me sad, but for clarity: it just makes me sad, the authors are entitled to use whatever license they want
- atomicnumber3 4y ago
  As someone with an AGPL sticker on my laptop - why does that make you sad?
  - ensignavenger 4y ago
    I always find it confusing what I can and can't do with the entire GPL family of licenses, specifically when something is considered a "derivative" work or not. The AGPL network provision makes this an even bigger issue. It also introduces a whole new layer of uncertainty. I also have seen a trend over the years of companies who adopt AGPL not caring about the open source community around the code. This is not specific to the AGPL, and not every user of it does so, but I see it correlating more often. Many companies refuse to use AGPL licensed code... some claim their lawyers are stupid and don't know what they are doing... but it can still prevent adoption of AGPL licensed products.