8 ms·
Think Prometheus, but for logs (not metrics). Simple, efficient, fast log store
- white_dragon88 4y agoWhy should I use this over something like cloudwatch?
- mdaniel 4y agoHeh, with a question like that: what else have you tried that leads you to be happy with cloudwatch?
- lopatin 4y agoHow do you search over logs if it's an index-free design?
- sch00lb0y 4y agoLike the idea of using parquet and datafusion for logs. Pretty promising
- fmajid 4y agoThere's a lot of open-source activity in log management lately. Apart from Loki, ther is Signoz and Uptrace.
- e12e 4y agoInteresting - looks like both of these are openTelementary + clickhouse for data. Qryn (loki on clickhouse) seems similar - but with different query and dashboard. Anyone evaluated these recently, or have experience using any of them in anger?
- qxip 4y agoThanks for mentioning our project! qryn (formerly cloki) is currently more focused on the polyglot factor and trying to unify logs, metrics and telemetry on a single stateless platform, easy to scale without hundreds of services and moving parts. At this stage, its a lightweight Grafana Cloud alternative just requiring clickhouse - no sidecar databases, redis, or plugins needed, and no new query languages or rules to learn. Latest info is at https://qryn.dev https://qryn.dev
- silasb 4y agoI noticed the UI just filters the data queried. Is this a WIP or will the search feature actually update the search query?
- didgetmaster 4y agoIs there currently a big market for new logging solutions? It has been years since I worked much with logs, so I don't know much about the current frustrations of those who work with them on a daily basis. There will always be potential for a 'better mousetrap' to take some market share from existing solutions, but are current ones bad enough that a great solution could come in an capture significant market share? Or are today's systems 'good enough' that any new solution would face huge headwinds, even if it has some cool features? The reason I ask is because I am building a new kind of general-purpose data management system (https://www.Didgets.com https://www.Didgets.com) that can handle all kinds of data. It does file data, relational tables, Key-Value stores and other kinds of NoSql data. I did a little proto-typing of it managing some simple logging data and it seemed to work really well (in my opinion). But I would need to do a bit more work to get a good demo going. I have limited resources, so I would only want to do that if there is a significant demand in the logging space for something new. What is the biggest 'pain point' for people working with log data today?
- tofuahdude 4y agoFor these kinds of established markets, you basically have to be 5x+ better or 5x+ cheaper. Then it's fine if the product is effectively the same.
- didgetmaster 4y ago5x+ better but effectively the same?? What does that mean?
- baq 4y agosay you build splunk but instead of $500k you only charge $100k for the same featureset. not sure how to make a 5x better splunk...
- didgetmaster 4y agoI know what 5x cheaper means but that was not what I was asking.
- jedberg 4y agoI've been doing ops for a long time, and I can't remember the last time I looked at an historical log (edit: I'm referring to application logs specifically stored in a central location). When something is broken, I turn on logging and look at the logs in real-time. If something was broken in the past and no longer is, I turn on logging for that thing so that I have logs if it happens again. The only use I can think of for historical logs is security auditing. In which case there is only a subset of logs you really need to trace a breach and they don't need to be searchable online -- you can pull them back from cold storage if you need to check for an incident. Moving and storing logs and making them all instantly searchable is awfully expensive for little benefit. Edit2: Disclaimer: This opinion does not apply to government regulated services like financial services.
- encoderer 4y agoCounter-point is that when a customer reports a problem I don’t want to have to say “can you try that again after I enable logging”.
- jedberg 4y agoWhen a customer reports a problem the first thing I try is to reproduce it (with logging on). If I can't reproduce it, does it matter? I can leave logging on and let them know I can't reproduce their problem but if it happens again to let me know.
- encoderer 4y agoIn your background of consumer engineering, yes it probably does not matter. But it’s not the same in SaaS
- jedberg 4y agoI've worked in SaaS for years, and I don't have any examples in my history where that's the case. Do you have an example of a case where historical logging was the only way to track down a problem?
- Volundr 4y ago> By default Parseable uses a public S3 compatible store (MinIO) to store the data. Please change this to your S3 bucket before storing any important data. Wait, am I reading that right? Does this mean by default the docker container is going to store my data in a random place on the internet? If so that's a crazy bad default.
- belthesar 4y agoWhoa. At first read, I assumed it was spinning MinIO alongside the Parseable daemon, but no, it's definitely sending data to a MinIO instance _they host_. https://github.com/parseablehq/parseable/blob/a7abb5dfe1ca573fbe4726228cc84d16e86ba28b/server/src/s3.rs#L37 https://github.com/parseablehq/parseable/blob/a7abb5dfe1ca57... No bueno.
- trueleo 4y ago> Please do not store any sensitive data on this server as the data is openly accessible. We'll delete the data on this server periodically. It is there for demo purpose ig It also gives you this warning when you run the build Warning: Parseable server is using default credentials. Setup your credentials with P_USERNAME and P_PASSWORD before storing production logs. Parseable server is using default object storage backend with public access. Setup your object storage backend with P_S3_URL before storing production logs.
- thayne 4y agoHaving a public place to send demo logs is fine. Sending logs there by default, rather than having to opt in to demo mode, is bad. The default should be to fail to start unless credentials for the data store are provided. Or alternatively store to the local filesystem by default.
- Volundr 4y agoThe problem is this means I'm one misconfiguration away from sending all my data to a stranger's server on the internet. If I misconfigure an Ansible playbook or kubernetes deploy I'm not going to see that warning. Frankly even if a sysadmin is configuring it by hand we shouldn't be counting on them noticing the warning for our security. I believe they have good intentions and are just trying to make it easy to try out, but this is just too big a footgun hanging out. Makes it untouchable for me.
- ngrilly 4y agoIf I understand correctly, Parseable is relying heavily on DataFusion for its SQL implementation: 1/ I know that Parquet and Arrow are very good for time series (mostly metric names, labels, timestamps and real values), but are they good for strings, which are quite important in logs? 2/ Is DataFusion able to execute an SQL request over a large number of large Parquet files without loading everything in memory at the same time, or can it process a Parquet as it reads it, in a streaming fashion?
- bityard 4y agoI like that it's (apparently) open source. Are you forming a company around this and do you plan to monetize it? I would find that information helpful to know whether I should consider it as a candidate for a tool in my day job, or disqualify it for my hobby projects.
- GordonS 4y agoWhat's the high availability story like? (I realise it might not be implemented yet, but presumably you have an idea of how you want to do it)
- devupio 4y agoIntroducing Parseable, simple, efficient, cloud native log storage and query system. As Developers, SRE, DevOps look to replace Elastic from their logging stack - we believe the log storage of future won't be another search and indexing engine in a new language. Parseable leverages Apache Arrow, Parquet and widely available object storage platforms for efficiency, cost effectiveness and performance.
- wmanley 4y agoAre you using S3 Select[1]? [1]: https://aws.amazon.com/blogs/aws/s3-glacier-select/ https://aws.amazon.com/blogs/aws/s3-glacier-select/
- wmanley 4y agoI really like the idea of storing logs in parquet format in S3. It's something I've wanted to play with for a long time, but never have had a use-case. I'm also a big fan of Rust. I imagine you've seen apenwarr's blog post on logging: https://apenwarr.ca/log/20190216 https://apenwarr.ca/log/20190216 With that in mind I'd like to read more about the architecture. Specifically: 1. Do you have separate components for ingest and querying that can be scaled independently? 2. How dumb is the log ingest path?
- abledon 4y agologged into https://demo.parseable.io/index.html https://demo.parseable.io/index.html with parseable/parseable as username/password... no data is showing for frontend/k6test feed
- 4y ago
- comice 4y agoCode here https://github.com/parseablehq/parseable https://github.com/parseablehq/parseable Written in Rust. GNU Affero General Public License v3.0
- _boffin_ 4y agoYour tagline seems strikingly similar to Loki’s tagline. What benefits does this provide that Loki doesn’t?
- senko 4y agoClickable link to Loki on GitHub: https://github.com/grafana/loki https://github.com/grafana/loki
- candiddevmike 4y agoI can't find much, this requires S3 so you can't use it with local disk. Doesn't seem simpler than Loki in that regard, just less features.
- baq 4y ago'requires S3' (had to read your comment) and 'simple' (headline) shouldn't be in the same sentence :/
- Hamuko 4y agoBut the "S" in "S3" stands for "Simple".
- seimour 4y agoWhat does the "3" stand for?
- dljsjr 4y agoSimple Storage Service (3 S's).
- deathanatos 4y agoIt means there are 3 S's. "Simple Storage Service", or "S3". (See https://aws.amazon.com/s3/ https://aws.amazon.com/s3/ — "Amazon Simple Storage Service (Amazon S3) is […]")
- zhala 4y agoHow does this differ from something like Loki + Thanos/Mimir?
- moffkalast 4y agoA guy that is stuck in a liver donation loop for stealing fire, but he stole wood instead? If you have to describe your project as "some other project but like this" you're already doing it wrong. Do you think Bing markets its maps by saying "Think Google Maps But Microsoft"?
- dpkirchner 4y agoWell, no, they don't have to because Bing Is Not Google.
- jayd16 4y agoYou're conflating comparing yourself to a competitor with a run of the mill simile to a similar architecture.
- kraquepype 4y agoThat headline worked fine for me... I like prometheus for metrics but wish I had a "prometheus" for logs. "Think Google Maps but for the grocery store (not roads)" would be more apt.
- imalerba 4y agoThat's Loki > Loki is a horizontally scalable, highly available, multi-tenant log aggregation system inspired by Prometheus. https://grafana.com/oss/loki/ https://grafana.com/oss/loki/
- bork1 4y agoI was really excited to see something like this. I really dislike my company's current logging solution (I mostly take issue with how slow querying logs is most of the time). Unfortunately the live demo didn't work. Maybe it was hugged to death? In any case it might be nice to get the demo more stable!