4 ms·

What's a good distributed log for 10-dev sized companies? :)

by Serow225 8y ago

jacobr1 8y ago
AWS Kinesis + s3
- Serow225 8y ago
  Thanks! I'm guessing you're referring to Kinesis Streams? Is there an OOB solution to persist the records past the default 168hours, or is this something that you have to build out yourself following some pattern?
  - mhotchen 8y ago
    You can configure it to output to S3 and the mechanism for that is easy to configure and has fault tolerance.
simonrobb 8y ago
OKLog, Humio, and Splunk are all worth checking out.
- atombender 8y ago
  OKLog has been abandoned by the author (the project is now read only on GitHub). Humio is not self-hosted or open source, so not really a fair comparison. It also seems targeted towards operational logs, i.e. system logging, traffic logging, auditing. Not things like data pipelines. Kafka and friends can be used for that kind of log, but they are more like databases; they use the term "log" in the sense of sequential and append-only. Same goes for Splunk, which does have a self-hosted version, but is extremely expensive, last I checked. The SaaS version is also extremely expensive.
  - geeters1 8y ago
    Humio does have a self-hosted version - but is closed source. You can download a trial and it supports most logs types via support for opensource log shippers, ie logstash and beats..along with other popular formats including Kafka. https://docs.humio.com/integrations/ https://docs.humio.com/integrations/ The UI is what simplifies analysis and visualization with live, real-time query and db.
atombender 8y ago
If you're willing to go cloud, then Google's BigQuery is a very good fit for sequential data. It's fully queryable, unlike most log-oriented databases, and it's extremely cheap compared to competing offerings (e.g. AWS RedShift).
manigandham 8y ago
Apache Bookkeeper. Or install Apache Pulsar and get full messaging capabilities with tiering out to S3 for infinite retention.
windexh8er 8y ago
Gravwell.

What's a good distributed log for 10-dev sized companies? :)

jacobr1 8y ago

AWS Kinesis + s3

Serow225 8y ago
Thanks! I'm guessing you're referring to Kinesis Streams? Is there an OOB solution to persist the records past the default 168hours, or is this something that you have to build out yourself following some pattern?
- mhotchen 8y ago
  You can configure it to output to S3 and the mechanism for that is easy to configure and has fault tolerance.

simonrobb 8y ago

OKLog, Humio, and Splunk are all worth checking out.

atombender 8y ago
OKLog has been abandoned by the author (the project is now read only on GitHub). Humio is not self-hosted or open source, so not really a fair comparison. It also seems targeted towards operational logs, i.e. system logging, traffic logging, auditing. Not things like data pipelines. Kafka and friends can be used for that kind of log, but they are more like databases; they use the term "log" in the sense of sequential and append-only. Same goes for Splunk, which does have a self-hosted version, but is extremely expensive, last I checked. The SaaS version is also extremely expensive.
- geeters1 8y ago
  Humio does have a self-hosted version - but is closed source. You can download a trial and it supports most logs types via support for opensource log shippers, ie logstash and beats..along with other popular formats including Kafka. https://docs.humio.com/integrations/ https://docs.humio.com/integrations/ The UI is what simplifies analysis and visualization with live, real-time query and db.

atombender 8y ago

If you're willing to go cloud, then Google's BigQuery is a very good fit for sequential data. It's fully queryable, unlike most log-oriented databases, and it's extremely cheap compared to competing offerings (e.g. AWS RedShift).

manigandham 8y ago

Apache Bookkeeper. Or install Apache Pulsar and get full messaging capabilities with tiering out to S3 for infinite retention.

windexh8er 8y ago

Gravwell.