3 ms·

Looking at the docker compose, this seems like a very complicated tool to run. You'd need Kafka, zookeeper, and Jaeger. All would need to be HA. Then also this

Looking at the docker compose, this seems like a very complicated tool to run. You'd need Kafka, zookeeper, and Jaeger. All would need to be HA. Then also this service. Not mentioning postgres because in theory you can use aurora or the like. How quick have your current customers been able to get up and running so far? And how much maintainence have they needed?

PaoloBarbolini 3y ago
According to the documentation[1] Kafka is just one of the supported inputs for ingestion, so it should be possible to run Quickwit without it if you're not intending to write logs into Kafka. Jaeger also seems like another optional dependency. Same also probably for Zookeeper? 1: https://quickwit.io/docs/ingest-data/kafka https://quickwit.io/docs/ingest-data/kafka
- fulmicoton 3y ago
  Zookeeper is not used by Quickwit at all. It is used by Kafka however.
  - mdaniel 3y ago
    I continue to see that dependency almost everywhere I see Kafka used, and yet KIP-500 <https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A...> and its Jira <https://issues.apache.org/jira/browse/KAFKA-9119 https://issues.apache.org/jira/browse/KAFKA-9119> are shown as "Resolved Apr-2021" allegedly in 2.8.0 I wonder if it's going to be the "python2.7" (or I guess more relevant "Java8") of running Kafka :-(
fulmicoton 3y ago
Most users do not use Kafka/Zookeeper. The only external service for them is a S3 bucket. They then use the PushAPI. It is perfectly fine if you have only a couple of TB a day. For the crazy large use cases, like the one described in the blog post, Kafka becomes necessary. At that scale, our users usually already have their data in Kafka or RedPanda and are actually happy to be able to get native integration: - their data does not need to be copied/replicated in a WAL "again" - we get exactly-once semantics Also, in 0.8, we will be adding proper support for distributed ingest. The feature is actually already implemented and was originally scheduled for 0.7. but we preferred to test it more before actually shipping it.
francoismassot 3y ago
Quickwit supports different data sources: Kafka, Pulsar, Google pubsub, ... and we have our own ingest API (not HA right now, but it will be the case in the next release in 1 month or so). Postgresql is not mandatory; it's also possible to use Quickwit with a metastore on S3. For large use cases, Postgresql is the way to go. I've seen users using Quickwit with metastore on S3, RDS, and Aurora. On the UI side, we have several users who have their own UI. Jaeger is used just for the UI part so it's quite simple to have it in HA, I don't thing it's hard to have HA for Grafana but I'm not sure on this point. Which docker compose did you look at?

PaoloBarbolini 3y ago

According to the documentation[1] Kafka is just one of the supported inputs for ingestion, so it should be possible to run Quickwit without it if you're not intending to write logs into Kafka. Jaeger also seems like another optional dependency. Same also probably for Zookeeper? 1: https://quickwit.io/docs/ingest-data/kafka https://quickwit.io/docs/ingest-data/kafka

fulmicoton 3y ago
Zookeeper is not used by Quickwit at all. It is used by Kafka however.
- mdaniel 3y ago
  I continue to see that dependency almost everywhere I see Kafka used, and yet KIP-500 <https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A...> and its Jira <https://issues.apache.org/jira/browse/KAFKA-9119 https://issues.apache.org/jira/browse/KAFKA-9119> are shown as "Resolved Apr-2021" allegedly in 2.8.0 I wonder if it's going to be the "python2.7" (or I guess more relevant "Java8") of running Kafka :-(

fulmicoton 3y ago

Most users do not use Kafka/Zookeeper. The only external service for them is a S3 bucket. They then use the PushAPI. It is perfectly fine if you have only a couple of TB a day. For the crazy large use cases, like the one described in the blog post, Kafka becomes necessary. At that scale, our users usually already have their data in Kafka or RedPanda and are actually happy to be able to get native integration: - their data does not need to be copied/replicated in a WAL "again" - we get exactly-once semantics Also, in 0.8, we will be adding proper support for distributed ingest. The feature is actually already implemented and was originally scheduled for 0.7. but we preferred to test it more before actually shipping it.

francoismassot 3y ago

Quickwit supports different data sources: Kafka, Pulsar, Google pubsub, ... and we have our own ingest API (not HA right now, but it will be the case in the next release in 1 month or so). Postgresql is not mandatory; it's also possible to use Quickwit with a metastore on S3. For large use cases, Postgresql is the way to go. I've seen users using Quickwit with metastore on S3, RDS, and Aurora. On the UI side, we have several users who have their own UI. Jaeger is used just for the UI part so it's quite simple to have it in HA, I don't thing it's hard to have HA for Grafana but I'm not sure on this point. Which docker compose did you look at?