6 ms·

Kafka and Druid are expensive and complicated components to run for an open core biz trying to help people save money against Datadog. This guarantees a lot of

Kafka and Druid are expensive and complicated components to run for an open core biz trying to help people save money against Datadog. This guarantees a lot of people won't take advantage of the "open" part of your open core but maybe that doesn't matter for your business anyway. I can't speak to Druid but I'm always puzzled when I hear about Kafka being used for metrics. Most metrics are timestamped and also support being calculated in ways that support out of order handling. It's true that people are being gouged on storage markup by monitoring companies but I don't think this particular approach is the solution. Obsessing over storage and querying costs isn't a good starting point for a startup so maybe driving good habits (stop collecting so much junk, keep it around less frequently, etc.) is a better route to help people save quiche. Either way good luck!

ankitnayan 6y ago
Nice thoughts, A few other users also pointed this out. We observed enterprise and other Observability SaaS vendors have some scripts and controllers to keep running these components. We plan to open-source that too. As you rightly pointed out running OSS needs man hours and we will try to remove those frictions. Also when working with Prometheus and Jaeger, we observed people anyhow have to use Kafka to handle scale and mostly OSS are good at start but become pretty complicated at handling scale. Eg, Prometheus long term storage solution is Cortex which itself is difficult to manage. In that case, Kafka should be better beast to handle than multiple moving components inside Cortex. We built SigNoz as a scalable alternative inspired from stream processing architecture. We will also be proving sampling strategies including tail-based sampling to retain important data and not unnecessarily clogging disks.
TameAntelope 6y ago
Kafka has a terrible reputation, but once you get familiar with it, it only occasionally lives up to that reputation, and oftentimes outperforms expectations by quite a bit.
- ankitnayan 6y ago
  I was pretty much surprised to see the results too. A single node Kafka with 2GB as xmx value, was ingesting at 4500 events/sec (around 1MB/s) on a single partition. I blogged my experiments with SigNoz's scale at https://signoz.io/blog/signoz-benchmarks/ https://signoz.io/blog/signoz-benchmarks/. Hoping to get better in fine-tuning configs and blogging.
  - gen220 6y ago
    I think the concerns raised in this thread are less regarding raw throughput, and more about (1) the complexity of the typical production Kafka deployment (2) the arguably unnecessary, highly complex ecosystem around Kafka that you have to pay people or companies to use effectively, (3) the history of problems regarding data loss with ZK/Kafka, caused by leadership election bugs.
    - ankitnayan 6y ago
      hmm..I get your point. I searched for Kafka alternatives for a bit before including it on our stack. Though, I couldn't find something more adopted by all. It would be good to know a few Kafka alternatives you prefer which can handle equivalent production scale?
      - dflock 6y ago
        Have an interface for a queuing system and support other things, not just Kafka. Ideally, you want a default/dev instance to ship with something super simple, zero setup and maybe in-memory - but allowing you to swap-in kafka or something more capable as needed.
        pranay01 6y ago
        That's an interesting point. Curious, would you use a project which supports a simple/in-memory datastore, but not anything which would be useful in production environment? Do you think that easy to get running and setup in dev environment valuable for adoption - even if it won't work in prod? I am trying to understand - what would be a good way to prioritise.
- stmw 6y ago
  Guess your were luckier? YMMV, I've found Kafka generally lives up to its terrible reputation -- and even when it doesn't, it's all somehow more difficult than its initial appeal. I certainly agree with others that inclusion of Kafka in an open-source package like this would discourage me from using it.
  - pranay01 6y ago
    Curious, is there an alternative to Kafka which would be more easier for you to adopt?
    - vladsanchez 6y ago
      - [RedPanda|https://vectorized.io/redpanda/ https://vectorized.io/redpanda/]? - [Apache Pulsar|https://pulsar.apache.org https://pulsar.apache.org]?
      - pranay01 6y ago
        We had checked out both these projects. Our view was that RedPanda was still an early project ( ~1.5K stars) and Pulsar was very similar to Kafka, and Kafka was more well known compared to Pulsar
    - stmw 6y ago
      some options: - https://pulsar.apache.org/ https://pulsar.apache.org/ - the many systems based on https://en.wikipedia.org/wiki/Advanced_Message_Queuing_Protocol https://en.wikipedia.org/wiki/Advanced_Message_Queuing_Proto... - cloud-specific queues (SNS Kinesis et al)
      - TameAntelope 6y ago
        Kafka does not fill the same use cases as these suggestions, this may be part of the trouble you were experiencing! My use of Kafka was as a "system of record", and attaching connectors to create views into the data from there. I could replay a Kafka topic into a MongoDB, run some analysis, and destroy the MongoDB instance.
gen220 6y ago
I’d totally agree that the operating costs (speaking eng time, which is more expensive than machines) of Kafka+ZK alone is quite high. If a company is not already using Kafka, they wouldn’t want to maintain it “just” to have a self hosted APM system. If I could make a recommendation to the developers of this system, it would be to focus on the interface with the streaming platform, before the implementation of using Kafka to support that interface. Ideally, one should be able to plug in and out the queueing system of preference. This will help adoption, and avoid coupling the success of your project to the implementation and success of Kafka.
- ankitnayan 6y ago
  I completely agree with you. For companies not already using Kafka, this will ask for a big commitment to self-host Kafka. You mentioned a great approach. Queueing system as a plugin. Thanks
- jwatte 6y ago
  Even if you maintain Kafka for business logic, you don't want to run your observability in the same Kafka cluster, because then when the business Kafka goes down, how will you debug it?
  - ankitnayan 6y ago
    Correct, Ideally monitoring stack should be outside the blast radius of other applications. Will handling another Kafka cluster (probably smaller than business Kafka) be a pain for the team given the team already knows managing one business Kafka. What do you think?

ankitnayan 6y ago

Nice thoughts, A few other users also pointed this out. We observed enterprise and other Observability SaaS vendors have some scripts and controllers to keep running these components. We plan to open-source that too. As you rightly pointed out running OSS needs man hours and we will try to remove those frictions. Also when working with Prometheus and Jaeger, we observed people anyhow have to use Kafka to handle scale and mostly OSS are good at start but become pretty complicated at handling scale. Eg, Prometheus long term storage solution is Cortex which itself is difficult to manage. In that case, Kafka should be better beast to handle than multiple moving components inside Cortex. We built SigNoz as a scalable alternative inspired from stream processing architecture. We will also be proving sampling strategies including tail-based sampling to retain important data and not unnecessarily clogging disks.

TameAntelope 6y ago

Kafka has a terrible reputation, but once you get familiar with it, it only occasionally lives up to that reputation, and oftentimes outperforms expectations by quite a bit.

ankitnayan 6y ago
I was pretty much surprised to see the results too. A single node Kafka with 2GB as xmx value, was ingesting at 4500 events/sec (around 1MB/s) on a single partition. I blogged my experiments with SigNoz's scale at https://signoz.io/blog/signoz-benchmarks/ https://signoz.io/blog/signoz-benchmarks/. Hoping to get better in fine-tuning configs and blogging.
- gen220 6y ago
  I think the concerns raised in this thread are less regarding raw throughput, and more about (1) the complexity of the typical production Kafka deployment (2) the arguably unnecessary, highly complex ecosystem around Kafka that you have to pay people or companies to use effectively, (3) the history of problems regarding data loss with ZK/Kafka, caused by leadership election bugs.
  - ankitnayan 6y ago
    hmm..I get your point. I searched for Kafka alternatives for a bit before including it on our stack. Though, I couldn't find something more adopted by all. It would be good to know a few Kafka alternatives you prefer which can handle equivalent production scale?
    - dflock 6y ago
      Have an interface for a queuing system and support other things, not just Kafka. Ideally, you want a default/dev instance to ship with something super simple, zero setup and maybe in-memory - but allowing you to swap-in kafka or something more capable as needed.
      - pranay01 6y ago
        That's an interesting point. Curious, would you use a project which supports a simple/in-memory datastore, but not anything which would be useful in production environment? Do you think that easy to get running and setup in dev environment valuable for adoption - even if it won't work in prod? I am trying to understand - what would be a good way to prioritise.
stmw 6y ago
Guess your were luckier? YMMV, I've found Kafka generally lives up to its terrible reputation -- and even when it doesn't, it's all somehow more difficult than its initial appeal. I certainly agree with others that inclusion of Kafka in an open-source package like this would discourage me from using it.
- pranay01 6y ago
  Curious, is there an alternative to Kafka which would be more easier for you to adopt?
  - vladsanchez 6y ago
    - [RedPanda|https://vectorized.io/redpanda/ https://vectorized.io/redpanda/]? - [Apache Pulsar|https://pulsar.apache.org https://pulsar.apache.org]?
    - pranay01 6y ago
      We had checked out both these projects. Our view was that RedPanda was still an early project ( ~1.5K stars) and Pulsar was very similar to Kafka, and Kafka was more well known compared to Pulsar
  - stmw 6y ago
    some options: - https://pulsar.apache.org/ https://pulsar.apache.org/ - the many systems based on https://en.wikipedia.org/wiki/Advanced_Message_Queuing_Protocol https://en.wikipedia.org/wiki/Advanced_Message_Queuing_Proto... - cloud-specific queues (SNS Kinesis et al)
    - TameAntelope 6y ago
      Kafka does not fill the same use cases as these suggestions, this may be part of the trouble you were experiencing! My use of Kafka was as a "system of record", and attaching connectors to create views into the data from there. I could replay a Kafka topic into a MongoDB, run some analysis, and destroy the MongoDB instance.

gen220 6y ago

I’d totally agree that the operating costs (speaking eng time, which is more expensive than machines) of Kafka+ZK alone is quite high. If a company is not already using Kafka, they wouldn’t want to maintain it “just” to have a self hosted APM system. If I could make a recommendation to the developers of this system, it would be to focus on the interface with the streaming platform, before the implementation of using Kafka to support that interface. Ideally, one should be able to plug in and out the queueing system of preference. This will help adoption, and avoid coupling the success of your project to the implementation and success of Kafka.

ankitnayan 6y ago
I completely agree with you. For companies not already using Kafka, this will ask for a big commitment to self-host Kafka. You mentioned a great approach. Queueing system as a plugin. Thanks
jwatte 6y ago
Even if you maintain Kafka for business logic, you don't want to run your observability in the same Kafka cluster, because then when the business Kafka goes down, how will you debug it?
- ankitnayan 6y ago
  Correct, Ideally monitoring stack should be outside the blast radius of other applications. Will handling another Kafka cluster (probably smaller than business Kafka) be a pain for the team given the team already knows managing one business Kafka. What do you think?