6 ms·

Wow, there is a lot here and what here is to a pretty impressive level of polish for how far along this is. The background of someone with a DX background come

Wow, there is a lot here and what here is to a pretty impressive level of polish for how far along this is. The background of someone with a DX background comes through! I will be looking into this a lot more. Here are a few comments, notes, and questions: * I like the focus on DX (especially compared to other OSS solutions) in your messaging here, and I think your hero messaging tells that story, but it isn't reinforced as much through the features/benefits section * It seems like clickhouse is obviously a big piece of the tech here, which is an obvious choice, but from my experience with high data rate ingest, especially logs, you can run into issues at larger scale. Is that something you expect to give options around in open source? Or is the cloud backend a bit different where you can offer that scale without making open source so complex? * I saw what is in OSS vs cloud and I think it is a reasonable way to segment, especially multi-tenancy, but do you see the split always being more management/security features? Or are you considering functional things? Especially with recent HashiCorp "fun" I think more and more it is useful to be open about what you think the split will be. Obviously that will evolve, but I think that sort of transparency is useful if you really want to grow the OSS side * on OSS, I was surprised to see MIT license. This is full featured enough and stand alone enough that AGPL (for server components) seems like a good middle ground. This also gives some options for potentially a license for an "enterprise" edition, as I am certain there is a market for a modern APM that can run all in a customer environment * On that note, I am curious what your target persona and GTM plan is looking like? This space is a a bit tricky IMHO, because small teams have so many options at okay price points, but the enterprise is such a difficult beast in switching costs. This looks pretty PLG focused atm, and I think for a first release it is impressive, but I am curious to know if you have more you are thinking to differentiate yourself in a pretty crowded space. Once again, really impressive what you have here and I will be checking it out more. If you have any more questions, happy to answer in thread or my email is in profile.

mikeshi42 3y ago
Thank you, really appreciate the feedback and encouragement! > It seems like clickhouse is obviously a big piece of the tech here, which is an obvious choice, but from my experience with high data rate ingest, especially logs, you can run into issues at larger scale. Is that something you expect to give options around in open source? Scaling any system can be challenging - our experience so far is that Clickhouse is a fraction of the overhead of systems like Elasticsearch has previously demanded luckily. That being said, I think there's always going to be a combination of learnings we'd love to open source for operators that are self-hosting/managing Clickhouse, and tooling we use internally that is purpose-built for our specific setup and workloads. > I saw what is in OSS vs cloud and I think it is a reasonable way to segment, especially multi-tenancy, but do you see the split always being more management/security features? Our current release - we've open sourced the vast majority of our feature set, including I think some novel features like event patterns that typically are SaaS-only and that'll definitely be the way we want to continue to operate. Given the nature of observability - we feel comfortable continuing to keep pushing a fully-featured OSS version while having a monetizable SaaS that focuses on the fact that it's completely managed, rather than needing to gate heavily based on features. > on OSS, I was surprised to see MIT license We want to make observability accessible and we think AGPL will accomplish the opposite of that. While we need to make money at the end of the day - we believe that a well-positioned enterprise + cloud offering is better suited to pull in those that are willing to pay, rather than forcing it via a license. I also love the MIT license and use it whenever I can :) > On that note, I am curious what your target persona and GTM plan is looking like? I think for small teams, imo the options available are largely untantilizing, it ranges from narrow tools like Cloudwatch to enterprise-oriented tools like New Relic or Datadog. We're working hard to make it easier for those kinds of teams to adopt good monitoring and observability from day 1, without the traditional requirement of needing an observability expert or dedicated SRE to get it set up. (Admittedly, we still have a ways to improve today!) On the enterprise side, switching costs are definitely high, but most enterprises are highly decentralized in decision making, where I routinely hear F500s having a handful of observability tools in production at a given time! I'll say it's not as locked-in as it seems :)
- addisonj 3y ago
  Thanks for the answers Mike! One more follow-up on the scale side (which I mentioned with sibling comment), it isn't so much about clickhouse itself, but about scaling up ingest. From my own experience and from talking with quite a few APM players (I previously worked in streaming space), a Kafka / durable log storage kind of becomes a requirement, so I was curious if you think at some point you need a log to further scale ingest. For enterprise side, I was previously in data streaming space and had quite a few conversations with APM players and companies building their own observability platforms, happy to chat and share more if that would be useful!
  - mikeshi42 3y ago
    Ah got it, yeah a queue of some sort is definitely useful when scaling up to buffer pre-inserted data. This is something on the OSS side we've kept open to implementation. However it's something that is highly coupled with infra footprint and internal SLA guarantees the user wants to preserve. It can range anywhere from just rely on client-side retries to setting up a HA Kafka cluster early in the ingestion pipeline. Similar to Elastic - I think a lot of architectures are available to choose on that side when users want to scale. Will reach out to connect!
- debarshri 3y ago
  One piece of advice here is, if you pitch yourself as a datadog competitor, then I would recommend replicating some of the GTM motions that datadog employed. For instance, you have an opportunity to go very upmarket, super enterprise orgs. You can do PLG, but ultimately every tool becomes SLG. I would recommend fine tuning that motion as that would be the one bringing larger contract 6 digit contracts and huge growth here. I have seen orgs remove datadog because of unpredictable pricing. If you do flat price self hosted platform, you will get attention. I dont think orgs would mind hosting clickhouse. You can also bundle it with your helm charts or initial proof of concept might have lower barrier. I know some orgs have million dollar annual contracts with datadog, a cheaper more predictable priced alternative will definitely get attention.
  - 3y ago
dangoodmanUT 3y ago
For clickhouse, just batch insert. They probably have something batching every few s before inserting directly to their hosted version
- dangoodmanUT 3y ago
  There's also async inserts
- addisonj 3y ago
  Generally, any sort of async/batch inserts will get you decently far, but still will have limitations well before you get to million rows a second, mostly because it is really difficult to get your batch size large enough from individual producers without some sort of aggregation, which that aggregation is a challenge if you care about durability. So often that means you need something like a Kafka to get the bulk ingest to really perform to get batch sizes large enough. That kind of gets into one of the challenges of OSS observabilility systems, you don't want to make the dependencies insane for someone who only has a few thousand logs a second, but generally at some point of scale you do need more.
- vadman97 3y ago
  ClickHouse Async insert docs [1]. We ran into some challenges with async inserts at highlight.io [2]. Namely, ClickHouse Cloud has an async flush size configured (that can't be changed AFAIK) that isn't large enough for our scale. Once you async insert more than can be flushed, you get back pressure on your application waiting to write while Clickhouse flushes the queue. We found that implementing our own batched flushing via kafka [3] is far more performant, allowing us to insert 500k+ RPS on the smallest cloud instance type. [1] https://clickhouse.com/docs/en/optimize/asynchronous-inserts https://clickhouse.com/docs/en/optimize/asynchronous-inserts [2] https://github.com/highlight/highlight/tree/main https://github.com/highlight/highlight/tree/main [3] https://github.com/highlight/highlight/blob/4d28451b1935796d2f400a7589e430b2fa2e0229/backend/worker/kafka_worker.go#L104-L174 https://github.com/highlight/highlight/blob/4d28451b1935796d...
fnord77 3y ago
Clickhouse is proprietary, though. I wonder why not Apache Druid
- zx8080 3y ago
  > Clickhouse is proprietary No. Clickhouse is opensource with Apache License [0]. [0] - https://github.com/ClickHouse/ClickHouse/blob/master/LICENSE https://github.com/ClickHouse/ClickHouse/blob/master/LICENSE
- deleted 3y ago
  [deleted]

mikeshi42 3y ago

Thank you, really appreciate the feedback and encouragement! > It seems like clickhouse is obviously a big piece of the tech here, which is an obvious choice, but from my experience with high data rate ingest, especially logs, you can run into issues at larger scale. Is that something you expect to give options around in open source? Scaling any system can be challenging - our experience so far is that Clickhouse is a fraction of the overhead of systems like Elasticsearch has previously demanded luckily. That being said, I think there's always going to be a combination of learnings we'd love to open source for operators that are self-hosting/managing Clickhouse, and tooling we use internally that is purpose-built for our specific setup and workloads. > I saw what is in OSS vs cloud and I think it is a reasonable way to segment, especially multi-tenancy, but do you see the split always being more management/security features? Our current release - we've open sourced the vast majority of our feature set, including I think some novel features like event patterns that typically are SaaS-only and that'll definitely be the way we want to continue to operate. Given the nature of observability - we feel comfortable continuing to keep pushing a fully-featured OSS version while having a monetizable SaaS that focuses on the fact that it's completely managed, rather than needing to gate heavily based on features. > on OSS, I was surprised to see MIT license We want to make observability accessible and we think AGPL will accomplish the opposite of that. While we need to make money at the end of the day - we believe that a well-positioned enterprise + cloud offering is better suited to pull in those that are willing to pay, rather than forcing it via a license. I also love the MIT license and use it whenever I can :) > On that note, I am curious what your target persona and GTM plan is looking like? I think for small teams, imo the options available are largely untantilizing, it ranges from narrow tools like Cloudwatch to enterprise-oriented tools like New Relic or Datadog. We're working hard to make it easier for those kinds of teams to adopt good monitoring and observability from day 1, without the traditional requirement of needing an observability expert or dedicated SRE to get it set up. (Admittedly, we still have a ways to improve today!) On the enterprise side, switching costs are definitely high, but most enterprises are highly decentralized in decision making, where I routinely hear F500s having a handful of observability tools in production at a given time! I'll say it's not as locked-in as it seems :)

addisonj 3y ago
Thanks for the answers Mike! One more follow-up on the scale side (which I mentioned with sibling comment), it isn't so much about clickhouse itself, but about scaling up ingest. From my own experience and from talking with quite a few APM players (I previously worked in streaming space), a Kafka / durable log storage kind of becomes a requirement, so I was curious if you think at some point you need a log to further scale ingest. For enterprise side, I was previously in data streaming space and had quite a few conversations with APM players and companies building their own observability platforms, happy to chat and share more if that would be useful!
- mikeshi42 3y ago
  Ah got it, yeah a queue of some sort is definitely useful when scaling up to buffer pre-inserted data. This is something on the OSS side we've kept open to implementation. However it's something that is highly coupled with infra footprint and internal SLA guarantees the user wants to preserve. It can range anywhere from just rely on client-side retries to setting up a HA Kafka cluster early in the ingestion pipeline. Similar to Elastic - I think a lot of architectures are available to choose on that side when users want to scale. Will reach out to connect!
debarshri 3y ago
One piece of advice here is, if you pitch yourself as a datadog competitor, then I would recommend replicating some of the GTM motions that datadog employed. For instance, you have an opportunity to go very upmarket, super enterprise orgs. You can do PLG, but ultimately every tool becomes SLG. I would recommend fine tuning that motion as that would be the one bringing larger contract 6 digit contracts and huge growth here. I have seen orgs remove datadog because of unpredictable pricing. If you do flat price self hosted platform, you will get attention. I dont think orgs would mind hosting clickhouse. You can also bundle it with your helm charts or initial proof of concept might have lower barrier. I know some orgs have million dollar annual contracts with datadog, a cheaper more predictable priced alternative will definitely get attention.
- 3y ago

dangoodmanUT 3y ago

For clickhouse, just batch insert. They probably have something batching every few s before inserting directly to their hosted version

dangoodmanUT 3y ago
There's also async inserts
addisonj 3y ago
Generally, any sort of async/batch inserts will get you decently far, but still will have limitations well before you get to million rows a second, mostly because it is really difficult to get your batch size large enough from individual producers without some sort of aggregation, which that aggregation is a challenge if you care about durability. So often that means you need something like a Kafka to get the bulk ingest to really perform to get batch sizes large enough. That kind of gets into one of the challenges of OSS observabilility systems, you don't want to make the dependencies insane for someone who only has a few thousand logs a second, but generally at some point of scale you do need more.
vadman97 3y ago
ClickHouse Async insert docs [1]. We ran into some challenges with async inserts at highlight.io [2]. Namely, ClickHouse Cloud has an async flush size configured (that can't be changed AFAIK) that isn't large enough for our scale. Once you async insert more than can be flushed, you get back pressure on your application waiting to write while Clickhouse flushes the queue. We found that implementing our own batched flushing via kafka [3] is far more performant, allowing us to insert 500k+ RPS on the smallest cloud instance type. [1] https://clickhouse.com/docs/en/optimize/asynchronous-inserts https://clickhouse.com/docs/en/optimize/asynchronous-inserts [2] https://github.com/highlight/highlight/tree/main https://github.com/highlight/highlight/tree/main [3] https://github.com/highlight/highlight/blob/4d28451b1935796d2f400a7589e430b2fa2e0229/backend/worker/kafka_worker.go#L104-L174 https://github.com/highlight/highlight/blob/4d28451b1935796d...

fnord77 3y ago

Clickhouse is proprietary, though. I wonder why not Apache Druid

zx8080 3y ago
> Clickhouse is proprietary No. Clickhouse is opensource with Apache License [0]. [0] - https://github.com/ClickHouse/ClickHouse/blob/master/LICENSE https://github.com/ClickHouse/ClickHouse/blob/master/LICENSE
deleted 3y ago
[deleted]