5 ms·

Interesting, that's not a problem I've come across before particularly - could you share more? Are you looking for setups for OSS ClickHouse or managed ClickHo

by sdairs 2y ago

Interesting, that's not a problem I've come across before particularly - could you share more? Are you looking for setups for OSS ClickHouse or managed ClickHouse services that solve it? Both Tinybird & ClickHouse Cloud are managed ClickHouse services that include ingest connectors without needing Kafka Estuary (an ETL tool) just released Dekaf which lets them appear as a Kafka broker by exposing a Kafka-compatible API, so you can connect it with ClickHouse as if it was Kafka, without actually having Kafka (though I'm not sure if this is in the open source Estuary Flow project or not, I have a feeling not) If you just want to play with CH, you can always use clickhouse-local or chDB which are more like DuckDB, running without a server, and work great for just talking to local files. If you don't need streams and are just working with files, you can also use them as an in-process/serverless transform engine - file arrives, read with chDB, process it however you need, export it as CH binary format, insert directly into your main CH. Nice little pattern than can run on a VM or in Lambda's.

maccard 2y ago
Sure - I work in games, and we stream eventsfrom clients that we want to store in Clickhouse. We've got a native desktop application written in C++ that generates a json payload (we control the format of this). We don't need OSS, but we don't want a SAAS service - we want on-prem (or self managed). Clickhouse Cloud would be fine, TinyBird not. > Estuary (an ETL tool) just released Dekaf which lets them appear as a Kafka broker by exposing a Kafka-compatible API This is definitely an improvement, but if it looks like kafka and sounds like kafka, I get a bit sus. > If you just want to play with CH, you can always use clickhouse-local I've done that, but getting from this to "streaming data" is where I get stuck. > If you don't need streams Afraid streams are what I'm dealing with..
- DeathArrow 2y ago
  What's wrong with using Postgres, MySQL or SQL server?
  - maccard 2y ago
    They work fine, but this is a thread on Clickhouse. Clickhouse sells itself as a DBMS for real time analytical reports, which is exactly what I want. But I can't compare the two because I've never managed to get it all stood up.
    - nrjames 2y ago
      I work in gaming and stream events into a self-hosted Clickhouse db without Kafka. We just use the CH python connector and send records in batches of 100K, using ReplacingMergeTree for backfills, etc. It works very well. Unless you truly need up-to-the-minute analytics, it’s super easy to schedule with Dagster or Airflow or whatever. We process 100M+ events per day this way.
      - maccard 2y ago
        How do you ensure retries, single entries, and not losing 100k entries if your app goes down? It's also kind of a bummer that the batches have to be inserted, when the tagline on Clickhouse's website is: > Build real-time data products that scale But, thanks for the clarification!
        nrjames 2y ago
        We do overlapping inserts and let ReplacingMergeTree remove the duplicates. You can use the FINAL statement on select queries if you’re concerned about queries returning duplicates before the Clickhouse backend performs the deduplication mutation.
        maccard 2y ago
        Ah, great. Thanks for the info!
        hedora 2y ago
        I've seen a few solutions in this space that use an RDBMS as a glorified spool file. So, append log entries to PG or MySQL or whatever over a rest endpoint (like the one splunk exposes to writers), and then have a few workers (for fault tolerance) that the 100K oldest entries in the table every few seconds, stick them into the "real-time" system, delete them from the DBMS and commit. I've never understood why this isn't just done better by the downstream product though. It's not that hard to implement a performant write ahead log from scratch. (Note that you can scale out the above arbitrarily, since there's no reason to limit yourself to one worker or one DBMS.)
- ruslan_talpa 2y ago
  I’ve got a lib/executable that you spin up and it gives you a rest api (read and write) for clickhouse if you are interested
  - latchkey 2y ago
    I know this is a separate point, but the pricing link on your website in the header, is broken.

by sdairs 2y ago

maccard 2y ago

Sure - I work in games, and we stream eventsfrom clients that we want to store in Clickhouse. We've got a native desktop application written in C++ that generates a json payload (we control the format of this). We don't need OSS, but we don't want a SAAS service - we want on-prem (or self managed). Clickhouse Cloud would be fine, TinyBird not. > Estuary (an ETL tool) just released Dekaf which lets them appear as a Kafka broker by exposing a Kafka-compatible API This is definitely an improvement, but if it looks like kafka and sounds like kafka, I get a bit sus. > If you just want to play with CH, you can always use clickhouse-local I've done that, but getting from this to "streaming data" is where I get stuck. > If you don't need streams Afraid streams are what I'm dealing with..

DeathArrow 2y ago
What's wrong with using Postgres, MySQL or SQL server?
- maccard 2y ago
  They work fine, but this is a thread on Clickhouse. Clickhouse sells itself as a DBMS for real time analytical reports, which is exactly what I want. But I can't compare the two because I've never managed to get it all stood up.
  - nrjames 2y ago
    I work in gaming and stream events into a self-hosted Clickhouse db without Kafka. We just use the CH python connector and send records in batches of 100K, using ReplacingMergeTree for backfills, etc. It works very well. Unless you truly need up-to-the-minute analytics, it’s super easy to schedule with Dagster or Airflow or whatever. We process 100M+ events per day this way.
    - maccard 2y ago
      How do you ensure retries, single entries, and not losing 100k entries if your app goes down? It's also kind of a bummer that the batches have to be inserted, when the tagline on Clickhouse's website is: > Build real-time data products that scale But, thanks for the clarification!
      - nrjames 2y ago
        We do overlapping inserts and let ReplacingMergeTree remove the duplicates. You can use the FINAL statement on select queries if you’re concerned about queries returning duplicates before the Clickhouse backend performs the deduplication mutation.
        maccard 2y ago
        Ah, great. Thanks for the info!
      - hedora 2y ago
        I've seen a few solutions in this space that use an RDBMS as a glorified spool file. So, append log entries to PG or MySQL or whatever over a rest endpoint (like the one splunk exposes to writers), and then have a few workers (for fault tolerance) that the 100K oldest entries in the table every few seconds, stick them into the "real-time" system, delete them from the DBMS and commit. I've never understood why this isn't just done better by the downstream product though. It's not that hard to implement a performant write ahead log from scratch. (Note that you can scale out the above arbitrarily, since there's no reason to limit yourself to one worker or one DBMS.)
ruslan_talpa 2y ago
I’ve got a lib/executable that you spin up and it gives you a rest api (read and write) for clickhouse if you are interested
- latchkey 2y ago
  I know this is a separate point, but the pricing link on your website in the header, is broken.