ashishbagri

3 ms·

1. 1.▲
   by ashishbagri 1y ago

   Real-time synthetic data generation with built in connectors https://github.com/glassflow/glassgen Next step is to extend it as a server module so you can run it remotely
2. 2.▲
   by ashishbagri 1y ago

   - We used our custom clickhouse sink which inserts records in batches using clickhouse native protocol (as recommend by clickhouse). Each insert is done in a single transaction so if an insertion has failed, partial record do not get insert
3. 3.▲
   by ashishbagri 1y ago

   Thanks for your question. In GlassFlow, we use NATs Jetstream to power deduplication (and KV store for joins as well). I see from your blog post that segment used rocksDB to power their deduplication pipeline. We actually considered using r
4. 4.▲
   by ashishbagri 1y ago

   Yes its true that if you just want to send data from Kafka to clickhouse and do not worry about duplicates, then there are several ways. we even covered them in a blog post -> https://www.glassflow.dev/blog/part-1-ka
5. 5.▲
   by ashishbagri 1y ago

   Thanks for taking a look! 1. The current implementation is just for clickhouse as we started with the segment of users building real time analytics with clickhouse in their stack. However we already learned during the way that streaming de
6. 6.▲
   by ashishbagri 1y ago

   Yes it would be easily possible to configure the tool to stream directly from NATs and skip Kafka completely. The reason we started with a managed Kafka connector (via the NATS Kafka Bridge) is because most of the early users sending data t
7. 7.▲
   Show HN: A real world streaming data generator in Python(github.com)

   1 points by ashishbagri 1y ago|0 comments