kleineshertz

4 ms·

1.
by kleineshertz 3y ago
Right. And this is where our paths go separate ways. As I see it, Capillaries users are not necessarily tech companies and they do not have much appetite for writing and maintaining a lot of code. All they want is to run, say, 50 kinds of w
2.
by kleineshertz 3y ago
Storm positions itself as a stream processing solution, while Capillaries is 100% batch-oriented.
3.
by kleineshertz 3y ago
It's always a balance. I have been working with teams on both side of the fence and I think I am well aware of the dangers of both: keeping the custom wheel running for years vs fighting the particularities of a third-party tool (up to
4.
by kleineshertz 3y ago
Regarding using in-memory storage. Early prototype of Capillaries used Redis for storage and the performance was stellar. I decided to drop it for two reasons. First, indexing mechanism required a root-level sorted set, and Redis cannot par
5.
by kleineshertz 3y ago
Parquet support is on the radar for sure, and I would like to have it before diving into database connector development.
6.
by kleineshertz 3y ago
ScyllaDB is definitely on the radar. The main reason I picked Cassandra on the prototyping stage was because default Cassandra configuration gave me much better performance then ScyllaDB (I know, it is supposed to be vice versa). Another ob
7.
by kleineshertz 3y ago
If it's not invented here, it can't be any good.
8.
by kleineshertz 3y ago
Maybe. The scenarios Capillaries is intended for do not need complex/flexible workflow, we just need some basic dependency rules (easy to implement) and really reliable scheduling (RabbitMQ).
9.
by kleineshertz 3y ago
Temporal is a different ecosystem (and a much more ambitious solution), but one of the principles is the same: users want a platform that solves scalability issues and lets them focus on biz logic and customer value.
10.
by kleineshertz 3y ago
Nothing wrong with this question. I do not have any experience with Spark, but I guess Capillaries belongs to the same or similar ecosystem. My understanding is that Spark is way more generic framework that revolves around DAG-defined workf
11.
Show HN: Capillaries: Distributed data processing with Go and Cassandra(capillaries.io)
70 points by kleineshertz 3y ago|25 comments
12.
by kleineshertz 3y ago
Capillaries is a distributed data processing platform that: - works with structured row-based data - splits data into batches that can be processed as separate jobs on multiple machines in parallel - allows scenarios that involve human oper
13.
Capillaries: Distributed data processing with Go and Cassandra(capillaries.io)
1 points by kleineshertz 3y ago|1 comments