adesh_nalpet

5 ms·

1.
by adesh_nalpet 1mo ago
Haha, it was evident. I'd echo the complexities of managing self-hosted Apache Kafka at scale. But because the Pico's semantics are close to Kafka, but not tightly coupled, it's quite feasible to support the Kafka protocol, *
2.
by adesh_nalpet 1mo ago
The team at opendata.dev/Responsive is killing it, with a strong, opinionated stance on what works well and what doesn't. I did consider building on SlateDB, but it was forcing an unnecessary primitive on streams. I have documente
3.
by adesh_nalpet 1mo ago
Precisely this! I might even add a section in the docs, “Not a replacement for Kafka,” under the FAQ. Where Kafka starts to fall short is routing. If you want to access the data of one user from user-events-topic, that’s expensive to do. Mo
4.
by adesh_nalpet 1mo ago
I'd say PicoMQ is close to being fully serverless, since nodes are stateless. But would miss out benefits of batching, caching and low-latency. S2-lite is pretty cool, but it's single-node only and built on SlateDB. So, PicoMQ is
5.
by adesh_nalpet 1mo ago
Totally agree! Tigris is awesome, although I haven't used it at scale. PicoMQ (even cluster mode) is a one click deployment on Fly.io and they use Tigris as the Object Store. https://picomq.com/docs/operations&#x2F
6.
by adesh_nalpet 1mo ago
Indeed! The binary even embeds the admin dashboard. A good starting point to get the hang of PicoMQ: https://picomq.com/docs/playground
7.
by adesh_nalpet 1mo ago
True, there are also a lot of open PRs and issues. I'm surprised no other developers or orgs are stepping in to keep the project afloat. No wonder the Apache Way goes a long way! Either way, their approach of building around an open-pr
8.
by adesh_nalpet 1mo ago
PicoMQ works with any S3-compatible object store. But I wasn't aware of GCS Rapid Bucket, it sounds a lot like AWS S3 Express, which is also zonal. And it does help with durability ACK latency quite a bit, keeping it closer to ~50ms. I
9.
by adesh_nalpet 1mo ago
Your welcome! And feel free to ask any questions as try out PicoMQ or raise on Github issues. The disk-staged WAL is currently tracked here: https://github.com/PicoMQ/picomq/issues/13 , I'm planning to sh
10.
by adesh_nalpet 1mo ago
On the same boat, and honestly, it’s not getting the attention it deserves. Given Databricks acquired ElectricSQL primarily for PGlite, the Durable Streams project is likely going to be abandoned unless new maintainers step in. I did, in fa
11.
by adesh_nalpet 1mo ago
Thank you! I'd say pretty close. But S2 is not open source, and the OSS version ( https://github.com/s2-streamstore/s2 ) is S2-Lite, which is single-node only and uses SlateDB. SlateDB is great, but it's not pu
12.
by adesh_nalpet 1mo ago
Yeah, much of the base cost I mentioned was from the compute, networking and S3 write costs, which could sustain a lot more messages for sure. It also depends on the number of active streams/rooms, the throughput bursts, as opposed to
13.
by adesh_nalpet 1mo ago
Thank you! I spent a decent chunk of time designing PicoMQ, so the documentation had a natural progression. Exactly, streams can essentially be rooms, and since the ordering is preserved, a Discord-like application is a strong use case. I’m
14.
by adesh_nalpet 1mo ago
You nailed it! With some of the similar products I've seen, they either inherit the Kafka protocol and hence the KRaft and other complexities, or go the other way with single-node only deployments, commonly just using SlateDB's si
15.
by adesh_nalpet 1mo ago
I'm glad you asked, and yes, PicoMQ does have some Kafka-like semantics. However, Kafka is great at being a huge pipe, so you'd create topics like tables. PicoMQ, on the other hand, recommends creating granular streams that make t
16.
Show HN: PicoMQ – Durable Streams over HTTP, on object storage(picomq.com)
159 points by adesh_nalpet 1mo ago|31 comments
17.
by adesh_nalpet 2y ago
Common opensource options (other than Spark and Flink): 1. Dremio: https://www.dremio.com/ 2. Trino: https://trino.io/ 3. Iceberg Java API: https://iceberg.apache.org/docs/1.6.1/ap
18.
by adesh_nalpet 2y ago
You can just used Iceberg Java API: https://iceberg.apache.org/docs/1.6.1/api/#file-level Use it with Dropwizard/Springboot, you get to expose rest APIs too.