4 ms·
Midway through I realized they're reinventing solved problems already. Mixpanel should take a look at Apache Iceberg for your writing and Apache Pulsar to keep
by ckdarby 5y ago
Midway through I realized they're reinventing solved problems already.
Mixpanel should take a look at Apache Iceberg for your writing and Apache Pulsar to keep your cost lower by not needing to keep 7 day retention in your pipeline once the messages are ack'ed by all consumers.
For replaying you can use Trino to just read from your Iceberg and insert back into your stream.
- i0exception 5y agoIt's a lot harder to make such drastic changes to something that's already serving production traffic at scale. It's okay to reinvent solved problems sometimes because the alternative is be a multi-year project with unclear returns. Also, the 7 day retention is a feature - it helps us recover data quickly in case of bugs in our storage code.
- ckdarby 5y agoI had written a novel of counter arguments and realized anyone who tries to tell someone that their bug is a feature isn't going to see the but for what it is. Best of luck, and God help the poor souls who have to maintain this when the original authors have left.