5 ms·

Show HN: AutoMQ - A Cost-Effective Kafka Distro That Can Autoscale in Seconds

58 points by wanshao 2y ago

jackbauer24 2y ago
Scaling Kafka has always been a headache for me, but seeing AutoMQ storing data entirely on S3, making Brokers stateless, no longer needing to migrate data for Kafka scaling. Awesome.
metadat 2y ago
They're making a lot of claims and in the readme present zero measurements. The official Kafka distro is pretty bulletproof since 7+ years ago. It'd be great if they could include more information for the (apparently). initiated! I am open to new things.. with at least a tad of supporting data attached :D Ymmv.
- deleted 2y ago
  [deleted]
- chenyang 2y ago
  Hi, thank you for your interest in AutoMQ. If you have any questions regarding cost-related data, here is an analysis report for your reference: https://docs.automq.com/docs/automq-s3kafka/EJBvwM3dNic6uYkZAWwc7nmrnae https://docs.automq.com/docs/automq-s3kafka/EJBvwM3dNic6uYkZ...
- wanshao 2y ago
  Thank you for your openness to new things. Your support is indeed very important to a new project. BTW, AutoMQ itself provides real comparison data tested on AWS, and the related code is open for users to reproduce these tests. For a cost comparison with Apache Kafka, you can refer to: https://www.automq.com/blog/automq-vs-apache-kafka-a-real-aws-cloud-bill-comparison https://www.automq.com/blog/automq-vs-apache-kafka-a-real-aw.... If you care about performance data, you can refer to our performance white paper: https://docs.automq.com/docs/automq-s3kafka/CYxlwqDBHitThCkxSl2cePxrnBc https://docs.automq.com/docs/automq-s3kafka/CYxlwqDBHitThCkx....
looopTools 2y ago
Seems nice, but are there any publicly available documentation to be able to evaluate its readiness and what languages are there libraries for? I may be blind but I cannot find any.
- jpgvm 2y ago
  If you need to ask you aren't ready for something like this. That said, it's basically Kafka with the topic-partition persistence swapped out. So the libraries etc are just the standard Kafka libraries because the frontend listener is unmodified.
  - chenyang 2y ago
    Yes, thank you for the clarification. AutoMQ has replaced the topic-partition storage with cloud-native S3Stream (https://github.com/AutoMQ/automq/tree/main/s3stream https://github.com/AutoMQ/automq/tree/main/s3stream) library, thereby harnessing the benefits of cloud EBS and S3.
    - jpgvm 2y ago
      One thing that isn't made clear is when writes are acknowledged. Specifically is a write acknowledged when it's written to Delta WAL or when it's uploaded to object storage? If writes are acknowledged when written to Delta WAL is it possible to lose acknowledged writes when an EBS volume becomes unavailable or does that whole partition become unwritable until the volume comes back? Or is Delta WAL itself replicated in a similar fashion to traditional Kafka storage?
      - chenyang 2y ago
        Yes, acknowledgments for writes occur once the data is committed to the EBS WAL, with each write operation bypassing the cache via Direct IO. Data is then asynchronously uploaded to S3. Given that EBS already ensures various levels of data durability, AutoMQ does not replicate data. Addressing your last question regarding the scenario when an EBS volume becomes unavailable: - AutoMQ maintains a minimal amount of data on EBS, for example, only 500MB, which can be easily cached in memory. If an EBS volume goes offline, we promptly upload all data to S3 and close all partitions on the affected broker. Subsequently, we redistribute the closed partitions to other brokers.
donglitao 2y ago
The Kafka API and S3 API have each become the de facto standard in the stream processing and object storage domains, respectively. Compared to other Kafka solutions, AutoMQ has embraced them more effectively. I believe AutoMQ has a long way to go.
- jpgvm 2y ago
  It's a bit unfortunate IMO because Kafka while perfectly adequate for streaming often winds up with someone wanting to use it like a work queue which predictably ends horribly. Ideally something like Pulsars API (which more resembles something like GCP Pub/Sub) would be the de facto as it is capable of handling both cases seamlessly. There was Kafka-on-Pulsar until recently until the developers essentially made it non-OSS which is pretty unfortunate for Pulsar adoption which is already low.
  - deleted 2y ago
    [deleted]
  - fortylove 2y ago
    Can you elaborate more on the part where using Kafka as a work queue ends horribly? Genuinely interested!
    - jpgvm 2y ago
      Essentially the problem is that Kafka doesn't have a way of managing acknowledgements on a per-message basis. This means that consumers from Kafka topics are assigned exclusive access on a per-partition basis and the consumer group manager only tracks acknowledged offset of each partition. As such you end up with a few main problems. The first is head of line blocking, what this means is that if a consumer reads a message from a topic it's unable to process or will take an inordinate amount of time to process it can't move forward without potentially having to replay every message since the problematic message if it doesn't want to risk not replaying a message that wasn't processed correctly. Secondly it means that you can end up with hot partitions if the "cost" of messages isn't uniformly distributed across partitions because load isn't balanced across consumers, i.e there is no work stealing or other mechanism for other consumers to help out processing a hot partition. Log systems with queue/subscription overlays like Pulsar and GCP Pub/Sub solve this by doing per-message acknowledgement (sometimes referred to as selective acknowledgement vs cumulative acknowledgement that Kafka does) usually by layering a persistent subscription abstraction over the top of the underlying log. This is in contrast to pure queue systems like RabbitMQ, SQS etc that use a heap or mailbox approach where messages are simply emptied out as they are processed and don't share the log style struction of systems like Kafka. So TLDR. If you use Kafka like a job queue you will end up in situations where queue processing gets stuck behind a single or patches of unprocessable messages. The mitigations for it aren't pretty. They either involve building your own selective acknowledgement layer or a series of retry queues that messages are pushed onto using Kafka transactions with a final dead-letter queue at the end etc. Instead either wait for https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A... if you really want Kafka or use something that already does this, i.e Pulsar.
threeseed 2y ago
Be interesting to see how well they do. Most of the Kafka deployments I've seen in recent years have been using cloud versions e.g. AWS Managed Kafka. So it's great that you save money on compute but it does mean you are now operating it yourself which then costs money.
- chenyang 2y ago
  Indeed, operating Kafka can be challenging and complex due to its nature as a stateful and distributed system. However, AutoMQ has adopted a cloud-native architecture, offloading storage to EBS and S3(https://docs.automq.com/docs/automq-s3kafka/Q8fNwoCDGiBOV6k8CDSccKKRn9d https://docs.automq.com/docs/automq-s3kafka/Q8fNwoCDGiBOV6k8...), eliminating the need for replication and rendering the Broker stateless, which simplifies operations significantly.
deleted 2y ago
[deleted]
lyx2000 2y ago
[dead]
Izeg 2y ago
Is it truly possible for AutoMQ to achieve 'scale-in/out' within seconds?
west0n 2y ago
I bet that all message queues and log databases will support S3, as these types of data generally have a large volume and aren't as economically valuable (don't get me wrong, what I mean is that these databases won't be frequently read and processed).
- chenyang 2y ago
  Can't agree more! S3 will be the modern data storage primitive. Also, the move towards shared storage and separating compute from storage is a key trend in cloud-native architecture, enhancing scalability and cost-efficiency.
TalktoCrystal 2y ago
This is interesting.
hack-tech 2y ago
[flagged]
panda-tech 2y ago
[flagged]

58 points by wanshao 2y ago

Scaling Kafka has always been a headache for me, but seeing AutoMQ storing data entirely on S3, making Brokers stateless, no longer needing to migrate data for Kafka scaling. Awesome.

metadat 2y ago

They're making a lot of claims and in the readme present zero measurements. The official Kafka distro is pretty bulletproof since 7+ years ago. It'd be great if they could include more information for the (apparently). initiated! I am open to new things.. with at least a tad of supporting data attached :D Ymmv.

deleted 2y ago
[deleted]
chenyang 2y ago
Hi, thank you for your interest in AutoMQ. If you have any questions regarding cost-related data, here is an analysis report for your reference: https://docs.automq.com/docs/automq-s3kafka/EJBvwM3dNic6uYkZAWwc7nmrnae https://docs.automq.com/docs/automq-s3kafka/EJBvwM3dNic6uYkZ...
wanshao 2y ago
Thank you for your openness to new things. Your support is indeed very important to a new project. BTW, AutoMQ itself provides real comparison data tested on AWS, and the related code is open for users to reproduce these tests. For a cost comparison with Apache Kafka, you can refer to: https://www.automq.com/blog/automq-vs-apache-kafka-a-real-aws-cloud-bill-comparison https://www.automq.com/blog/automq-vs-apache-kafka-a-real-aw.... If you care about performance data, you can refer to our performance white paper: https://docs.automq.com/docs/automq-s3kafka/CYxlwqDBHitThCkxSl2cePxrnBc https://docs.automq.com/docs/automq-s3kafka/CYxlwqDBHitThCkx....

looopTools 2y ago

Seems nice, but are there any publicly available documentation to be able to evaluate its readiness and what languages are there libraries for? I may be blind but I cannot find any.

jpgvm 2y ago
If you need to ask you aren't ready for something like this. That said, it's basically Kafka with the topic-partition persistence swapped out. So the libraries etc are just the standard Kafka libraries because the frontend listener is unmodified.
- chenyang 2y ago
  Yes, thank you for the clarification. AutoMQ has replaced the topic-partition storage with cloud-native S3Stream (https://github.com/AutoMQ/automq/tree/main/s3stream https://github.com/AutoMQ/automq/tree/main/s3stream) library, thereby harnessing the benefits of cloud EBS and S3.
  - jpgvm 2y ago
    One thing that isn't made clear is when writes are acknowledged. Specifically is a write acknowledged when it's written to Delta WAL or when it's uploaded to object storage? If writes are acknowledged when written to Delta WAL is it possible to lose acknowledged writes when an EBS volume becomes unavailable or does that whole partition become unwritable until the volume comes back? Or is Delta WAL itself replicated in a similar fashion to traditional Kafka storage?
    - chenyang 2y ago
      Yes, acknowledgments for writes occur once the data is committed to the EBS WAL, with each write operation bypassing the cache via Direct IO. Data is then asynchronously uploaded to S3. Given that EBS already ensures various levels of data durability, AutoMQ does not replicate data. Addressing your last question regarding the scenario when an EBS volume becomes unavailable: - AutoMQ maintains a minimal amount of data on EBS, for example, only 500MB, which can be easily cached in memory. If an EBS volume goes offline, we promptly upload all data to S3 and close all partitions on the affected broker. Subsequently, we redistribute the closed partitions to other brokers.

donglitao 2y ago

The Kafka API and S3 API have each become the de facto standard in the stream processing and object storage domains, respectively. Compared to other Kafka solutions, AutoMQ has embraced them more effectively. I believe AutoMQ has a long way to go.

jpgvm 2y ago
It's a bit unfortunate IMO because Kafka while perfectly adequate for streaming often winds up with someone wanting to use it like a work queue which predictably ends horribly. Ideally something like Pulsars API (which more resembles something like GCP Pub/Sub) would be the de facto as it is capable of handling both cases seamlessly. There was Kafka-on-Pulsar until recently until the developers essentially made it non-OSS which is pretty unfortunate for Pulsar adoption which is already low.
- deleted 2y ago
  [deleted]
- fortylove 2y ago
  Can you elaborate more on the part where using Kafka as a work queue ends horribly? Genuinely interested!
  - jpgvm 2y ago
    Essentially the problem is that Kafka doesn't have a way of managing acknowledgements on a per-message basis. This means that consumers from Kafka topics are assigned exclusive access on a per-partition basis and the consumer group manager only tracks acknowledged offset of each partition. As such you end up with a few main problems. The first is head of line blocking, what this means is that if a consumer reads a message from a topic it's unable to process or will take an inordinate amount of time to process it can't move forward without potentially having to replay every message since the problematic message if it doesn't want to risk not replaying a message that wasn't processed correctly. Secondly it means that you can end up with hot partitions if the "cost" of messages isn't uniformly distributed across partitions because load isn't balanced across consumers, i.e there is no work stealing or other mechanism for other consumers to help out processing a hot partition. Log systems with queue/subscription overlays like Pulsar and GCP Pub/Sub solve this by doing per-message acknowledgement (sometimes referred to as selective acknowledgement vs cumulative acknowledgement that Kafka does) usually by layering a persistent subscription abstraction over the top of the underlying log. This is in contrast to pure queue systems like RabbitMQ, SQS etc that use a heap or mailbox approach where messages are simply emptied out as they are processed and don't share the log style struction of systems like Kafka. So TLDR. If you use Kafka like a job queue you will end up in situations where queue processing gets stuck behind a single or patches of unprocessable messages. The mitigations for it aren't pretty. They either involve building your own selective acknowledgement layer or a series of retry queues that messages are pushed onto using Kafka transactions with a final dead-letter queue at the end etc. Instead either wait for https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A... if you really want Kafka or use something that already does this, i.e Pulsar.

threeseed 2y ago

Be interesting to see how well they do. Most of the Kafka deployments I've seen in recent years have been using cloud versions e.g. AWS Managed Kafka. So it's great that you save money on compute but it does mean you are now operating it yourself which then costs money.

chenyang 2y ago
Indeed, operating Kafka can be challenging and complex due to its nature as a stateful and distributed system. However, AutoMQ has adopted a cloud-native architecture, offloading storage to EBS and S3(https://docs.automq.com/docs/automq-s3kafka/Q8fNwoCDGiBOV6k8CDSccKKRn9d https://docs.automq.com/docs/automq-s3kafka/Q8fNwoCDGiBOV6k8...), eliminating the need for replication and rendering the Broker stateless, which simplifies operations significantly.

deleted 2y ago

[deleted]

lyx2000 2y ago

[dead]

Izeg 2y ago

Is it truly possible for AutoMQ to achieve 'scale-in/out' within seconds?

west0n 2y ago

I bet that all message queues and log databases will support S3, as these types of data generally have a large volume and aren't as economically valuable (don't get me wrong, what I mean is that these databases won't be frequently read and processed).

chenyang 2y ago
Can't agree more! S3 will be the modern data storage primitive. Also, the move towards shared storage and separating compute from storage is a key trend in cloud-native architecture, enhancing scalability and cost-efficiency.

TalktoCrystal 2y ago

This is interesting.

hack-tech 2y ago

[flagged]

panda-tech 2y ago

[flagged]