3 ms·

I would argue that Kinesis is not the way to go for quick POC unless you're tied to the JVM. Pulsar is still niche in most enterprise. Kafka is not dead, ther

I would argue that Kinesis is not the way to go for quick POC unless you're tied to the JVM. Pulsar is still niche in most enterprise. Kafka is not dead, there are many enterprises (including 2 successful ones I've worked at in the past 5 years) that have built POCs and successful products on Kafka. Its supports all language performantly and has tons of community support. I would argue there is nothing better to build a POC on.

andonisus 6y ago
Why would you be tied to the JVM by using Kinesis? You can write a client library for any language. We did it for Go.
- agacera 6y ago
  Because the official client from AWS is written in Java. It is possible to write clients in any language, however, it is not that simple especially when you need to handle logic of scalling out or in your kinesis stream (that will split or join shards) and when you have multiple consumers in the same consumer group (you will need a distributed locking mechanism and a logic to steal locks if one consumer dies). So it is not trivial.
  - etxm 6y ago
    Sure it is. AWS ships a bunch of clients for Kinesis as well as a kinesis agent for shipping log files. https://docs.aws.amazon.com/sdk-for-go/api/service/kinesis/ https://docs.aws.amazon.com/sdk-for-go/api/service/kinesis/
    - agacera 6y ago
      This is an SDK not a full client for Kinesis. This is the only official implementation from AWS: https://github.com/awslabs/amazon-kinesis-client https://github.com/awslabs/amazon-kinesis-client The client handles horizontal scalling, checkpoiting, shards split and shards merges. Using just the SDK, you have to build this yourself (unless you are using Kinesis for use cases that dont need it to be done correctly). This is the doc for developing consumers using the SDK https://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-sdk.html https://docs.aws.amazon.com/streams/latest/dev/developing-co... And in the second paragraph of this documentation: "These examples discuss the Kinesis Data Streams API and use the AWS SDK for Java to get data from a stream. However, for most use cases, you should prefer using the Kinesis Client Library (KCL) . For more information, see Developing KCL 1.x Consumers."

andonisus 6y ago

Why would you be tied to the JVM by using Kinesis? You can write a client library for any language. We did it for Go.

agacera 6y ago
Because the official client from AWS is written in Java. It is possible to write clients in any language, however, it is not that simple especially when you need to handle logic of scalling out or in your kinesis stream (that will split or join shards) and when you have multiple consumers in the same consumer group (you will need a distributed locking mechanism and a logic to steal locks if one consumer dies). So it is not trivial.
- etxm 6y ago
  Sure it is. AWS ships a bunch of clients for Kinesis as well as a kinesis agent for shipping log files. https://docs.aws.amazon.com/sdk-for-go/api/service/kinesis/ https://docs.aws.amazon.com/sdk-for-go/api/service/kinesis/
  - agacera 6y ago
    This is an SDK not a full client for Kinesis. This is the only official implementation from AWS: https://github.com/awslabs/amazon-kinesis-client https://github.com/awslabs/amazon-kinesis-client The client handles horizontal scalling, checkpoiting, shards split and shards merges. Using just the SDK, you have to build this yourself (unless you are using Kinesis for use cases that dont need it to be done correctly). This is the doc for developing consumers using the SDK https://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-sdk.html https://docs.aws.amazon.com/streams/latest/dev/developing-co... And in the second paragraph of this documentation: "These examples discuss the Kinesis Data Streams API and use the AWS SDK for Java to get data from a stream. However, for most use cases, you should prefer using the Kinesis Client Library (KCL) . For more information, see Developing KCL 1.x Consumers."