4 ms·

Seems like a useful reworking of SQS, but all the hard work is being done in the client: "client library automatically handle complex issues like adapting to ch

by fizx 13y ago

Seems like a useful reworking of SQS, but all the hard work is being done in the client: "client library automatically handle complex issues like adapting to changes in stream volume, load-balancing streaming data, coordinating distributed services, and processing data with fault-tolerance." Unfortunately, there's no explanation of the mechanics of coordination and fault tolerance, so the hard part appears to be vaporware.

javajammer 13y ago
The currently available docs reveal the client-nodes coordinate through a DynamoDB table. Processing with the library yields "at least once" semantics. http://docs.aws.amazon.com/kinesis/latest/dev/kinesis-record-processor-app.html http://docs.aws.amazon.com/kinesis/latest/dev/kinesis-record...
- fizx 13y ago
  thanks!
vosper 13y ago
> Unfortunately, there's no explanation of the mechanics of coordination and fault tolerance, so the hard part appears to be vaporware. I think it's unfair to call it vaporware - Amazon doesn't tend to release vaporware. You can also be fairly confident this has been in private beta for some time, so we'll probably see a few blog posts about it from some of their privileged (big spending) clients - typically someone like Netflix or AirBnB. But I agree it would be nice to get some more information on the details. As for the client library handling load-balancing, fault tolerance, etc - that might not be ideal, but as long as I don't have to do it myself then it might be okay.
- fizx 13y ago
  The client handling it is ideal from a systems perspective, because the app won't forget to be fault tolerant on its connection to the server. Its less ideal from a maintenance perspective, because there will need to be feature-rich clients in Java and C (with dynamic language bindings). Applications will be running many many versions of the clients. Also, for coordination, the clients will need to communicate, so there may be configuration and/or firewall issues for the app to resolve. It will be interesting to see Amazon make this tradeoff for what I believe is the first time.
  - aluskuiuc 13y ago
    It's not exactly the first time, but close - the Simple Workflow Service has client helper libraries for both Java and Ruby.

by fizx 13y ago

javajammer 13y ago

The currently available docs reveal the client-nodes coordinate through a DynamoDB table. Processing with the library yields "at least once" semantics. http://docs.aws.amazon.com/kinesis/latest/dev/kinesis-record-processor-app.html http://docs.aws.amazon.com/kinesis/latest/dev/kinesis-record...

fizx 13y ago
thanks!

vosper 13y ago

> Unfortunately, there's no explanation of the mechanics of coordination and fault tolerance, so the hard part appears to be vaporware. I think it's unfair to call it vaporware - Amazon doesn't tend to release vaporware. You can also be fairly confident this has been in private beta for some time, so we'll probably see a few blog posts about it from some of their privileged (big spending) clients - typically someone like Netflix or AirBnB. But I agree it would be nice to get some more information on the details. As for the client library handling load-balancing, fault tolerance, etc - that might not be ideal, but as long as I don't have to do it myself then it might be okay.

fizx 13y ago
The client handling it is ideal from a systems perspective, because the app won't forget to be fault tolerant on its connection to the server. Its less ideal from a maintenance perspective, because there will need to be feature-rich clients in Java and C (with dynamic language bindings). Applications will be running many many versions of the clients. Also, for coordination, the clients will need to communicate, so there may be configuration and/or firewall issues for the app to resolve. It will be interesting to see Amazon make this tradeoff for what I believe is the first time.
- aluskuiuc 13y ago
  It's not exactly the first time, but close - the Simple Workflow Service has client helper libraries for both Java and Ruby.