4 ms·
I'm not being negative about this, it's a cool setup. But just to be clear, the pattern (not marketing terms) is doing change data capture (essentially the dat
by sixdimensional 8y ago
I'm not being negative about this, it's a cool setup.
But just to be clear, the pattern (not marketing terms) is doing change data capture (essentially the database transaction log) to a message queue, with message/job processors that can take any action, including writing the messages to other databases.
Kinesis is SQLStream underneath, which is probably why the lifetime of messages is limited - it's not originally intended to be Kafka or a durable message queue.
EDIT: Note above, when SQLStream first came out it didn't seem intended as a long term store. That was like really early on when I saw it at Strata. It looks like they made the storage engine pluggable and Kafka is an option too, so my statement above is likely incorrect.
Lambda is being used as a distributed message/job processor, much like any worker process processing a queue would be scaled up.
- otterley 8y ago> Kinesis is SQLStream underneath That's the first I've heard of this. Any citations you can provide to substantiate this claim? My understanding is that SQLStream is an event streaming processor, which would make it a potential Kafka consumer, not a basis for a durable message queue.
- sixdimensional 8y agoAmazon Web Services Has Licensed SQLstream Technology for Amazon ... https://sqlstream.com/amazon-web-services-has-licensed-sqlstream-technology-for-amazon-kinesis-analytics-service-launched-today/ https://sqlstream.com/amazon-web-services-has-licensed-sqlst...
- ryanworl 8y agoKinesis Analytics uses Kinesis Streams as a data source. They are not the same service.
- sixdimensional 8y agoI see your point, although I sense they are quite closely integrated? Additional info.. https://www.datanami.com/2016/08/17/sql-service-heart-amazon-kinesis-analytics/ https://www.datanami.com/2016/08/17/sql-service-heart-amazon...
- otterley 8y agoThere are multiple different products under the same "Kinesis" branding umbrella, among them: * Kinesis Streams (the product being discussed in the post) * Kinesis Firehose * Kinesis Analytics Most people, when they discuss Kinesis, mean Kinesis Streams specifically. The other products are Streams consumers.
- pixelmonkey 8y agoI don't have this on authority, but I believe Kinesis Streams is an AWS-maintained fork of Kafka around 0.7. Firehose is a wrapper around Lambda + Kinesis Streams. And Kinesis Analytics is a SQLStream wrapper with Kinesis Streams as its input. That's my best understanding, anyway.
- coredog64 8y agoI heard a rumor that Amazon wanted to offer Kafka as a managed service, couldn't get it to work they way they wanted, and then said "f--k this" and wrote something that is functionally similar.
- pixelmonkey 8y agoThat's wholly possible. They might have just copied the "rough design" of topics/consumers and queue-as-log, but clean-house implemented it. I wouldn't be surprised. Kafka was a pretty minimal codebase at that stage.
- jamestimmins 8y ago"Lambda is being used as a distributed message/job processor, much like any worker process processing a queue would be scaled up." Is this just clarification of the architecture, ie how Lambda fits into pre-existing patterns, or are you suggesting that since Lambda is being used as a distributed job processor, there would be a better tool for that specific task?
- sixdimensional 8y agoJust clarification. Lately, I've found myself being asked to explain to others what it is that these new things do and found relating it to existing architectural patterns was helpful. Not saying there is or isn't a better tool than Lambda - it's a new option, just clarifying the role it fits.