4 ms·

Why local state is a fundamental primitive in stream processing

47 points by datascientist 12y ago

capkutay 12y ago
Shameless plug: at WebAction we have an enterprise-grade (exactly once processing, scalable, built-in streaming data integration) SQL-based streaming platform. It's clear real-time analytics is headed in this direction and we have some interesting problems to solve. If anyone is interested in chatting about it, feel free to email me john@webaction.com
vosper 12y ago
Is anyone using Samza outside of LinkedIn? I've been interested in it for a while, and I'd love to hear your experiences.
mamcx 12y ago
(I read also http://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying http://engineering.linkedin.com/distributed-systems/log-what...). I wish to solve the problem of sync data from mobile to a master database, and is about invoices, orders and that kind of data. Here, is necessary to preserve the order of the data actions. The data must return back to each client, and the clients could be offline from time to time. In some cases for a couple of days. I have total control on the solution, so I could move to any direction, but my only hard requirement is that the data in the client is on sqlite. I struggle to build a reliable solution, but I understand that a log based could be a good idea. Wonder what way could lead me closer to the solution?
- rakoo 12y ago
  So you need to process your events linearly, but you don't need to store them linearly. Here's a way you could achieve this: - keep sqlite - use Pouchdb on top of it to sync your data to your server, typically Couchdb - Since you know the order of events, tag each of them with an application-level sequence number - On your server, when you receive an event, check all events that happened before, in order. For each event, if it has not been processed, process it
  - mamcx 12y ago
    But I need to duplicate the data to pouch, then move it to couch. My question is how do the sync myself. How build the log so it be usefull to this task? To be clear, how is the algorithm for this? Any hint or open source example?

47 points by datascientist 12y ago

Shameless plug: at WebAction we have an enterprise-grade (exactly once processing, scalable, built-in streaming data integration) SQL-based streaming platform. It's clear real-time analytics is headed in this direction and we have some interesting problems to solve. If anyone is interested in chatting about it, feel free to email me john@webaction.com

vosper 12y ago

Is anyone using Samza outside of LinkedIn? I've been interested in it for a while, and I'd love to hear your experiences.

mamcx 12y ago

(I read also http://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying http://engineering.linkedin.com/distributed-systems/log-what...). I wish to solve the problem of sync data from mobile to a master database, and is about invoices, orders and that kind of data. Here, is necessary to preserve the order of the data actions. The data must return back to each client, and the clients could be offline from time to time. In some cases for a couple of days. I have total control on the solution, so I could move to any direction, but my only hard requirement is that the data in the client is on sqlite. I struggle to build a reliable solution, but I understand that a log based could be a good idea. Wonder what way could lead me closer to the solution?

rakoo 12y ago
So you need to process your events linearly, but you don't need to store them linearly. Here's a way you could achieve this: - keep sqlite - use Pouchdb on top of it to sync your data to your server, typically Couchdb - Since you know the order of events, tag each of them with an application-level sequence number - On your server, when you receive an event, check all events that happened before, in order. For each event, if it has not been processed, process it
- mamcx 12y ago
  But I need to duplicate the data to pouch, then move it to couch. My question is how do the sync myself. How build the log so it be usefull to this task? To be clear, how is the algorithm for this? Any hint or open source example?