Same here :) Right now, we're just starting to automate a bunch of work that our one overworked analyst does manually with GA, FB marketing insights, and a bunc

Same here :) Right now, we're just starting to automate a bunch of work that our one overworked analyst does manually with GA, FB marketing insights, and a bunch of third party services. She is asking dev to extract data sets that crash Excel when she imports them :/ We need to integrate all of this data into one somewhat consistent stream. Then whenever she asks for a data set, we (I) need to ask her what she's going to do with it, and whip up a little stream processing service that can answer that question continuously. Preferably in chat, not with a big fancy dashboard, because really, all those graphs are just a means to an end. Her output isn't a graph, it's a single paragraph recommendation or a boolean answer to a vague question. I've just started looking into Apache Kafka and Storm. The whole stream oriented paradigm looks really promising. If we could archive a huge torrent of incoming events, and then build little stream processing services over time, as specific questions come in from analytics, and replay old events through the system, I think that would put us in the position we want to be in. I've been playing around with Storm on my laptop in clojure (Clojure is awesomesauce), and contemplating how we should deploy this. As far as deployment goes, what I would like to do, is set up a big fancy Mesos cluster to run all these little stream processors, and heck, might as well run all of our product services in it too. What I will probably have to settle for, is starting off on a single EC2 instance that occasionally has little accidents. My ops skill are... a work in progress. Anyway, if you want to get in touch, shoot me an email or something. daniel.ross at kasra.co. It would be really cool to watch several of these systems built up in parallel, so that we could all see what is working and what problems need to be solved by a bigger group than our own. It isn't easy convincing the CEO to let me open source stuff like this, but it also isn't easy to hire developers who can learn a new sub-field and build something cool in it, at least not for a relatively small company.