4 ms·

The post says that latency is an issue. In this case I would look at using Kinesis (AWS's hosted Kafka equivalent) which can barf data directly into Redshift (a

• dayjah 13y ago
  Our latency concerns are "can our product managers make decisions quickly", to that end we're OK with a 24 hour latency for stats. The fact that we're at 5 hour latency gives us a lot of breathing room. Kinesis looks really interesting, and is definitely something that we're going to look at once we work out our ETL process. The order of priority for us has been: 1 - Get a pipeline up and running 2 - Make it robust 3 - Make it fast. Pipeline v3, our current one, satisfies (2). We expect to be working on (3) in the near future. ETL is the latter part of (2). (3) results in powering dashboards, we expect those to contain a lot of joined data and having a robust ETL process is pretty key to that.
  • alexatkeplar 13y ago
    Yes - Kinesis is awesome. This is already a little out of date but shows how we're now porting our open source event pipeline to on top of Kinesis: http://snowplowanalytics.com/blog/2014/02/04/snowplow-0.9.0-released-with-beta-kinesis-support/ http://snowplowanalytics.com/blog/2014/02/04/snowplow-0.9.0-...
• teleclimber 13y ago
  Kinesis was released in mid-November 2013 so this team's effort predates that service. I think Kinesis could replace the first three boxes in their diagram, and do it in real-time. (I haven't used kinesis so I could be wrong.) It's amazing how fast big data infrastructure is evolving. For anybody looking to build something it seems your chosen solution will be obsolete by the time you release.