4 ms·
Our latency concerns are "can our product managers make decisions quickly", to that end we're OK with a 24 hour latency for stats. The fact that we're at 5 hour
by dayjah 13y ago
Our latency concerns are "can our product managers make decisions quickly", to that end we're OK with a 24 hour latency for stats. The fact that we're at 5 hour latency gives us a lot of breathing room.
Kinesis looks really interesting, and is definitely something that we're going to look at once we work out our ETL process.
The order of priority for us has been:
1 - Get a pipeline up and running
2 - Make it robust
3 - Make it fast.
Pipeline v3, our current one, satisfies (2). We expect to be working on (3) in the near future. ETL is the latter part of (2). (3) results in powering dashboards, we expect those to contain a lot of joined data and having a robust ETL process is pretty key to that.
- alexatkeplar 13y agoYes - Kinesis is awesome. This is already a little out of date but shows how we're now porting our open source event pipeline to on top of Kinesis: http://snowplowanalytics.com/blog/2014/02/04/snowplow-0.9.0-released-with-beta-kinesis-support/ http://snowplowanalytics.com/blog/2014/02/04/snowplow-0.9.0-...