5 ms·
>For us, latency means the time between when the _event_ actually occurs in real life (user clicks on a button) and when it showed up in all of our analytics in
by ktamura 9y ago
>For us, latency means the time between when the _event_ actually occurs in real life (user clicks on a button) and when it showed up in all of our analytics integrations (Mixpanel, Intercom, etc.). As developers, this is the definition of "latency" that really matters for us because it represents the minimum time before we can take action on our users' behaviors.
This. I call it "end-to-end latency" and it doesn't get nearly as much attention as it should. This is, by the way, why it's a bad idea to use most OLAPs (like Redshift) as a backend for Segment if you want "real-time" analytics. Column-oriented OLAPs are not designed for real-time ingest because either they don't support streaming inserts (Redshift) or it's not particularly reliable (BigQuery).
- vgt 9y agoCan you please elaborate on why you think BigQuery's streaming inserts are not particularly reliable? Separately, you can always use BigQuery's federated query capability of Bigtable. (work at G)
- ktamura 9y ago>Can you please elaborate on why you think BigQuery's streaming inserts are not particularly reliable? I've heard this from many users through my work on Fluentd (in fact, many end up using fluent-plugin-bigquery because they don't want to deal with the minutiae of insertId and whatnot on their own). Also, I don't think a lot of people fully understand the quota policy and get tripped up: https://cloud.google.com/bigquery/quota-policy https://cloud.google.com/bigquery/quota-policy >Separately, you can always use BigQuery's federated query capability of Bigtable. Yes, and very few people understand what that means (I had to look up, and I am reasonably educated on GCP). This is not necessarily product shortcomings but packaging deficiencies that can be addressed with better product marketing.