4 ms·

Latency can be more or less questionable depending on when the stopwatch button is pressed. For example, in VR, the gold standard is "motion to photon" -- anyth

Latency can be more or less questionable depending on when the stopwatch button is pressed. For example, in VR, the gold standard is "motion to photon" -- anything else is just not representative of what actual latency means for actual users (i.e. getting motion sick). There are many definitions of latency, so it's important to choose the right one (the one that matters for your users). For us, latency means the time between when the _event_ actually occurs in real life (user clicks on a button) and when it showed up in all of our analytics integrations (Mixpanel, Intercom, etc.). As developers, this is the definition of "latency" that really matters for us because it represents the minimum time before we can take action on our users' behaviors. The reason I'm very skeptical is because what we were actually seeing was orders-of-magnitude differences from the 44 ms "latency" claimed on your status graph and what we were measuring ourselves (~1-10 hours for us) -- not just 10% differences which would have been more than acceptable for our use case. For example, Intercom support were able to identify a four hour delay from when the /identify actually took place and when Segment sent the event to Intercom. Segment Support then mentioned a race condition with their Intercom integration that they had yet to solve (we ran into this ourselves -- we had to guarantee we called /identify BEFORE calling any /track calls). With Mixpanel, not only were we experiencing multi-hour delays, we also experienced many nonsensically mis-ordered events when using Segment (even with timestamping -- it turned out that Mixpanel still relied on events being ordered properly within a 2-minute window -- something we were only able to guarantee by directly sending events to Mixpanel ourselves). Segment's response: "Our infrastructure is built on top of a queue system that accommodates high scalability. The drawback to this is that sometimes events are received by our API in a mixed up order because we have multiple queues feeding events to multiple workers." These problems were all resolved immediately when we started issuing direct calls to our various analytics providers.

calvinfo 9y ago
Hey, I’m one of the Segment founders. I helped instrument a lot of these metrics and built the status page Peter linked to above. I’m hoping I can help shed some light on some of those numbers. The 44ms you’re referring to is the time it takes our API to respond to an incoming call and ingest an event. It’s certainly not the wall clock time, but it is a good measure of the overall health of the system. Your feedback about its prominence is definitely good–it’s our goal to be transparent, not misleading. We’ll change the area where it’s displayed shortly. If you’re looking for the ‘end-to-end’ wall clock time, you can find that a little further down the page. For every event coming through our pipeline, we average the time from ingestion to successful delivery and display those metrics on a per-destination basis. You can see right now that the Google Analytics end-to-end delivery latency is ~400ms within the past hour, and is pretty consistently near that number. We’ve also developed internal systems which break down exactly how many events from a given source were delivered to a given destination, and what the latency distribution for those events was. The ordering problem you mention is indeed tricky. Like TCP, if we wanted to keep a loose ordering, we’d have to keep a window and ensure the partner API would then re-order messages appropriately. If you want a total ordering, this window of delivery _has_ to be 1 for a given user. It does have some pretty serious implications on the throughput of the system, so we’ve been working with both Mixpanel and Intercom (we’re users ourselves) to try and solve the issue just with timestamps. Ideally, partners would be able to re-order events received based upon time, which is what we do inside of customers’ data warehouses. As far as the deliverability issues you mentioned, I’m terribly sorry to hear that we failed you here. We’ve hit some scaling bottlenecks that we’ve been working hard to fix–and we do our best to keep the status page updated whenever we have production incidents. All that said, reliability is our top focus as a company. Teams present their SLA metrics on a weekly basis at all hands, and it’s a key part of our monthly board reporting. We’ll be surfacing these metrics and event traces inside the webapp so as a customer can see exactly where your data is, and what has been delivered. And we’re in the process of building an entirely revamped pipeline that will provide better deliverability guarantees. We plan on sharing the architecture on the blog once it’s all rolled out. Giving you transparency into where your data is stored and how it is processed is exactly what we want to achieve as a company. If there’s anything I missed–please reach out! I’m calvin at segment.
- deleted 9y ago
  [deleted]
ktamura 9y ago
>For us, latency means the time between when the _event_ actually occurs in real life (user clicks on a button) and when it showed up in all of our analytics integrations (Mixpanel, Intercom, etc.). As developers, this is the definition of "latency" that really matters for us because it represents the minimum time before we can take action on our users' behaviors. This. I call it "end-to-end latency" and it doesn't get nearly as much attention as it should. This is, by the way, why it's a bad idea to use most OLAPs (like Redshift) as a backend for Segment if you want "real-time" analytics. Column-oriented OLAPs are not designed for real-time ingest because either they don't support streaming inserts (Redshift) or it's not particularly reliable (BigQuery).
- vgt 9y ago
  Can you please elaborate on why you think BigQuery's streaming inserts are not particularly reliable? Separately, you can always use BigQuery's federated query capability of Bigtable. (work at G)
  - ktamura 9y ago
    >Can you please elaborate on why you think BigQuery's streaming inserts are not particularly reliable? I've heard this from many users through my work on Fluentd (in fact, many end up using fluent-plugin-bigquery because they don't want to deal with the minutiae of insertId and whatnot on their own). Also, I don't think a lot of people fully understand the quota policy and get tripped up: https://cloud.google.com/bigquery/quota-policy https://cloud.google.com/bigquery/quota-policy >Separately, you can always use BigQuery's federated query capability of Bigtable. Yes, and very few people understand what that means (I had to look up, and I am reasonably educated on GCP). This is not necessarily product shortcomings but packaging deficiencies that can be addressed with better product marketing.

calvinfo 9y ago

Hey, I’m one of the Segment founders. I helped instrument a lot of these metrics and built the status page Peter linked to above. I’m hoping I can help shed some light on some of those numbers. The 44ms you’re referring to is the time it takes our API to respond to an incoming call and ingest an event. It’s certainly not the wall clock time, but it is a good measure of the overall health of the system. Your feedback about its prominence is definitely good–it’s our goal to be transparent, not misleading. We’ll change the area where it’s displayed shortly. If you’re looking for the ‘end-to-end’ wall clock time, you can find that a little further down the page. For every event coming through our pipeline, we average the time from ingestion to successful delivery and display those metrics on a per-destination basis. You can see right now that the Google Analytics end-to-end delivery latency is ~400ms within the past hour, and is pretty consistently near that number. We’ve also developed internal systems which break down exactly how many events from a given source were delivered to a given destination, and what the latency distribution for those events was. The ordering problem you mention is indeed tricky. Like TCP, if we wanted to keep a loose ordering, we’d have to keep a window and ensure the partner API would then re-order messages appropriately. If you want a total ordering, this window of delivery _has_ to be 1 for a given user. It does have some pretty serious implications on the throughput of the system, so we’ve been working with both Mixpanel and Intercom (we’re users ourselves) to try and solve the issue just with timestamps. Ideally, partners would be able to re-order events received based upon time, which is what we do inside of customers’ data warehouses. As far as the deliverability issues you mentioned, I’m terribly sorry to hear that we failed you here. We’ve hit some scaling bottlenecks that we’ve been working hard to fix–and we do our best to keep the status page updated whenever we have production incidents. All that said, reliability is our top focus as a company. Teams present their SLA metrics on a weekly basis at all hands, and it’s a key part of our monthly board reporting. We’ll be surfacing these metrics and event traces inside the webapp so as a customer can see exactly where your data is, and what has been delivered. And we’re in the process of building an entirely revamped pipeline that will provide better deliverability guarantees. We plan on sharing the architecture on the blog once it’s all rolled out. Giving you transparency into where your data is stored and how it is processed is exactly what we want to achieve as a company. If there’s anything I missed–please reach out! I’m calvin at segment.

deleted 9y ago
[deleted]

ktamura 9y ago

>For us, latency means the time between when the _event_ actually occurs in real life (user clicks on a button) and when it showed up in all of our analytics integrations (Mixpanel, Intercom, etc.). As developers, this is the definition of "latency" that really matters for us because it represents the minimum time before we can take action on our users' behaviors. This. I call it "end-to-end latency" and it doesn't get nearly as much attention as it should. This is, by the way, why it's a bad idea to use most OLAPs (like Redshift) as a backend for Segment if you want "real-time" analytics. Column-oriented OLAPs are not designed for real-time ingest because either they don't support streaming inserts (Redshift) or it's not particularly reliable (BigQuery).

vgt 9y ago
Can you please elaborate on why you think BigQuery's streaming inserts are not particularly reliable? Separately, you can always use BigQuery's federated query capability of Bigtable. (work at G)
- ktamura 9y ago
  >Can you please elaborate on why you think BigQuery's streaming inserts are not particularly reliable? I've heard this from many users through my work on Fluentd (in fact, many end up using fluent-plugin-bigquery because they don't want to deal with the minutiae of insertId and whatnot on their own). Also, I don't think a lot of people fully understand the quota policy and get tripped up: https://cloud.google.com/bigquery/quota-policy https://cloud.google.com/bigquery/quota-policy >Separately, you can always use BigQuery's federated query capability of Bigtable. Yes, and very few people understand what that means (I had to look up, and I am reasonably educated on GCP). This is not necessarily product shortcomings but packaging deficiencies that can be addressed with better product marketing.