6 ms·

Had nothing but problems with their product over the years. In 2012, they had a bug in their analytics.js which resulted in me losing about 10k new users. Last

Had nothing but problems with their product over the years. In 2012, they had a bug in their analytics.js which resulted in me losing about 10k new users. Last year, I was experiencing multi-day delays between an event and it appearing in Mixpanel, Intercom, etc. Prior this year, I was using Branch and then realized how many sacrifices you make when you use Segment – they will never be up to date with all the most recent SDKs/APIs, so you will always be pulling your hair out. We ended up using Segment + 4 other services, it would have been better to not use Segment at all. I hope they sort these problems out, and build a better developer product. Right now, they're focused on the enterprise, and I'm amazed their product is sufficient. I'm writing this because their team is amazing and kind, but I hope they improve their product.

pkrein 9y ago
Shoot. I'm sorry this happened. I'd love to chat more about your experience with Branch SDK/APIs recently. I found your email in another HN comment and will reach out shortly. On the other two issues... Analytics.js launched on HN in December 2012. It had some bugs in its early form, but it has certainly become a lot more stable in the five years since then. Today it runs without issue on 500m+ browsers and hundreds of thousands of websites each day. For event delivery, you can see real-time event deliverability and latency stats on our public status page: https://status.segment.com https://status.segment.com For most services we deliver data in a few hundred milliseconds, but some services consistently drop data. We retry that data over the next 12 hours, and this actually increases the deliverability to our partner endpoints. (For example, we're improving deliverability to Iron.io from 95.4% to 98.4% right now.)
- Hydraulix989 9y ago
  Latency can be more or less questionable depending on when the stopwatch button is pressed. For example, in VR, the gold standard is "motion to photon" -- anything else is just not representative of what actual latency means for actual users (i.e. getting motion sick). There are many definitions of latency, so it's important to choose the right one (the one that matters for your users). For us, latency means the time between when the _event_ actually occurs in real life (user clicks on a button) and when it showed up in all of our analytics integrations (Mixpanel, Intercom, etc.). As developers, this is the definition of "latency" that really matters for us because it represents the minimum time before we can take action on our users' behaviors. The reason I'm very skeptical is because what we were actually seeing was orders-of-magnitude differences from the 44 ms "latency" claimed on your status graph and what we were measuring ourselves (~1-10 hours for us) -- not just 10% differences which would have been more than acceptable for our use case. For example, Intercom support were able to identify a four hour delay from when the /identify actually took place and when Segment sent the event to Intercom. Segment Support then mentioned a race condition with their Intercom integration that they had yet to solve (we ran into this ourselves -- we had to guarantee we called /identify BEFORE calling any /track calls). With Mixpanel, not only were we experiencing multi-hour delays, we also experienced many nonsensically mis-ordered events when using Segment (even with timestamping -- it turned out that Mixpanel still relied on events being ordered properly within a 2-minute window -- something we were only able to guarantee by directly sending events to Mixpanel ourselves). Segment's response: "Our infrastructure is built on top of a queue system that accommodates high scalability. The drawback to this is that sometimes events are received by our API in a mixed up order because we have multiple queues feeding events to multiple workers." These problems were all resolved immediately when we started issuing direct calls to our various analytics providers.
  - calvinfo 9y ago
    Hey, I’m one of the Segment founders. I helped instrument a lot of these metrics and built the status page Peter linked to above. I’m hoping I can help shed some light on some of those numbers. The 44ms you’re referring to is the time it takes our API to respond to an incoming call and ingest an event. It’s certainly not the wall clock time, but it is a good measure of the overall health of the system. Your feedback about its prominence is definitely good–it’s our goal to be transparent, not misleading. We’ll change the area where it’s displayed shortly. If you’re looking for the ‘end-to-end’ wall clock time, you can find that a little further down the page. For every event coming through our pipeline, we average the time from ingestion to successful delivery and display those metrics on a per-destination basis. You can see right now that the Google Analytics end-to-end delivery latency is ~400ms within the past hour, and is pretty consistently near that number. We’ve also developed internal systems which break down exactly how many events from a given source were delivered to a given destination, and what the latency distribution for those events was. The ordering problem you mention is indeed tricky. Like TCP, if we wanted to keep a loose ordering, we’d have to keep a window and ensure the partner API would then re-order messages appropriately. If you want a total ordering, this window of delivery _has_ to be 1 for a given user. It does have some pretty serious implications on the throughput of the system, so we’ve been working with both Mixpanel and Intercom (we’re users ourselves) to try and solve the issue just with timestamps. Ideally, partners would be able to re-order events received based upon time, which is what we do inside of customers’ data warehouses. As far as the deliverability issues you mentioned, I’m terribly sorry to hear that we failed you here. We’ve hit some scaling bottlenecks that we’ve been working hard to fix–and we do our best to keep the status page updated whenever we have production incidents. All that said, reliability is our top focus as a company. Teams present their SLA metrics on a weekly basis at all hands, and it’s a key part of our monthly board reporting. We’ll be surfacing these metrics and event traces inside the webapp so as a customer can see exactly where your data is, and what has been delivered. And we’re in the process of building an entirely revamped pipeline that will provide better deliverability guarantees. We plan on sharing the architecture on the blog once it’s all rolled out. Giving you transparency into where your data is stored and how it is processed is exactly what we want to achieve as a company. If there’s anything I missed–please reach out! I’m calvin at segment.
andrenotgiant 9y ago
Just to offer perspective from the other side, we've been using their product since 2015, we send millions of events through Segment every day without a problem. There was one short outage I can remember in 2015, the only other issues we ever had, we traced back to misconfiguration on our side. I think Segment's core code is _really solid_, it shows in how fast a relatively small company is able to iterate, update and release new product. I think their biggest challenge is educating potential users on what they do (no exact competitor) and educating new users on how to set it all up properly (gotta read the documentation.)
- jdotjdot 9y ago
  I agree here--we've never had any availability problems like this, and using Segment has been delightful. I think judging a 6-year old startup based on what its product was one year into the company is rather unreasonable. To put this into perspective, 80% of the company's lifespan has occurred after the grandparent's negative experience. Things change dramatically in that period of time.

pkrein 9y ago

Shoot. I'm sorry this happened. I'd love to chat more about your experience with Branch SDK/APIs recently. I found your email in another HN comment and will reach out shortly. On the other two issues... Analytics.js launched on HN in December 2012. It had some bugs in its early form, but it has certainly become a lot more stable in the five years since then. Today it runs without issue on 500m+ browsers and hundreds of thousands of websites each day. For event delivery, you can see real-time event deliverability and latency stats on our public status page: https://status.segment.com https://status.segment.com For most services we deliver data in a few hundred milliseconds, but some services consistently drop data. We retry that data over the next 12 hours, and this actually increases the deliverability to our partner endpoints. (For example, we're improving deliverability to Iron.io from 95.4% to 98.4% right now.)

Hydraulix989 9y ago
Latency can be more or less questionable depending on when the stopwatch button is pressed. For example, in VR, the gold standard is "motion to photon" -- anything else is just not representative of what actual latency means for actual users (i.e. getting motion sick). There are many definitions of latency, so it's important to choose the right one (the one that matters for your users). For us, latency means the time between when the _event_ actually occurs in real life (user clicks on a button) and when it showed up in all of our analytics integrations (Mixpanel, Intercom, etc.). As developers, this is the definition of "latency" that really matters for us because it represents the minimum time before we can take action on our users' behaviors. The reason I'm very skeptical is because what we were actually seeing was orders-of-magnitude differences from the 44 ms "latency" claimed on your status graph and what we were measuring ourselves (~1-10 hours for us) -- not just 10% differences which would have been more than acceptable for our use case. For example, Intercom support were able to identify a four hour delay from when the /identify actually took place and when Segment sent the event to Intercom. Segment Support then mentioned a race condition with their Intercom integration that they had yet to solve (we ran into this ourselves -- we had to guarantee we called /identify BEFORE calling any /track calls). With Mixpanel, not only were we experiencing multi-hour delays, we also experienced many nonsensically mis-ordered events when using Segment (even with timestamping -- it turned out that Mixpanel still relied on events being ordered properly within a 2-minute window -- something we were only able to guarantee by directly sending events to Mixpanel ourselves). Segment's response: "Our infrastructure is built on top of a queue system that accommodates high scalability. The drawback to this is that sometimes events are received by our API in a mixed up order because we have multiple queues feeding events to multiple workers." These problems were all resolved immediately when we started issuing direct calls to our various analytics providers.
- calvinfo 9y ago
  Hey, I’m one of the Segment founders. I helped instrument a lot of these metrics and built the status page Peter linked to above. I’m hoping I can help shed some light on some of those numbers. The 44ms you’re referring to is the time it takes our API to respond to an incoming call and ingest an event. It’s certainly not the wall clock time, but it is a good measure of the overall health of the system. Your feedback about its prominence is definitely good–it’s our goal to be transparent, not misleading. We’ll change the area where it’s displayed shortly. If you’re looking for the ‘end-to-end’ wall clock time, you can find that a little further down the page. For every event coming through our pipeline, we average the time from ingestion to successful delivery and display those metrics on a per-destination basis. You can see right now that the Google Analytics end-to-end delivery latency is ~400ms within the past hour, and is pretty consistently near that number. We’ve also developed internal systems which break down exactly how many events from a given source were delivered to a given destination, and what the latency distribution for those events was. The ordering problem you mention is indeed tricky. Like TCP, if we wanted to keep a loose ordering, we’d have to keep a window and ensure the partner API would then re-order messages appropriately. If you want a total ordering, this window of delivery _has_ to be 1 for a given user. It does have some pretty serious implications on the throughput of the system, so we’ve been working with both Mixpanel and Intercom (we’re users ourselves) to try and solve the issue just with timestamps. Ideally, partners would be able to re-order events received based upon time, which is what we do inside of customers’ data warehouses. As far as the deliverability issues you mentioned, I’m terribly sorry to hear that we failed you here. We’ve hit some scaling bottlenecks that we’ve been working hard to fix–and we do our best to keep the status page updated whenever we have production incidents. All that said, reliability is our top focus as a company. Teams present their SLA metrics on a weekly basis at all hands, and it’s a key part of our monthly board reporting. We’ll be surfacing these metrics and event traces inside the webapp so as a customer can see exactly where your data is, and what has been delivered. And we’re in the process of building an entirely revamped pipeline that will provide better deliverability guarantees. We plan on sharing the architecture on the blog once it’s all rolled out. Giving you transparency into where your data is stored and how it is processed is exactly what we want to achieve as a company. If there’s anything I missed–please reach out! I’m calvin at segment.

andrenotgiant 9y ago

Just to offer perspective from the other side, we've been using their product since 2015, we send millions of events through Segment every day without a problem. There was one short outage I can remember in 2015, the only other issues we ever had, we traced back to misconfiguration on our side. I think Segment's core code is _really solid_, it shows in how fast a relatively small company is able to iterate, update and release new product. I think their biggest challenge is educating potential users on what they do (no exact competitor) and educating new users on how to set it all up properly (gotta read the documentation.)

jdotjdot 9y ago
I agree here--we've never had any availability problems like this, and using Segment has been delightful. I think judging a 6-year old startup based on what its product was one year into the company is rather unreasonable. To put this into perspective, 80% of the company's lifespan has occurred after the grandparent's negative experience. Things change dramatically in that period of time.