4 ms·

Any individual connection is simple. Website to Mixpanel? Easy. Mobile app to Facebook App Events? Also simple. Both website and mobile app to warehouse? Bigger

by pkrein 9y ago

Any individual connection is simple. Website to Mixpanel? Easy. Mobile app to Facebook App Events? Also simple. Both website and mobile app to warehouse? Bigger investment but still doable. Data out of Stripe and Zendesk into the warehouse? Also reasonable straight forward to build, more annoying to maintain, very annoying to scale. The issue is that companies need all of this together. And they need it now. Many enterprises have 100+ business units each using 50+ tools and all of that needs to be connected together. So the issue isn't that building a single connection is hard. It's that managing N interconnected things is N^2 hard, and N^2 gets big very very fast. Wrangling that mess is just way too complicated for an open source project... even though analytics.js is open source, the vast, vast majority of a.js users only use it through Segment.

irq11 9y ago
I've personally built all of the things you're talking about, on real products, with lots of traffic, and it just isn't that hard. Annoying, yes, but arguably a core competency of any data team, and something you want to think very carefully about outsourcing. Also, most teams won't have to "scale" many of the things you're talking about...that's a problem that you only have to solve when you're the central point of failure for lots of companies. As for the n^2 problem, yes, I can see that argument, but in my experience, n is always a small number (i.e. 2-10), and they rarely need to cross-communicate in a fully connected graph. In short, piping everything through an expensive, centralized point-of-failure run by strangers seems like such a risky proposition that I can't imagine many companies would have the integration complexity to justify the risks. But again, people seem to be finding value in it, so maybe I'm missing something. I was honestly expecting you to tell me something about advanced data warehousing or integrated analytics tools or something like that. If you tell me that integrating with segment automatically starts correlating session data across providers, well...that starts to be compelling.
- pclark 9y ago
  Go tell a marketer that if they use Segment they can onboard new marketing tools and it'll 'just work' without any engineering time, and they'll shit a brick. (the data warehouse part is imho folly.)
  - irq11 9y ago
    Yes, I see this value. But again, that's something that could rather easily exist as an open-source, self-hosted tool, and I kind of wonder why the middleman hasn't been optimized away. We're in a world where people would rather self-host github to save a few bucks...this is an easier problem.
    - pclark 9y ago
      open source is basically dead courtesy of SaaS. people would much rather pay a few bucks than self-host. hell, there are entire successful companies that 'just' host open source software and wrap it up in a SaaS experience.
- hilbertseries 9y ago
  You're just too technical and think everyone is on your level. I work for a mobile analytics provider and we once sent a json log lines file format to a customer, who replied that they didn't know how to parse it and asked us for CSV instead. So, if that's the caliber of developer you have, ask again how easy everything you just talked about is.
  - irq11 9y ago
    I'm open to the argument that most development teams are utterly incompetent. I'm just somewhat skeptical of that argument.

by pkrein 9y ago

irq11 9y ago

I've personally built all of the things you're talking about, on real products, with lots of traffic, and it just isn't that hard. Annoying, yes, but arguably a core competency of any data team, and something you want to think very carefully about outsourcing. Also, most teams won't have to "scale" many of the things you're talking about...that's a problem that you only have to solve when you're the central point of failure for lots of companies. As for the n^2 problem, yes, I can see that argument, but in my experience, n is always a small number (i.e. 2-10), and they rarely need to cross-communicate in a fully connected graph. In short, piping everything through an expensive, centralized point-of-failure run by strangers seems like such a risky proposition that I can't imagine many companies would have the integration complexity to justify the risks. But again, people seem to be finding value in it, so maybe I'm missing something. I was honestly expecting you to tell me something about advanced data warehousing or integrated analytics tools or something like that. If you tell me that integrating with segment automatically starts correlating session data across providers, well...that starts to be compelling.

pclark 9y ago
Go tell a marketer that if they use Segment they can onboard new marketing tools and it'll 'just work' without any engineering time, and they'll shit a brick. (the data warehouse part is imho folly.)
- irq11 9y ago
  Yes, I see this value. But again, that's something that could rather easily exist as an open-source, self-hosted tool, and I kind of wonder why the middleman hasn't been optimized away. We're in a world where people would rather self-host github to save a few bucks...this is an easier problem.
  - pclark 9y ago
    open source is basically dead courtesy of SaaS. people would much rather pay a few bucks than self-host. hell, there are entire successful companies that 'just' host open source software and wrap it up in a SaaS experience.
hilbertseries 9y ago
You're just too technical and think everyone is on your level. I work for a mobile analytics provider and we once sent a json log lines file format to a customer, who replied that they didn't know how to parse it and asked us for CSV instead. So, if that's the caliber of developer you have, ask again how easy everything you just talked about is.
- irq11 9y ago
  I'm open to the argument that most development teams are utterly incompetent. I'm just somewhat skeptical of that argument.