3 ms·

Just yesterday I discussed a blog post that I have on my mind, with somebody from dbt, on the rise of the open source analytics stack. there's a bunch of great

Just yesterday I discussed a blog post that I have on my mind, with somebody from dbt, on the rise of the open source analytics stack. there's a bunch of great hosted tools out there, across ETL, workflows, dashboards, etc. Think Fivetran, Segment, Matillion, Periscope, etc. And then of course the warehouses like Snowflake, Redshift, etc. But I think there are three issues with that stack, roughly like this (I've got to do some more thinking, would appreciate your input): - Privacy: you have your customer data flying around in all these different tools. it's hard to impossible to track your compliance - cost: all these vendors charge in some way by data volume, MAUs, etc. - you get taxed multiple times for the same data stream. It all adds up. - control: your data is subject to pre-determined schemas, proprietary formats, black boxes, etc. - mismatch for the same metric across different tools, less flexibility to manipulate your data and pack up and go elsewhere. I think there's a valid open source alternative for every layer of the stack: Segment --> Rudder Labs, Snowplow Matillion --> Airflow, dbt Fivetran --> Stitch / Singer Periscope, Looker, Tableau, etc. --> Metabase, Superset Warehouses --> just yesterday I learned about materialize.io here on HN And then add open source products like PostHog, that add additional value for very specific use cases (in this case product analytics). Not arguing the value of the hosted products. They are amazing to use if you just get started. But there's a great open source "stack" available that long term likely will be more transparent, more flexible, and cheaper. Would love your thoughts!

curo 7y ago
For many projects I've often looked for easy solutions that would handle exposing data connectors to the end user of a project. Somewhat like an embedded Zapier with community maintained connectors (e.g., for the end user to grab SalesForce data and sync it with your account). I suppose Singer might be the closest thing from your list above, but still you'd have to build out a large amount of auth & end user tooling to get it to work. Every B2B SaaS developer these days has to build in a ton of integrations. Even an affordable hosting service for this would work well (embedded integration where you don't require your customer to sign up or pay for a second product).
memn0nis 7y ago
The interesting part to me here is that all the non-open source tools are very big, successful companies
- james_impliu 7y ago
  The business model for OS has been really tough historically, which probably has made raising money harder, so growing as rapidly as the SAAS companies therefore really hard. It's quickly becoming clear (GitLab/Mattermost/Sentry) that there are some great ways to build enormous companies like this though. And, that's of course assuming you want to build a huge company :) At a personal level, we found that this kind of business is just more fun to build... making cool stuff in the open and if we do a good job, getting inbound interest from bigger companies that have developers who need to use our tech at scale. I don't think it'd work for everything - if you are a tool that developers don't interact with much, then I'd imagine it's tougher to build a real community.
  - capkutay 7y ago
    There's also some innate complications with the nature of a company adopting an open source tool. it's classic 'build vs buy', so if a company is investing in engineers building their own in-house solutions on top of open source tools, the additional cost of licensing support/additional tools doesn't really make sense. so there's immediate friction to monetize open source users. on the other hand, SaaS products are optimized to sell to business groups who want the hard parts taken care of for them and they perceive value in saving time/money thanks to the SaaS product. I've seen both situations (monetizing open source versus monetizing a SaaS product) first hand, and it's clear that open source companies can be at a bit of a disadvantage. If they DO monetize their users, it's usually via their own SaaS offering to augment the open source tool.
soumyadeb 7y ago
Founder of RudderLabs (RudderStack now) here. You nailed it - Control, Cost & Privacy (Data-Ownership) are the main drivers we have seen in people adopting us. Kind of related to Control is the whole drive to adopting open-Source technologies. We engineers always wanted open-source but increasingly larger enterprises are interested too - maybe to avoid vendor locking or maybe just because cool engineers (who grew up adopting open-source stuff) are finally getting into decision making positions. Shameless plug - we wrote a blog on setting up an Open Source Analytics Stack with one of our deployments highlighting these issues. https://dev.to/rudderstack/how-to-build-an-open-source-analytics-stack-using-rudderstack-and-apache-superset-14h2 https://dev.to/rudderstack/how-to-build-an-open-source-analy... PostHog looks awesome!! Congrats on your launch. Would love to collaborate and share notes.
- dreamer7 7y ago
  It's really unfortunate how HN culture (awesome in most respects) forces you to use defensive phrases like "shameless plug" for sharing work that you are really proud of and the community is proud of as well
  - dang 7y ago
    That's not an HN thing so much as an internet thing: https://www.google.com/search?ei=95VPXs6rG8_4-gSVzZuIBw&q=%22shameless+plug%22 https://www.google.com/search?ei=95VPXs6rG8_4-gSVzZuIBw&q=%2.... I wouldn't take the self-deprecating language literally. It's probably a useful signal. It communicates "I'm a genuine community member sharing my work with the community" in a way that is hard for spammers to replicate.
lapnitnelav 7y ago
Completely agree with you there, I think it's especially true for an often overlooked yet massive subset of the businesses : the smaller players. They tend to have neither the resources nor knowledge that there's a world out there beyond the regular free to use but you pay with everybody's data and consolidate their power big corp (FB, Google, ...) I think the way forward would be something akin to CloudFormation with presets on your own server (DO / Scaleway / AWS / ...), something managed over which you still have ownership and can plug off. My axis is particularly around marketing but I suppose you could expand. PS : I would add Redash to the list of BI tools.

curo 7y ago

For many projects I've often looked for easy solutions that would handle exposing data connectors to the end user of a project. Somewhat like an embedded Zapier with community maintained connectors (e.g., for the end user to grab SalesForce data and sync it with your account). I suppose Singer might be the closest thing from your list above, but still you'd have to build out a large amount of auth & end user tooling to get it to work. Every B2B SaaS developer these days has to build in a ton of integrations. Even an affordable hosting service for this would work well (embedded integration where you don't require your customer to sign up or pay for a second product).

memn0nis 7y ago

The interesting part to me here is that all the non-open source tools are very big, successful companies

james_impliu 7y ago
The business model for OS has been really tough historically, which probably has made raising money harder, so growing as rapidly as the SAAS companies therefore really hard. It's quickly becoming clear (GitLab/Mattermost/Sentry) that there are some great ways to build enormous companies like this though. And, that's of course assuming you want to build a huge company :) At a personal level, we found that this kind of business is just more fun to build... making cool stuff in the open and if we do a good job, getting inbound interest from bigger companies that have developers who need to use our tech at scale. I don't think it'd work for everything - if you are a tool that developers don't interact with much, then I'd imagine it's tougher to build a real community.
- capkutay 7y ago
  There's also some innate complications with the nature of a company adopting an open source tool. it's classic 'build vs buy', so if a company is investing in engineers building their own in-house solutions on top of open source tools, the additional cost of licensing support/additional tools doesn't really make sense. so there's immediate friction to monetize open source users. on the other hand, SaaS products are optimized to sell to business groups who want the hard parts taken care of for them and they perceive value in saving time/money thanks to the SaaS product. I've seen both situations (monetizing open source versus monetizing a SaaS product) first hand, and it's clear that open source companies can be at a bit of a disadvantage. If they DO monetize their users, it's usually via their own SaaS offering to augment the open source tool.

soumyadeb 7y ago

Founder of RudderLabs (RudderStack now) here. You nailed it - Control, Cost & Privacy (Data-Ownership) are the main drivers we have seen in people adopting us. Kind of related to Control is the whole drive to adopting open-Source technologies. We engineers always wanted open-source but increasingly larger enterprises are interested too - maybe to avoid vendor locking or maybe just because cool engineers (who grew up adopting open-source stuff) are finally getting into decision making positions. Shameless plug - we wrote a blog on setting up an Open Source Analytics Stack with one of our deployments highlighting these issues. https://dev.to/rudderstack/how-to-build-an-open-source-analytics-stack-using-rudderstack-and-apache-superset-14h2 https://dev.to/rudderstack/how-to-build-an-open-source-analy... PostHog looks awesome!! Congrats on your launch. Would love to collaborate and share notes.

dreamer7 7y ago
It's really unfortunate how HN culture (awesome in most respects) forces you to use defensive phrases like "shameless plug" for sharing work that you are really proud of and the community is proud of as well
- dang 7y ago
  That's not an HN thing so much as an internet thing: https://www.google.com/search?ei=95VPXs6rG8_4-gSVzZuIBw&q=%22shameless+plug%22 https://www.google.com/search?ei=95VPXs6rG8_4-gSVzZuIBw&q=%2.... I wouldn't take the self-deprecating language literally. It's probably a useful signal. It communicates "I'm a genuine community member sharing my work with the community" in a way that is hard for spammers to replicate.

lapnitnelav 7y ago

Completely agree with you there, I think it's especially true for an often overlooked yet massive subset of the businesses : the smaller players. They tend to have neither the resources nor knowledge that there's a world out there beyond the regular free to use but you pay with everybody's data and consolidate their power big corp (FB, Google, ...) I think the way forward would be something akin to CloudFormation with presets on your own server (DO / Scaleway / AWS / ...), something managed over which you still have ownership and can plug off. My axis is particularly around marketing but I suppose you could expand. PS : I would add Redash to the list of BI tools.