4 ms·

GitPrime is a great exemplar of a whole class of companies that are waiting to get built: domain-specific business intelligence (BI) tools. And it’s gotten radi

GitPrime is a great exemplar of a whole class of companies that are waiting to get built: domain-specific business intelligence (BI) tools. And it’s gotten radically easier to build something like GitPrime even in the last year. You can use an automated data pipeline like Fivetran (my company) to centralize all your users data in a single multi-tenant Snowflake data warehouse, and then build your analytics on top of that foundation. Finally, you can sell direct access to the curated data via data sharing as an enterprise feature. It’s a highly repeatable pattern. There should be a GitPrime for every industry.

grantlmiller 7y ago
Generally agree on your broad point, but not on the architecture. I'm biased, but I would suggest that if you follow GitPrime's lead, you architect with OSS components that you can deploy on-prem as it was crucial to their success: https://blog.replicated.com/gitprime-enterprise-saas/ https://blog.replicated.com/gitprime-enterprise-saas/
- georgewfraser 7y ago
  Trying to build a vertical analytics offering on top of OSS increases the level of difficulty by 100x. I’m not saying that it’s never the right answer, but I am saying there’s a large category of companies waiting to be built where the primary value is the expertise around how to analyze the data, not the infrastructure. If you’re analyzing data that comes from cloud data sources anyway, there’s really no sense in “deploying”. CIOs ask for it, you say no, you do the security compliances and eventually they come around.
  - grantlmiller 7y ago
    "If you’re analyzing data that comes from cloud data sources anyway, there’s really no sense in “deploying”." - This is only true if the security controls that your team, application, infrastructure has in place is matches the major cloud providers (i.e. Salesforce, Google, AWS, Microsoft). Even then, spreading your data around to 1,000 different SaaS vendors increases the surface area for attack/loss by 1000x. "Trying to build a vertical analytics offering on top of OSS increases the level of difficulty by 100x" - 100x is hyperbole, it significantly harder before OSS was focused on operations, but now there is an HA Helm chart, or even an K8s operator for most of the popular OSS components. It might still be slightly harder today, but organizations that want to pull insights from THEIR data often value the proprietary nature of that data.
    - meritt 7y ago
      Keep in mind you're replying to the CEO of a YC W13 company that wasn't acquired two months ago for $150M by Google (that was his competitor, Alooma [1]) and wasn't acquired in November for $60M by Talend (that was Stitch Data, another competitor [2]). Your points are absolutely valid, you're just barking up the wrong tree. Both of those were SaaS-based ETL connector companies that were acquired while executing a similar enough strategy, and they both saw nice exits. If you look at his comment history, you'll notice he mentions Snowflake at every single opportunity. Snowflake raised $450M last October (~$1B valuation), so they have a nice warchest for strategic acquisitions. [1] https://techcrunch.com/2019/02/19/google-acquires-cloud-migration-platform-alooma/ https://techcrunch.com/2019/02/19/google-acquires-cloud-migr... [2] https://technical.ly/philly/2018/11/07/stitch-acquired-by-talend/ https://technical.ly/philly/2018/11/07/stitch-acquired-by-ta...
    - georgewfraser 7y ago
      I stand by 100x in this context. OSS is way behind the commercial analytical databases. And deploying a horizontally scaleable analytical database is a completely different ballgame than, say, Postgres.
      - babuskov 7y ago
        I don't have much experience with the exact topic, but anyone throwing phrases like "100x more difficult" very quickly loses credence with people. How do you measure 100x? How come it isn't 90x or 112x? Does that mean what one engineer can do with a commercial database, you would need 100 people do the the same thing with OSS in the same time frame? If you used a phrase like "difficulty on another level" or something descriptive like that instead, I'm sure people would be more interested to hear what you have to say.
cosmie 7y ago
While highly repeatable, the biggest issue I've seen with this pattern is actually the data pipeline component. Specifically: - The pricing structures don't tend to be amenable to this operating model. Most of the newer SaaS ETL services like Fivetran and Alooma tend to structure their pricing discrimination/tiering to favor a higher volume of throughput through a low number of individual connectors/sources. At least the last time I was sourcing one of these solutions, trying to follow the above pattern (a high volume of individual connections with potentially low volume of events from each source) was almost impossible to negotiate without a substantial increase in cost for any given volume of events. - The pricing structures also tend to be unfriendly to unpredictable and unknown event volume, particularly at the pre-launch phase. While that's not a problem if you're aiming towards the enterprise space or have investment capital to burn, it's a non-starter for people looking to bootstrap an idea that may or may not gain any market adoption. Industry- and domain-specific BI tools are inherently niche, and tend to have a market discovery phase as you align with that industry. Committing to minimal annual event volume for your data pipelines before you've even attempted to go to market pushes you towards rolling your own pipelines and away from using any of the tools like Fivetran that would have simplified the go to market process in the first place. - Data security and legal compliance. It's quite possibly more of an issue with my current environment and client pool, which skews more towards the enterprise. But my current job is infinitely more difficult because leveraging SaaS-based ETL tools like Fivetran, Alooma, Funnel.io, Supermetrics, etc are all absolute non-starters. Between GDPR, the new California privacy laws, and all of the press on data leaks over the last few years, any solution that includes "sending internal data to external companies" or "providing credentials for internal data sources to external parties" triggers instant lawyer and IT security review. And that includes going all the way down the rabbit hole - if we try to use a vendor, then we need to know and audit who all they give access to, etc. Which basically means we have to roll our own pipelines and keep them internal or on a client's own cloud account. At least for a lot of interesting use cases for domain-specific BI tools, I could see that same caginess coming into play.
- georgewfraser 7y ago
  I can’t speak to the others, but Fivetran pricing is based on connectors, not volume, and we have a “powered by Fivetran” price sheet for multitenant scenarios. The security reviews when there are 3 parties are a pain. It can be overcome, the benefit of getting the data infrastructure automated just has to justify the extra sales effort.
hsikka 7y ago
This is really cool, I've been thinking about your comment for the past 15 minutes. I can imagine a swath of these BI tools, and also something that could be used in a consumer setting to manage your finances, health/exercise, mood, and more. Sort of like a personalOS.
dalbasal 7y ago
What a fantastic, practical HN comment. Care to list out some domains?

grantlmiller 7y ago

Generally agree on your broad point, but not on the architecture. I'm biased, but I would suggest that if you follow GitPrime's lead, you architect with OSS components that you can deploy on-prem as it was crucial to their success: https://blog.replicated.com/gitprime-enterprise-saas/ https://blog.replicated.com/gitprime-enterprise-saas/

georgewfraser 7y ago
Trying to build a vertical analytics offering on top of OSS increases the level of difficulty by 100x. I’m not saying that it’s never the right answer, but I am saying there’s a large category of companies waiting to be built where the primary value is the expertise around how to analyze the data, not the infrastructure. If you’re analyzing data that comes from cloud data sources anyway, there’s really no sense in “deploying”. CIOs ask for it, you say no, you do the security compliances and eventually they come around.
- grantlmiller 7y ago
  "If you’re analyzing data that comes from cloud data sources anyway, there’s really no sense in “deploying”." - This is only true if the security controls that your team, application, infrastructure has in place is matches the major cloud providers (i.e. Salesforce, Google, AWS, Microsoft). Even then, spreading your data around to 1,000 different SaaS vendors increases the surface area for attack/loss by 1000x. "Trying to build a vertical analytics offering on top of OSS increases the level of difficulty by 100x" - 100x is hyperbole, it significantly harder before OSS was focused on operations, but now there is an HA Helm chart, or even an K8s operator for most of the popular OSS components. It might still be slightly harder today, but organizations that want to pull insights from THEIR data often value the proprietary nature of that data.
  - meritt 7y ago
    Keep in mind you're replying to the CEO of a YC W13 company that wasn't acquired two months ago for $150M by Google (that was his competitor, Alooma [1]) and wasn't acquired in November for $60M by Talend (that was Stitch Data, another competitor [2]). Your points are absolutely valid, you're just barking up the wrong tree. Both of those were SaaS-based ETL connector companies that were acquired while executing a similar enough strategy, and they both saw nice exits. If you look at his comment history, you'll notice he mentions Snowflake at every single opportunity. Snowflake raised $450M last October (~$1B valuation), so they have a nice warchest for strategic acquisitions. [1] https://techcrunch.com/2019/02/19/google-acquires-cloud-migration-platform-alooma/ https://techcrunch.com/2019/02/19/google-acquires-cloud-migr... [2] https://technical.ly/philly/2018/11/07/stitch-acquired-by-talend/ https://technical.ly/philly/2018/11/07/stitch-acquired-by-ta...
  - georgewfraser 7y ago
    I stand by 100x in this context. OSS is way behind the commercial analytical databases. And deploying a horizontally scaleable analytical database is a completely different ballgame than, say, Postgres.
    - babuskov 7y ago
      I don't have much experience with the exact topic, but anyone throwing phrases like "100x more difficult" very quickly loses credence with people. How do you measure 100x? How come it isn't 90x or 112x? Does that mean what one engineer can do with a commercial database, you would need 100 people do the the same thing with OSS in the same time frame? If you used a phrase like "difficulty on another level" or something descriptive like that instead, I'm sure people would be more interested to hear what you have to say.

cosmie 7y ago

While highly repeatable, the biggest issue I've seen with this pattern is actually the data pipeline component. Specifically: - The pricing structures don't tend to be amenable to this operating model. Most of the newer SaaS ETL services like Fivetran and Alooma tend to structure their pricing discrimination/tiering to favor a higher volume of throughput through a low number of individual connectors/sources. At least the last time I was sourcing one of these solutions, trying to follow the above pattern (a high volume of individual connections with potentially low volume of events from each source) was almost impossible to negotiate without a substantial increase in cost for any given volume of events. - The pricing structures also tend to be unfriendly to unpredictable and unknown event volume, particularly at the pre-launch phase. While that's not a problem if you're aiming towards the enterprise space or have investment capital to burn, it's a non-starter for people looking to bootstrap an idea that may or may not gain any market adoption. Industry- and domain-specific BI tools are inherently niche, and tend to have a market discovery phase as you align with that industry. Committing to minimal annual event volume for your data pipelines before you've even attempted to go to market pushes you towards rolling your own pipelines and away from using any of the tools like Fivetran that would have simplified the go to market process in the first place. - Data security and legal compliance. It's quite possibly more of an issue with my current environment and client pool, which skews more towards the enterprise. But my current job is infinitely more difficult because leveraging SaaS-based ETL tools like Fivetran, Alooma, Funnel.io, Supermetrics, etc are all absolute non-starters. Between GDPR, the new California privacy laws, and all of the press on data leaks over the last few years, any solution that includes "sending internal data to external companies" or "providing credentials for internal data sources to external parties" triggers instant lawyer and IT security review. And that includes going all the way down the rabbit hole - if we try to use a vendor, then we need to know and audit who all they give access to, etc. Which basically means we have to roll our own pipelines and keep them internal or on a client's own cloud account. At least for a lot of interesting use cases for domain-specific BI tools, I could see that same caginess coming into play.

georgewfraser 7y ago
I can’t speak to the others, but Fivetran pricing is based on connectors, not volume, and we have a “powered by Fivetran” price sheet for multitenant scenarios. The security reviews when there are 3 parties are a pain. It can be overcome, the benefit of getting the data infrastructure automated just has to justify the extra sales effort.

hsikka 7y ago

This is really cool, I've been thinking about your comment for the past 15 minutes. I can imagine a swath of these BI tools, and also something that could be used in a consumer setting to manage your finances, health/exercise, mood, and more. Sort of like a personalOS.

dalbasal 7y ago

What a fantastic, practical HN comment. Care to list out some domains?