4 ms·
Hi everyone, my name is Rick and with my co-founders Sven and Théo we are building Snowboard (https://www.snowboard.software/ https://www.snowboard.software/).
by zurfer 5y ago
Hi everyone, my name is Rick and with my co-founders Sven and Théo we are building Snowboard (https://www.snowboard.software/ https://www.snowboard.software/). Snowboard is a fast and automated data catalog for the Snowflake data cloud. It helps teams find, understand and trust their data assets.
In past jobs, I (Rick) rolled out Snowflake at a manufacturer with 11000 employees and $2B in revenue, and Théo built the data team and infrastructure at a B2B company that raised $200M. My experience introducing Snowflake was that after 2 years and more than 1000 tables and many more columns, it was hard to find data and tedious to understand it, and users would not trust the data without a lot of upfront due diligence.
We index all data and visualization assets that are connected, making it easy to find them. We profile data continuously to show what's inside, how it is changing, and make it easy to document and tag information. What helps us surface a lot of context is that we parse Snowflake's query history. This lets us provide automated data lineage and indicate data freshness and usage.
Most of our customers that see their actual data flows are surprised by the complexity they have built. Some see for the first time how their data flows work and how many redundant data assets they have created from one base table.
For now, we focus on the Snowflake ecosystem and don't support the long-tail of other database and analytics solutions. This lets us "overfit" to the needs and problems of companies that sit on this stack, e.g.: Fivetran > DBT > Snowflake > Tableau. Under the hood we leverage the flexibility of ArangoDB to offer a fast application that offers great search speed and also graph workloads e.g. for data lineage.
Our initial market is 5k Snowflake customers, but the global data warehousing market consists of 250k companies. We charge per indexed data asset. The number of data assets in companies is doubling on average every 2 years, so this market is taking off.
We offer both cloud and self-hosted. Self-hosted means a Docker Compose configuration that starts 3 Docker containers. The only data that is transmitted to our system is a license check in this case.
Your questions and comments are welcome. Thank you!
- bob33212 5y agoAre customers able to fix the problems identified by your product? I ask because I built a SaaS product that helped customers understand their data inside a legacy accounting product. Similarly it looked in the meta data on the back end of that accounting product and built visualizations. The problem we had was that customers didn't know how to or didn't want to fix the problems that our tool found. We had to hire accountants to go into the accounting tool and fix their problems. So it ended up being more of a tool for our accountants and less of a stand alone SaaS tool that customers actually used.
- zurfer 5y agoGreat question. Only because we show technical debt, does not mean it gets solved. We are still learning with our first users how to approach these things. Further down the road I can also see that consultants will find it useful to help their clients clean up their data pipelines and reduce complexity.
- lizen_one 5y agoThat is a very interesting idea. As far as I understand, you generate the information purely read only. Column statistics: Sample data randomly and calculate estimator from that. Data Lineage/usage: From the queries (and 'similar' stuff like materialized views. It seems like that currenltly have only lineage on table level, i.e. which table depends on which tables. Do you plan to implement data lineage on column level, i.e. which column depends on which column? This sounds challenging and very interesting. I tried to watch the demo video on your website, but it was too small and maximizing the video is disabled (Firefox). So I had to click through to youtube. I'm not sure if this is intended.
- zurfer 5y agoThanks for your thoughtful comment and pointing out the video problem! It should be fixed now. Column Level Usage is something we are looking into. We also have some ideas. It seems like a solvable problem but as you said challenging. :)
- tapeless_tunic 5y agoWorking as a data engineer, I can see how big of a problem this is. At my company, we waste loads of time cleaning up garbage tables, finding the right data, and duplicating work. Does snowflake have any plans of building a similar cataloging infrastructure?
- zurfer 5y agoThank you for the data point. :) To answer your question: there is multiple parts to it, I am certain they will tackle some parts like search in the future. I don't see them solving lineage or data monitoring anytime soon. Also cross-system overview (e.g. with Tableau) won't be coming.
- chocks 5y agoSnowboard looks great. Congrats on the launch! At my work we currently use this open source tool https://www.amundsen.io/amundsen/ https://www.amundsen.io/amundsen/ mostly to search schemas and tables. Curious to see the differences with Snowboard.
- zurfer 5y agoThank you! To give an opinionated position regarding Amundsen: Amundsen is a great open source project with many cool ideas. However, we found that the experience on Snowflake was quite shallow and much more insights and context on data assets are possible. E.g. data profiles, lineage or usage information are not readily available in Amundsen. You can customize and extend Amundsen to do more, but most data teams should build and innovate in other areas closer to the business. ;) Most companies use Snowflake over the Apache Spark open source stack exactly because of that. If you are interested to learn more, please reach out: rick at snowboard.software