4 ms·

This is an article from Jan 2022 when we were a company of 10, and now are a company of ~80. Worth some observations that: - We're still using Fivetran for th

This is an article from Jan 2022 when we were a company of 10, and now are a company of ~80. Worth some observations that: - We're still using Fivetran for the EL stages. Costs are much more significant than they were before and we're looking (for the high volume sources) into options like DataStream as cost savers, but it's not unmanageable. - dbt is still working great, even if we've done a lot of investment having now built a 5 person data team (BI, DA, DE) around it. - Still use Metabase but have some frustrations and are considering other options. - We no longer use Stitch :tada: There's a post that followed this on improvements we made to our setup that may be interesting: https://incident.io/blog/updated-data-stack https://incident.io/blog/updated-data-stack The OP is still full of relevant, useful information, though (imo, of course).

ancieque 3y ago
We have looked into metabase ourselves initially as well but found it lacking compared to Superset. What are your thoughts on that? Note that we build dashboards for customers that end up having some complex SQL directly in Superset.
- lawrjone 3y ago
  We do very similar things ourselves: our insights product (https://incident.io/learn https://incident.io/learn) uses Metabase to power the dashboards. The data that goes into those insights can be quite complex and the queries are actually threading JSON parameters through into BigQuery SQL queries using JavaScript UDFs to power the filters in the dashboard (show incidents with these custom field values). This works pretty well with signed Metabase dashboard links. We have hit limitations with Metabase though. Performance of the instance can be a bit unpredictable and their support has been poor when things do go wrong, with very little willingness to take our feedback into account for new product features. For that reason and more (such as more flexible dashboards) we’re going to move ourselves to Omni (https://omni.co/ https://omni.co/) for internal business analytics use cases, and will reconsider Metabase for our customer facing product dashboards when we do. Omni may work for these or we might build them bespoke, we’ll see at the time.
  - Cyphase 3y ago
    Do you have any thoughts on Superset? Did you consider it as a candidate? For anyone who doesn't know: https://superset.apache.org/ https://superset.apache.org/ (There's at least one service that offers managed Superset hosting if that's what you're looking for; it's easy to find so I won't link it here.)
    - code_biologist 3y ago
      I recently ran a little shootout between Superset, Metabase, and Lightdash — all open source with hosted options. All have nontrivial weaknesses but I ended up picking Lightdash. Superset the best of them at data visualization but I honestly found it almost useless for self-serve BI by business users if you have existing star schema. This issue on how to do joins in Superset (with stalebot making a mess XD) is everything difficult about Superset for BI in a nutshell. https://github.com/apache/superset/issues/8645 https://github.com/apache/superset/issues/8645 Metabase is pretty great and it's definitely the right choice for a startup looking to get low cost BI set up. It still has a very table centric view, but feels built for _BI_ rather than visualization alone. Lightdash has significant warts (YAML, pivoting being done in the frontend, no symmetric aggregates) but the Looker inspiration is obvious and it makes it easy to present _groups of tables_ to business users ready to rock. I liked Looker before Google acquired it. My business users are comfortable with star and snowflake schemas (not that they know those words) and it was easy to drop Lightdash on top of our existing data warehouse.
    - lawrjone 3y ago
      I don’t think we did consider this, probably because we have a preference to buy instead of build with tools like these and prefer a team who can respond to our feedback. It looks like a promising tool though! I’m sure we’ll blog about our experience with the new tooling once we’ve moved over, the team will no doubt have a lot to say about it.
      - Cyphase 3y ago
        I had a feeling from the article and your comments, which is why I mentioned the hosted service. :) From their website[0]: > Preset was founded by the original creator of Apache Superset™. Our team of experts contributes over 75% of all commits to the open-source software project. I'd be interested to see your blog post, regardless of tool. [0] https://preset.io/ https://preset.io/
yevpats 3y ago
Check out https://news.ycombinator.com/item?id=38812087 https://news.ycombinator.com/item?id=38812087 for high performance and high load connectors. (Maintainer here :) )
- konradb 3y ago
  This seems to be the link to this story itself?

ancieque 3y ago

We have looked into metabase ourselves initially as well but found it lacking compared to Superset. What are your thoughts on that? Note that we build dashboards for customers that end up having some complex SQL directly in Superset.

lawrjone 3y ago
We do very similar things ourselves: our insights product (https://incident.io/learn https://incident.io/learn) uses Metabase to power the dashboards. The data that goes into those insights can be quite complex and the queries are actually threading JSON parameters through into BigQuery SQL queries using JavaScript UDFs to power the filters in the dashboard (show incidents with these custom field values). This works pretty well with signed Metabase dashboard links. We have hit limitations with Metabase though. Performance of the instance can be a bit unpredictable and their support has been poor when things do go wrong, with very little willingness to take our feedback into account for new product features. For that reason and more (such as more flexible dashboards) we’re going to move ourselves to Omni (https://omni.co/ https://omni.co/) for internal business analytics use cases, and will reconsider Metabase for our customer facing product dashboards when we do. Omni may work for these or we might build them bespoke, we’ll see at the time.
- Cyphase 3y ago
  Do you have any thoughts on Superset? Did you consider it as a candidate? For anyone who doesn't know: https://superset.apache.org/ https://superset.apache.org/ (There's at least one service that offers managed Superset hosting if that's what you're looking for; it's easy to find so I won't link it here.)
  - code_biologist 3y ago
    I recently ran a little shootout between Superset, Metabase, and Lightdash — all open source with hosted options. All have nontrivial weaknesses but I ended up picking Lightdash. Superset the best of them at data visualization but I honestly found it almost useless for self-serve BI by business users if you have existing star schema. This issue on how to do joins in Superset (with stalebot making a mess XD) is everything difficult about Superset for BI in a nutshell. https://github.com/apache/superset/issues/8645 https://github.com/apache/superset/issues/8645 Metabase is pretty great and it's definitely the right choice for a startup looking to get low cost BI set up. It still has a very table centric view, but feels built for _BI_ rather than visualization alone. Lightdash has significant warts (YAML, pivoting being done in the frontend, no symmetric aggregates) but the Looker inspiration is obvious and it makes it easy to present _groups of tables_ to business users ready to rock. I liked Looker before Google acquired it. My business users are comfortable with star and snowflake schemas (not that they know those words) and it was easy to drop Lightdash on top of our existing data warehouse.
  - lawrjone 3y ago
    I don’t think we did consider this, probably because we have a preference to buy instead of build with tools like these and prefer a team who can respond to our feedback. It looks like a promising tool though! I’m sure we’ll blog about our experience with the new tooling once we’ve moved over, the team will no doubt have a lot to say about it.
    - Cyphase 3y ago
      I had a feeling from the article and your comments, which is why I mentioned the hosted service. :) From their website[0]: > Preset was founded by the original creator of Apache Superset™. Our team of experts contributes over 75% of all commits to the open-source software project. I'd be interested to see your blog post, regardless of tool. [0] https://preset.io/ https://preset.io/

yevpats 3y ago

Check out https://news.ycombinator.com/item?id=38812087 https://news.ycombinator.com/item?id=38812087 for high performance and high load connectors. (Maintainer here :) )

konradb 3y ago
This seems to be the link to this story itself?