3 ms·

I run a data team. We do two things: * Provide a clean minimal view of the data that other teams can pivot table on top of or download in the BI tool. We do on

I run a data team. We do two things: * Provide a clean minimal view of the data that other teams can pivot table on top of or download in the BI tool. We do onboarding, support and all that for this data. * Provide dashboards and reports for more complicated ongoing questions that require data that's not cleaned yet. We don't use Looker but based on the sales demo it seems optimized for this type of workflow. The core analytics team maintains the data and complex workflows while other teams have business analysts for day to day questions.

haddr 5y ago
This is the direction where most BI work is heading for: you need a team of data engineers that can make sure that the data for your BI dashboards is delivered and in a good quality. Everything else that doesn’t need it is probably simple enough to be run as some PowerBI dashboard on top of production DB. I think today most effort in those heavier BI cases is spent on data transformations and making sure that data flows/batch jobs run uninterrupted.
- dikei 5y ago
  If you're not careful, the data engineers team is often overloaded with requests from the analysis team. With big data warehouse, more often than not, the fancy dashboards would not run fast enough without a dedicate pipeline that materialize data before hand. Too many useless dashboards and your data team would not have enough manpower to do anything else other than operating the existing pipelines. As a result, new dashboards become slower than old dashboards due to lack of maintenance/optimization, and everyone complain about the poor performance.
- dspillett 5y ago
  > dashboard on top of production DB A lot of what users need can be done this way in my experience. Also giving their people access to customise reports/exports can go even further. I'd advise against direct access to “real” production though: have the dashboard and custom reports run against a (probably readonly) replica of some sort¹ so runaway reports don't impact live app performance. [1] via replication, log shipping, etc... — all major databases should support at least one method that can achieve this with minimal latency in updating the replica under normal conditions
  - marcinzm 5y ago
    >A lot of what users need can be done this way in my experience. In my experience, no, as the company gets to any complexity and size. The production DB is often optimized for use by the ORM/language of choice or there's twenty of them for microservices. This makes it really awkward to do some things in SQL. So now your engineering team is getting requests to keep changing the prod DB schema so analytics can be done more easily. More over the prod DB is likely 10% of the data your business needs. There's a reason companies like Fivetran exist to slurp fifty data sources into a warehouse. Marketing wants google and facebook and mailchimp data. Operations wants saleforce data. Finance wants stripe data. And they want all of this tied to the inhouse data. Sure you can make reports in each of those services but then you need to tie them together and then people complain of differences in metrics and then you have a bad time. edit: Not to mention analytics on what users do. Always fun when there's a billion row table in your production db for that.
Guest42 5y ago
Having used looker, it felt rather clunky and proprietary I was generally able to create better results via power bi, ssrs, or shiny
- marcinzm 5y ago
  As I understood it the goal of Looker isn't to make the lives of people who can use something like Shiny better but rather the lives of people who use pivot tables 99% of the time. So rather than having technical analysts making dashboards you have business analysts making dashboards (or at least digging into generic dashboards).

haddr 5y ago

This is the direction where most BI work is heading for: you need a team of data engineers that can make sure that the data for your BI dashboards is delivered and in a good quality. Everything else that doesn’t need it is probably simple enough to be run as some PowerBI dashboard on top of production DB. I think today most effort in those heavier BI cases is spent on data transformations and making sure that data flows/batch jobs run uninterrupted.

dikei 5y ago
If you're not careful, the data engineers team is often overloaded with requests from the analysis team. With big data warehouse, more often than not, the fancy dashboards would not run fast enough without a dedicate pipeline that materialize data before hand. Too many useless dashboards and your data team would not have enough manpower to do anything else other than operating the existing pipelines. As a result, new dashboards become slower than old dashboards due to lack of maintenance/optimization, and everyone complain about the poor performance.
dspillett 5y ago
> dashboard on top of production DB A lot of what users need can be done this way in my experience. Also giving their people access to customise reports/exports can go even further. I'd advise against direct access to “real” production though: have the dashboard and custom reports run against a (probably readonly) replica of some sort¹ so runaway reports don't impact live app performance. [1] via replication, log shipping, etc... — all major databases should support at least one method that can achieve this with minimal latency in updating the replica under normal conditions
- marcinzm 5y ago
  >A lot of what users need can be done this way in my experience. In my experience, no, as the company gets to any complexity and size. The production DB is often optimized for use by the ORM/language of choice or there's twenty of them for microservices. This makes it really awkward to do some things in SQL. So now your engineering team is getting requests to keep changing the prod DB schema so analytics can be done more easily. More over the prod DB is likely 10% of the data your business needs. There's a reason companies like Fivetran exist to slurp fifty data sources into a warehouse. Marketing wants google and facebook and mailchimp data. Operations wants saleforce data. Finance wants stripe data. And they want all of this tied to the inhouse data. Sure you can make reports in each of those services but then you need to tie them together and then people complain of differences in metrics and then you have a bad time. edit: Not to mention analytics on what users do. Always fun when there's a billion row table in your production db for that.

Guest42 5y ago

Having used looker, it felt rather clunky and proprietary I was generally able to create better results via power bi, ssrs, or shiny

marcinzm 5y ago
As I understood it the goal of Looker isn't to make the lives of people who can use something like Shiny better but rather the lives of people who use pivot tables 99% of the time. So rather than having technical analysts making dashboards you have business analysts making dashboards (or at least digging into generic dashboards).