4 ms·

And if say that if you are in the position where you can run DW workloads on your prod database you probably don't need a data warehouse in the first place. Da

by Guthur 5y ago

And if say that if you are in the position where you can run DW workloads on your prod database you probably don't need a data warehouse in the first place. Data warehouse workloads tend to be very IO intensive and could be highly disruptive to a production db. ETL is hard but it's a price to pay to isolate these two very different workloads.

chii 5y ago
DW is a capability, not a scale problem. DW lets you gain insight from your available data, whether there's a lot of data, or not. DW with postgres is probably the right choice if your data isn't petabytes level.
- Guthur 5y ago
  DW is a scale problem, what you are describing is data analytics. DW has always described the storage solution not the workload.
  - inopinatus 5y ago
    From a company perspective, DW is an implementation detail for the utility it provides, which is analysis and reporting. These activities have such different lifecycles, accountabilities, and controls, to operational data stewardship, that the pattern of a BI team operating independently from the corporate IT unit may arise by governance pressures alone. Differences in storage architecture and data schema aren't considered when it comes to carving up the org chart, but they're certainly enabled by the consequences: in my experience of large-to-very-large enterprises it's common for BI teams to take practically vertical control of their entire stack, from the racks to the apps (or in public clouds, one or more separated accounts), and all the infrastructure in between, a classic sighting of Conway's law in action. More than once I've pointed at such teams as an explainer when pitching a service-team approach to standing up a new application. (Caveat, lest it backfire if you try the same, check who you're talking to first; they're not always fans of "the BI gang")
    - pm90 5y ago
      I hate to admit this, but I’m definitely one of the folks who isn’t a fan of the BI gang that operates in my workplace. They seem to have a very superficial understanding of how anything works. Their stuff constantly breaks on schema changes, and their response is never “hey, this seems like a broken workflow. How do we improve it?”. Its always “@here our DW may be broken but we will get it back up soon” and radio silence as they try to privately figure out how to fix something they don’t quite understand. Not to mention that they have hardly any monitoring/alerting, which means that the data in the DW is not reliable, which can be really frustrating for users. Overall I get the sense that these are unskilled folks who oversold what they could do but ultimately have very little clue what’s really going on.
      - jerglingu 5y ago
        A more common experience with BI teams than people are willing to confess to, especially when the business the BI team is meant to be tracking and measuring is growing faster than the team can manage. In my experiences and and observations, this phenomenon where a BI teams' processes/products are unstable is usually because of a team's lack of depth in DBA skills. Not necessarily the lack of a DBA, but the lack of any engineer with any eye or attention given towards environmental stability, optimization, and the traditional "janitorial" clean up work that can tend to be neglected in the BI space.
      - FranzFerdiNaN 5y ago
        If the BI teams stuff constantly breaks because production databases change I would say the software engineers are doing a shit job. It’s the engineers responsibility to communicate such breaking changes in advance to teams that rely on their products.
        pm90 5y ago
        I see your point, but Engineers should not have to worry about dB schema changes breaking anything. The contract they provide externally is via APIs. The BI team slurping up dB binlogs must not create an expectation of support from engineers.
- mjburgess 5y ago
  This is the problem with business people being let loose to sell ideas without the technical depth. A data warehouse is not a capability, it's a data system. Company A with a warehouse and company B without do not necessarily differ in any capabilities.
devit 5y ago
You can run the workloads on (possibly partial) replicas of the production database running on another set of machines.

by Guthur 5y ago

chii 5y ago

DW is a capability, not a scale problem. DW lets you gain insight from your available data, whether there's a lot of data, or not. DW with postgres is probably the right choice if your data isn't petabytes level.

Guthur 5y ago
DW is a scale problem, what you are describing is data analytics. DW has always described the storage solution not the workload.
- inopinatus 5y ago
  From a company perspective, DW is an implementation detail for the utility it provides, which is analysis and reporting. These activities have such different lifecycles, accountabilities, and controls, to operational data stewardship, that the pattern of a BI team operating independently from the corporate IT unit may arise by governance pressures alone. Differences in storage architecture and data schema aren't considered when it comes to carving up the org chart, but they're certainly enabled by the consequences: in my experience of large-to-very-large enterprises it's common for BI teams to take practically vertical control of their entire stack, from the racks to the apps (or in public clouds, one or more separated accounts), and all the infrastructure in between, a classic sighting of Conway's law in action. More than once I've pointed at such teams as an explainer when pitching a service-team approach to standing up a new application. (Caveat, lest it backfire if you try the same, check who you're talking to first; they're not always fans of "the BI gang")
  - pm90 5y ago
    I hate to admit this, but I’m definitely one of the folks who isn’t a fan of the BI gang that operates in my workplace. They seem to have a very superficial understanding of how anything works. Their stuff constantly breaks on schema changes, and their response is never “hey, this seems like a broken workflow. How do we improve it?”. Its always “@here our DW may be broken but we will get it back up soon” and radio silence as they try to privately figure out how to fix something they don’t quite understand. Not to mention that they have hardly any monitoring/alerting, which means that the data in the DW is not reliable, which can be really frustrating for users. Overall I get the sense that these are unskilled folks who oversold what they could do but ultimately have very little clue what’s really going on.
    - jerglingu 5y ago
      A more common experience with BI teams than people are willing to confess to, especially when the business the BI team is meant to be tracking and measuring is growing faster than the team can manage. In my experiences and and observations, this phenomenon where a BI teams' processes/products are unstable is usually because of a team's lack of depth in DBA skills. Not necessarily the lack of a DBA, but the lack of any engineer with any eye or attention given towards environmental stability, optimization, and the traditional "janitorial" clean up work that can tend to be neglected in the BI space.
    - FranzFerdiNaN 5y ago
      If the BI teams stuff constantly breaks because production databases change I would say the software engineers are doing a shit job. It’s the engineers responsibility to communicate such breaking changes in advance to teams that rely on their products.
      - pm90 5y ago
        I see your point, but Engineers should not have to worry about dB schema changes breaking anything. The contract they provide externally is via APIs. The BI team slurping up dB binlogs must not create an expectation of support from engineers.
mjburgess 5y ago
This is the problem with business people being let loose to sell ideas without the technical depth. A data warehouse is not a capability, it's a data system. Company A with a warehouse and company B without do not necessarily differ in any capabilities.

devit 5y ago

You can run the workloads on (possibly partial) replicas of the production database running on another set of machines.