4 ms·

It seems that there’s a philosophy with these kinds od cloud data services: measure everything then figure out what to measure after. It seems that the better

It seems that there’s a philosophy with these kinds od cloud data services: measure everything then figure out what to measure after. It seems that the better approach is to make a hypothesis first, collect only the data you need, and then analyze. Otherwise it seems the indirect costs of retooling around “measure all the things indiscriminately” and the direct costs of paying for such services don’t seem worth it. Can someone offer a reasonable rebuttal?

staticautomatic 3y ago
It doesn't always make sense to "collect only the data you need." For example, someone in our org will ostensibly have a good reason for setting up an event in Google Analytics. As the head of analytics I have no use for their event but I'm going to end up ingesting it into BigQuery anyway because the built-in GA4->BQ transfer sends ALL the data, and I'm fine with that because it's damn near free for me to ingest and store. I could instead "collect only what I need" by running the transfer through an ELT tool and filtering out that event, but why would I bother doing that work and paying the ELT vendor money when Google will do it for free and charge me like $1K/year to store 100M rows of everything collected in GA4?
- williamcotton 3y ago
  That cost analysis seems incomplete. How much did it cost to implement in the first place? That seems like millions of dollars in capitalization before service expenses are even considered!
  - staticautomatic 3y ago
    Sorry but I can't tell if you're being sarcastic or not. Google Analytics and BigQuery are both nominally free, the service expense is $1K/year, and the implementation cost literally a few minutes of my time.
    - williamcotton 3y ago
      I’m not being sarcastic. There is engineering labor that requires hooking up all of these events. Then there is maintenance costs. And many more costs, direct or otherwise.
      - staticautomatic 3y ago
        With respect to the example I gave, no, there actually isn’t. In fact I don’t even understand what you think the work and costs involved are.
        williamcotton 3y ago
        I’ve seen organizations spend millions just in engineering labor costs for such endeavors. But sure, for small companies with limited domain events to begin with, the costs are relatively lower, and probably with the lower domain complexity it is less of a share of engineering time overall. But there are still opportunity costs! It’s not free to install dependencies across numerous code bases and hook into and label the events, even for a small business.
        fifilura 3y ago
        OP suggests using "built-in GA4->BQ transfer [to send] ALL the data". This seems like an extremely small endeavor.
        williamcotton 3y ago
        So yes, for a small business that just drops a GA snippet into the bottom of their app it is inexpensive. I’m not sure that this qualifies as the “modern data stack”. That’s also not what happens when a larger company integrates Segment into their applications! That takes many engineering months to set up and then maintain for the lifetime of the app. If the full costs are analyzed, is the money spent well? There’s sone irony here in that much effort is put towards analysis of user behavior but very little effort put towards accounting for where the money is spent internally. That is why there is a slowdown in customers using these MDS products, not what the author of the article claims, rather that when feet are put to the fire, expensive services that haven’t paid off their debt are given the axe.
neighbour 3y ago
The guy you're arguing with in the reply thread has a point but I think you're talking past each other because you're coming at it from two different sides. I understand his point about syncing everything if you're a small enough company or the data is small enough because the cost is immaterial. I agree with most of what you're saying and have seen the downfalls of "just sync all our data" first hand. At a late stage start-up I worked at, we would often have teams (finance, CX, etc.) implement a shiny new tool and immediately request to have that data accessible for analytics/reporting. If we did blindly agree, the data would often never be used and it would run up our Fivetran costs to sync the data and our Github actions bill to build the dbt models and include it in our modeling. We eventually landed on an approach that considered the following: - Does the tool already have reporting baked in? If so, just use that until you need more. - How often are you really going to be using it and what makes this data so special that you need to join it to our other data? - How much work will be involved to implement a sync for this data? Does Fivetran or Segment have a connector readily available? - If the tool doesn't have reporting built-in and the team did have a genuine business case for the data being synced, we would figure out which specific objects from the source that needed to be synced and sync them at a fairly infrequent cadence (once a day). So while I offer no reasonable rebuttal, I wanted to explain a happy medium that worked for us.
- williamcotton 3y ago
  Thanks for the insight! As for communicating the issue to administration I feel that putting things in terms of costs in dollars, even if the approach is as arbitrary as any other form of internal accounting, can help make the case. Management is as susceptible to the marketing of panaceas as much as engineers! They must be informed of a more true cost.

staticautomatic 3y ago

It doesn't always make sense to "collect only the data you need." For example, someone in our org will ostensibly have a good reason for setting up an event in Google Analytics. As the head of analytics I have no use for their event but I'm going to end up ingesting it into BigQuery anyway because the built-in GA4->BQ transfer sends ALL the data, and I'm fine with that because it's damn near free for me to ingest and store. I could instead "collect only what I need" by running the transfer through an ELT tool and filtering out that event, but why would I bother doing that work and paying the ELT vendor money when Google will do it for free and charge me like $1K/year to store 100M rows of everything collected in GA4?

williamcotton 3y ago
That cost analysis seems incomplete. How much did it cost to implement in the first place? That seems like millions of dollars in capitalization before service expenses are even considered!
- staticautomatic 3y ago
  Sorry but I can't tell if you're being sarcastic or not. Google Analytics and BigQuery are both nominally free, the service expense is $1K/year, and the implementation cost literally a few minutes of my time.
  - williamcotton 3y ago
    I’m not being sarcastic. There is engineering labor that requires hooking up all of these events. Then there is maintenance costs. And many more costs, direct or otherwise.
    - staticautomatic 3y ago
      With respect to the example I gave, no, there actually isn’t. In fact I don’t even understand what you think the work and costs involved are.
      - williamcotton 3y ago
        I’ve seen organizations spend millions just in engineering labor costs for such endeavors. But sure, for small companies with limited domain events to begin with, the costs are relatively lower, and probably with the lower domain complexity it is less of a share of engineering time overall. But there are still opportunity costs! It’s not free to install dependencies across numerous code bases and hook into and label the events, even for a small business.
        fifilura 3y ago
        OP suggests using "built-in GA4->BQ transfer [to send] ALL the data". This seems like an extremely small endeavor.
        williamcotton 3y ago
        So yes, for a small business that just drops a GA snippet into the bottom of their app it is inexpensive. I’m not sure that this qualifies as the “modern data stack”. That’s also not what happens when a larger company integrates Segment into their applications! That takes many engineering months to set up and then maintain for the lifetime of the app. If the full costs are analyzed, is the money spent well? There’s sone irony here in that much effort is put towards analysis of user behavior but very little effort put towards accounting for where the money is spent internally. That is why there is a slowdown in customers using these MDS products, not what the author of the article claims, rather that when feet are put to the fire, expensive services that haven’t paid off their debt are given the axe.

neighbour 3y ago

The guy you're arguing with in the reply thread has a point but I think you're talking past each other because you're coming at it from two different sides. I understand his point about syncing everything if you're a small enough company or the data is small enough because the cost is immaterial. I agree with most of what you're saying and have seen the downfalls of "just sync all our data" first hand. At a late stage start-up I worked at, we would often have teams (finance, CX, etc.) implement a shiny new tool and immediately request to have that data accessible for analytics/reporting. If we did blindly agree, the data would often never be used and it would run up our Fivetran costs to sync the data and our Github actions bill to build the dbt models and include it in our modeling. We eventually landed on an approach that considered the following: - Does the tool already have reporting baked in? If so, just use that until you need more. - How often are you really going to be using it and what makes this data so special that you need to join it to our other data? - How much work will be involved to implement a sync for this data? Does Fivetran or Segment have a connector readily available? - If the tool doesn't have reporting built-in and the team did have a genuine business case for the data being synced, we would figure out which specific objects from the source that needed to be synced and sync them at a fairly infrequent cadence (once a day). So while I offer no reasonable rebuttal, I wanted to explain a happy medium that worked for us.

williamcotton 3y ago
Thanks for the insight! As for communicating the issue to administration I feel that putting things in terms of costs in dollars, even if the approach is as arbitrary as any other form of internal accounting, can help make the case. Management is as susceptible to the marketing of panaceas as much as engineers! They must be informed of a more true cost.