4 ms·

> building a data warehouse when you don't know what sorts of reports you want to run or how the data will generally be used after you collect it. Hi Bob, I c

> building a data warehouse when you don't know what sorts of reports you want to run or how the data will generally be used after you collect it. Hi Bob, I can't tell you what reports I want or what we'll do with the data until you've first collected the data for analysis. Thanks!

bob1029 6y ago
Why are you vacuuming up data if you don't even know what sort of outcomes you want to look for? How do you know you are even pulling from the right sources? When you do finally figure out why you are in business, you may find that all the data you gathered is worthless. Sometimes it's really innocuous stuff like storing datetime values without enough resolution or accuracy. Only when you know how you plan to use the data can you ensure you are gathering it properly. Teaching developers to collect all the data they can in hopes of being able to write arbitrary reports down the line is reckless behavior. Hyperscalers love pushing this ideology with inane bullshit like "Datalakes", so you can see where a conflict of interest may arise in the industry.
- 9nGQluzmnq3M 6y ago
  Ever worked with an enterprise company? I've worked on a project (in a peripheral role, fortunately) where the entirety of the CIO's mandate was to "use AI", full stop. So the team was tasked with putting together a data warehouse that would collect random logs and app metrics, and then sprinkle AI fairy dust of their choice onto it to generate Insights(tm). This kind of thing happens all. the time. in large companies where everybody involved wants either money or bullet points for their resume, and the actual outcome is nearly irrelevant.
- majewsky 6y ago
  When I implement a new service, I have to make an executive decision about which metrics to collect (operation counts, durations, storage usage, etc.). Before the first incident, I cannot know with 100% certainty which metrics are useful. Only after the incident can I look at the metrics and see which ones did weird things before and during the incident, and that helps drive my decisions on what to alert on. I don't see how I could do it differently. I can only truly know which metrics are useful from experience with past incidents, but I can only gain this experience when the metrics were already collected during the past incident.

bob1029 6y ago

Why are you vacuuming up data if you don't even know what sort of outcomes you want to look for? How do you know you are even pulling from the right sources? When you do finally figure out why you are in business, you may find that all the data you gathered is worthless. Sometimes it's really innocuous stuff like storing datetime values without enough resolution or accuracy. Only when you know how you plan to use the data can you ensure you are gathering it properly. Teaching developers to collect all the data they can in hopes of being able to write arbitrary reports down the line is reckless behavior. Hyperscalers love pushing this ideology with inane bullshit like "Datalakes", so you can see where a conflict of interest may arise in the industry.

9nGQluzmnq3M 6y ago
Ever worked with an enterprise company? I've worked on a project (in a peripheral role, fortunately) where the entirety of the CIO's mandate was to "use AI", full stop. So the team was tasked with putting together a data warehouse that would collect random logs and app metrics, and then sprinkle AI fairy dust of their choice onto it to generate Insights(tm). This kind of thing happens all. the time. in large companies where everybody involved wants either money or bullet points for their resume, and the actual outcome is nearly irrelevant.
majewsky 6y ago
When I implement a new service, I have to make an executive decision about which metrics to collect (operation counts, durations, storage usage, etc.). Before the first incident, I cannot know with 100% certainty which metrics are useful. Only after the incident can I look at the metrics and see which ones did weird things before and during the incident, and that helps drive my decisions on what to alert on. I don't see how I could do it differently. I can only truly know which metrics are useful from experience with past incidents, but I can only gain this experience when the metrics were already collected during the past incident.