4 ms·

Yep, we have a expensive reporting tool, can too all kinds of things, query every type of database we have... Number one function used. Export to Excel, and

Yep, we have a expensive reporting tool, can too all kinds of things, query every type of database we have... Number one function used. Export to Excel, and normally it is on reports of basic table join queries, none of the advanced things it can do with the data

SoftTalker 4y ago
This is because businesspeople like to fiddle with formulas, graphs, and filters untill they isolate (i.e. cherry-pick) the subset of data that [makes them look good | supports their agendas]. They don't want canned reports. Excel makes this easy because they already know how to do those things in Excel.
- Godel_unicode 4y ago
  People get really mad when you call this p-hacking, btw. I used to try and make this point to ”analysts” that I worked with, that science involves coming up with a hypothesis and then looking at the data. The response I got often enough that I’m sure it’s being taught somewhere was that you “need to let the data tell it’s own story”. Turns out lots of people have decided what the conclusion is and are taking that conclusion and the data, and then blindly munging until they get a path which connects them.
  - axelf4 4y ago
    As my professor said: All good statistics is done before you have looked at the data.
    - sideshowb 4y ago
      But where do you get your hypothesis from?
      - beckingz 4y ago
        your hypothetical model of reality. But that's a lot of work so businesses don't want to pay for it.
      - Tams80 4y ago
        Whatever orifice you desire.
    - merouan 4y ago
      True, for clean data. You can’t clean data without looking at it (a lot), though.
      - vanviegen 4y ago
        If you have data that's so 'dirty' that you can't decide on the filtering rules in advance (or based on only historic data), then what you have is garbage, not data. Therefore, we could call the art of shaping this into meaningful stories garbage science.
        WastingMyTime89 4y ago
        Tell me in a comment you have never worked with business data in your life. Business data is full of minor inconsistencies which are not obvious until you sit in front of it. Products are sold by different units. Reporting ranges and aggregates are slightly different. Subsidiaries use categories which are close but not exactly identical. There is generally plenty of massaging to do before you can get the information you need.
  - SoftTalker 4y ago
    > I’m sure it’s being taught somewhere was that you “need to let the data tell it’s own story”. AWS is pushing this message on every NFL broadcast with their Next Gen Stats ads. "The data tells us all"
  - extr 4y ago
    I think a lot of times unless you are operating at hyperscale and bps matter, most business data analysis is more art than science. Ex, if you have < 200 sales in a month and you want to report this, it's totally normal to want to pull it into excel and fiddle with the numbers, make ad-hoc corrections, divide those 200 sales up in very subjective ways, etc. The smaller the company the more the reporting layer is only a weak approximation of reality, there isn't enough there to create a narrative around. "Hypothesis testing" such data is honestly a waste of time. Of course you want to be intellectually honest about it, and I agree that cherry-picking the "right" data can be a big problem in the wrong kind of organization. I remember one time a Jr analyst I managed was asked by the CEO to create a certain chart. I was not looped in. The CEO then used that chart to convince himself and many others on the executive team to make a huge product change that ended up being a complete disaster.
  - deleted 4y ago
    [deleted]
- FormerBandmate 4y ago
  Excel also makes it easier to experiment. Pandas requires you to change an entire column which can frequently be processor-intensive, take forever, and be hard to understand. With Excel, you can just take a random cell, try something out, and if it comes up with nonsense, undo it
  - twobitshifter 4y ago
    Why not df.head()… ?
    - FormerBandmate 4y ago
      That can also take a while, and it isn’t as easy to view the values side by side. Excel is simpler (although way less capable) a lot of the time
- dan-robertson 4y ago
  It is sometimes more possible to determine if your analysis is wrong in excel because you can look at the numbers and the numbers have meanings. It can be harder to debug things if you can’t see intermediate computations. I guess I would say that excel prevents some kinds of bad problems with faulty analyses but opens you up to other problems like overfitting or the cherry-picking you describe. But it also seems impossible to produce a reporting tool that both gives useful analyses and doesn’t do incorrect analyses.
- marniewebb 4y ago
  As one of those business people, I find I’m not very good at requesting reports. There is a gap between what I think is interesting and what actually is interesting. I use spreadsheets (and sometimes visualization tools) to help bridge the gap and make a better on going request to our reporting team. Often, this is learning for me. I have a bunch of Stephen Few’s books and use exported CSV files to figure out which reports are useful to me and my org. When I find them, I do make a request for standardized reporting. These often become the basis for regular review meetings. In those meetings, we still come up with instances where we need to export to CSV and get into the data to understand what we are looking at. Our work is going through a big shift this year that means our historic data is not helpful in predicting trends. That’s increased the need for this kind of engagement with the underlying data.

SoftTalker 4y ago

This is because businesspeople like to fiddle with formulas, graphs, and filters untill they isolate (i.e. cherry-pick) the subset of data that [makes them look good | supports their agendas]. They don't want canned reports. Excel makes this easy because they already know how to do those things in Excel.

Godel_unicode 4y ago
People get really mad when you call this p-hacking, btw. I used to try and make this point to ”analysts” that I worked with, that science involves coming up with a hypothesis and then looking at the data. The response I got often enough that I’m sure it’s being taught somewhere was that you “need to let the data tell it’s own story”. Turns out lots of people have decided what the conclusion is and are taking that conclusion and the data, and then blindly munging until they get a path which connects them.
- axelf4 4y ago
  As my professor said: All good statistics is done before you have looked at the data.
  - sideshowb 4y ago
    But where do you get your hypothesis from?
    - beckingz 4y ago
      your hypothetical model of reality. But that's a lot of work so businesses don't want to pay for it.
    - Tams80 4y ago
      Whatever orifice you desire.
  - merouan 4y ago
    True, for clean data. You can’t clean data without looking at it (a lot), though.
    - vanviegen 4y ago
      If you have data that's so 'dirty' that you can't decide on the filtering rules in advance (or based on only historic data), then what you have is garbage, not data. Therefore, we could call the art of shaping this into meaningful stories garbage science.
      - WastingMyTime89 4y ago
        Tell me in a comment you have never worked with business data in your life. Business data is full of minor inconsistencies which are not obvious until you sit in front of it. Products are sold by different units. Reporting ranges and aggregates are slightly different. Subsidiaries use categories which are close but not exactly identical. There is generally plenty of massaging to do before you can get the information you need.
- SoftTalker 4y ago
  > I’m sure it’s being taught somewhere was that you “need to let the data tell it’s own story”. AWS is pushing this message on every NFL broadcast with their Next Gen Stats ads. "The data tells us all"
- extr 4y ago
  I think a lot of times unless you are operating at hyperscale and bps matter, most business data analysis is more art than science. Ex, if you have < 200 sales in a month and you want to report this, it's totally normal to want to pull it into excel and fiddle with the numbers, make ad-hoc corrections, divide those 200 sales up in very subjective ways, etc. The smaller the company the more the reporting layer is only a weak approximation of reality, there isn't enough there to create a narrative around. "Hypothesis testing" such data is honestly a waste of time. Of course you want to be intellectually honest about it, and I agree that cherry-picking the "right" data can be a big problem in the wrong kind of organization. I remember one time a Jr analyst I managed was asked by the CEO to create a certain chart. I was not looped in. The CEO then used that chart to convince himself and many others on the executive team to make a huge product change that ended up being a complete disaster.
- deleted 4y ago
  [deleted]
FormerBandmate 4y ago
Excel also makes it easier to experiment. Pandas requires you to change an entire column which can frequently be processor-intensive, take forever, and be hard to understand. With Excel, you can just take a random cell, try something out, and if it comes up with nonsense, undo it
- twobitshifter 4y ago
  Why not df.head()… ?
  - FormerBandmate 4y ago
    That can also take a while, and it isn’t as easy to view the values side by side. Excel is simpler (although way less capable) a lot of the time
dan-robertson 4y ago
It is sometimes more possible to determine if your analysis is wrong in excel because you can look at the numbers and the numbers have meanings. It can be harder to debug things if you can’t see intermediate computations. I guess I would say that excel prevents some kinds of bad problems with faulty analyses but opens you up to other problems like overfitting or the cherry-picking you describe. But it also seems impossible to produce a reporting tool that both gives useful analyses and doesn’t do incorrect analyses.
marniewebb 4y ago
As one of those business people, I find I’m not very good at requesting reports. There is a gap between what I think is interesting and what actually is interesting. I use spreadsheets (and sometimes visualization tools) to help bridge the gap and make a better on going request to our reporting team. Often, this is learning for me. I have a bunch of Stephen Few’s books and use exported CSV files to figure out which reports are useful to me and my org. When I find them, I do make a request for standardized reporting. These often become the basis for regular review meetings. In those meetings, we still come up with instances where we need to export to CSV and get into the data to understand what we are looking at. Our work is going through a big shift this year that means our historic data is not helpful in predicting trends. That’s increased the need for this kind of engagement with the underlying data.