4 ms·

> Statistics. Seriously, I really wish I had studied more of it in school. Basically goes for anyone in the STEM field, IMO. I really wish more people invested

> Statistics. Seriously, I really wish I had studied more of it in school. Basically goes for anyone in the STEM field, IMO. I really wish more people invested in statistics and data analysis classes. People take you more seriously in a business setting when you can say "Email A resulted in a response rate of 80%". I usually hear "We think Email A is better because we feel it in our gut". Ok, not those words exactly, but that's the point. Looking at data, understanding it, and directly applying it to your job is a hugely underrated skill.

holy_city 7y ago
A bigger issue I've seen is that "gut feeling" comes from a misplaced sense of confidence. Like say, sample size. I can't tell you how many times I've heard engineers say "the data isn't significant because the sample size is too small." If you have the data, calculate the confidence interval! Most of the time you don't need hundreds to thousands of data points to be reasonably confident, just a few dozen. I remember the example distinctly from my sophomore engineering stats course, I don't know why everyone else has forgotten it.
- opportune 7y ago
  In my experience when the data is very small it is almost always also biased towards how easy it was to gather, which also makes it non representative. Think about it, if it were as easy to let n=5000 as it were to let n=25, you would always pick 5000. You only pick n=25 because of the low effort involved, which often means proximity. A very common example is when some software feature is A/B tested only internally, or even only tested on the team that developed it. It introduces a lot of bias in users’ technical competence, willingness to understand/understanding of the new behavior, how the environment is set up, etc.
- m12k 7y ago
  I think most people overlook/don't know that the needed sample size depends not just on the confidence you want, but also on how big the effect you want to measure is. E.g. if landing page A has a conversion rate of 50% and B has one of 55%, that's going to take a lot of sampling to prove. But if A has 40% and B has 80%, then that's going to show up in the samples very quickly. But exactly which question you ask affects the needed sample size greatly - e.g. showing that 'B performs better than A' will take fewer samples than 'B performs at least 20% better than A'. This makes it much harder to have a correct intuition about needed samples sizes.
cosmie 7y ago
I went to school for statistics, and work in "analytics" (a catch all term for anything from basic reporting to analytics infrastructure to conversion rate optimization experiments). You're absolutely right in that people tend to take you more seriously if you come with numbers. But it's such a kangaroo court[1] that it drives me nuts. The instrumentation and implementation to support that sort of data-driven approach is usually far too lacking to give it the amount of merit it receives. Once you take do a first principle's sanity check of things, you learn that no one on the business side has a solid understanding of what "response rate" is actually referring to. Then when you look at the technical implementation, you realize that there's little reconciliation between what it's actually representative of and what anyone on the business side think it's representative of. Never underestimate someone's gut feeling, especially so if it's from an individual in the trenches. More often than not, dissonance between gut feelings and data point to an issue with the data. Not necessarily that the data is wrong, just that it isn't fully representative of the context it's being collected in and should be trusted accordingly. [1] https://en.wikipedia.org/wiki/Kangaroo_court https://en.wikipedia.org/wiki/Kangaroo_court
- commandlinefan 7y ago
  Any recommendations for somebody who ignored stats more than he should have as an undergraduate and wants to try to catch up? I've gone through and worked all of the exercises in my old calculus textbook so I have a good (fresh) handle on calculus now. Where's the best place to go next?
  - baldeagle 7y ago
    Understanding the distributions helped me the most. My favorite class was something like ‘single point estimates’ because it covered a lot about why mean and std dev are so powerful, but also sometimes so broken at conveying meaning.
  - cosmie 7y ago
    Depends on the context for trying to catch up; both where you're at in life and what you're wanting to get out of it. If you want to connect, feel free to reach out to the email in my profile. I may not be the best resource for best places to go next depending on what you're after, but may be able to help out. A few general points though: - Every field has it's own flavor of statistics. Supply chain, marketing, industrial engineering, business operations, finance, etc. Most practitioners will bastardize a technique or methodology common in their field before reaching out to another one for something more appropriate. Keep this in mind when looking at things, as you can find a lot of cases where the general premise for a technique is no longer valid, but practitioners are still going on momentum. You can also find some really neat nuggets/advancements that can be generalized and applied to another field. Although this can be difficult to suss out, as every industry tends to develop their own vernacular to refer to a particular set of base statistical techniques. - Focus less on the math and more on the applicability of a particular technique or methodology to a situation. Generally speaking, statistical techniques are nothing more than sophisticated heuristics. Their validity, applicability, and actionability are entirely dependent on the situation they're applied in and the particular heuristics (statistical techniques) chosen. Understanding the techniques that are out there, what their applications are, and what their limitations are is far more useful than focusing purely on the math. The math can always be looked up once you know what to look up. - Design of experiments[1] is a critical and often overlooked concept. It's rarely done in practice, and even when it is it's rarely more than a superficial attempt. But is a hugely important concept to understand how to approach a problem space. - The output of a statistical analysis ranges from "checks the box of measuring something but so disconnected from observed reality that it'll otherwise be ignored" to "interesting but not robust enough to make decision on" to "directionally accurate" to "willing to make decisions based on confidence intervals". Understanding where your analysis stands on that scale, and where it needs to stand to meet your needs, is critical. Align your efforts with your needs, and set expectations accordingly. [1] https://www.jmp.com/en_ch/applications/design-of-experiments.html https://www.jmp.com/en_ch/applications/design-of-experiments...
deleted 7y ago
[deleted]
mooreds 7y ago
Also suggest reading "How to Lie With Statistics". Classic, short book about fundamental stats concepts everyone should know.

holy_city 7y ago

A bigger issue I've seen is that "gut feeling" comes from a misplaced sense of confidence. Like say, sample size. I can't tell you how many times I've heard engineers say "the data isn't significant because the sample size is too small." If you have the data, calculate the confidence interval! Most of the time you don't need hundreds to thousands of data points to be reasonably confident, just a few dozen. I remember the example distinctly from my sophomore engineering stats course, I don't know why everyone else has forgotten it.

opportune 7y ago
In my experience when the data is very small it is almost always also biased towards how easy it was to gather, which also makes it non representative. Think about it, if it were as easy to let n=5000 as it were to let n=25, you would always pick 5000. You only pick n=25 because of the low effort involved, which often means proximity. A very common example is when some software feature is A/B tested only internally, or even only tested on the team that developed it. It introduces a lot of bias in users’ technical competence, willingness to understand/understanding of the new behavior, how the environment is set up, etc.
m12k 7y ago
I think most people overlook/don't know that the needed sample size depends not just on the confidence you want, but also on how big the effect you want to measure is. E.g. if landing page A has a conversion rate of 50% and B has one of 55%, that's going to take a lot of sampling to prove. But if A has 40% and B has 80%, then that's going to show up in the samples very quickly. But exactly which question you ask affects the needed sample size greatly - e.g. showing that 'B performs better than A' will take fewer samples than 'B performs at least 20% better than A'. This makes it much harder to have a correct intuition about needed samples sizes.

cosmie 7y ago

I went to school for statistics, and work in "analytics" (a catch all term for anything from basic reporting to analytics infrastructure to conversion rate optimization experiments). You're absolutely right in that people tend to take you more seriously if you come with numbers. But it's such a kangaroo court[1] that it drives me nuts. The instrumentation and implementation to support that sort of data-driven approach is usually far too lacking to give it the amount of merit it receives. Once you take do a first principle's sanity check of things, you learn that no one on the business side has a solid understanding of what "response rate" is actually referring to. Then when you look at the technical implementation, you realize that there's little reconciliation between what it's actually representative of and what anyone on the business side think it's representative of. Never underestimate someone's gut feeling, especially so if it's from an individual in the trenches. More often than not, dissonance between gut feelings and data point to an issue with the data. Not necessarily that the data is wrong, just that it isn't fully representative of the context it's being collected in and should be trusted accordingly. [1] https://en.wikipedia.org/wiki/Kangaroo_court https://en.wikipedia.org/wiki/Kangaroo_court

commandlinefan 7y ago
Any recommendations for somebody who ignored stats more than he should have as an undergraduate and wants to try to catch up? I've gone through and worked all of the exercises in my old calculus textbook so I have a good (fresh) handle on calculus now. Where's the best place to go next?
- baldeagle 7y ago
  Understanding the distributions helped me the most. My favorite class was something like ‘single point estimates’ because it covered a lot about why mean and std dev are so powerful, but also sometimes so broken at conveying meaning.
- cosmie 7y ago
  Depends on the context for trying to catch up; both where you're at in life and what you're wanting to get out of it. If you want to connect, feel free to reach out to the email in my profile. I may not be the best resource for best places to go next depending on what you're after, but may be able to help out. A few general points though: - Every field has it's own flavor of statistics. Supply chain, marketing, industrial engineering, business operations, finance, etc. Most practitioners will bastardize a technique or methodology common in their field before reaching out to another one for something more appropriate. Keep this in mind when looking at things, as you can find a lot of cases where the general premise for a technique is no longer valid, but practitioners are still going on momentum. You can also find some really neat nuggets/advancements that can be generalized and applied to another field. Although this can be difficult to suss out, as every industry tends to develop their own vernacular to refer to a particular set of base statistical techniques. - Focus less on the math and more on the applicability of a particular technique or methodology to a situation. Generally speaking, statistical techniques are nothing more than sophisticated heuristics. Their validity, applicability, and actionability are entirely dependent on the situation they're applied in and the particular heuristics (statistical techniques) chosen. Understanding the techniques that are out there, what their applications are, and what their limitations are is far more useful than focusing purely on the math. The math can always be looked up once you know what to look up. - Design of experiments[1] is a critical and often overlooked concept. It's rarely done in practice, and even when it is it's rarely more than a superficial attempt. But is a hugely important concept to understand how to approach a problem space. - The output of a statistical analysis ranges from "checks the box of measuring something but so disconnected from observed reality that it'll otherwise be ignored" to "interesting but not robust enough to make decision on" to "directionally accurate" to "willing to make decisions based on confidence intervals". Understanding where your analysis stands on that scale, and where it needs to stand to meet your needs, is critical. Align your efforts with your needs, and set expectations accordingly. [1] https://www.jmp.com/en_ch/applications/design-of-experiments.html https://www.jmp.com/en_ch/applications/design-of-experiments...

deleted 7y ago

[deleted]

mooreds 7y ago

Also suggest reading "How to Lie With Statistics". Classic, short book about fundamental stats concepts everyone should know.