7 ms·
Model Explainability is The Next Data Science Superpower
- dannykwells 8y agoI mentor individuals looking to transition into data science. I can say without a doubt that those who do the best (get the best jobs) are those with domain expertise and who are able to leverage that expertise in a data science realm. Of course, domain expertise is hard to come by and takes years to develop. I look forward to data science leaving far behind the notion that "you too can become a data scientist with 3 weeks of Python + scikit-learn".
- bilbo0s 8y ago>"you too can become a data scientist with 3 weeks of Python + scikit-learn"... I don't think any serious practitioners were out there thinking in this manner. But there is a gold rush right now, and any gold rush will attract its share of charlatans.
- cschmidt 8y agoThere are a whole ton of students coming out of one year masters DS programs, who don’t know that much more.
- softwaredoug 8y agoObligatory xkcd https://xkcd.com/1831/ https://xkcd.com/1831/
- softwaredoug 8y agoI see it all the time in my domain (search). Novice data scientists getting into a new field quickly get frustrated with existing practitioners. Existing practitioners want to use tried and true methods; data scientists want to reinvent the field with ML. The existing practitioners don’t know how to manage the usually black-box ML solution. Data scientists don't know the domain enough to be effective. So it doesn’t go far. It’s a dynamic that sadly burns out overly optimistic, but smart data scientists and sadly leaves a negative impression on existing practitioners on the promise of ML. Those practitioners stick their head back in a hole instead of innovating. All to say with more explainable ML (and more humility on everyone’s part) more progress would be made.
- minimaxir 8y ago> I don't think any serious practitioners were out there thinking in this manner. But a lot of thought pieces / YouTubers are pushing it, which is a problem.
- XuMiao 8y agoIn fact, many quantatitive researchers in their domain are already data scientists without using modern tools like machine learning. Being able to use these tools is only the beginning of a new practicing and advancing. I consider it the responsibility of computer scientists and engineers to make the tools better and easier. But unfortunately, it's that one has to become a good computer scientist first then a domain expert. We are not doing enough on innovations of tooling.
- Jedi72 8y agoComputing people in general seem to have forgotten the whole reason our field exists is to build tools for other people, the ones who do the "real work". With the possible exception of video games nothing we do with computers is useful or interesting on its own, its only point is to assist other fields. So we can have all the advances in the world but if we cant make them into useful tools, what was the point.
- itronitron 8y agowell, as long as your serious practitioners aren't trying to get hired they should be just fine
- 0815test 8y agoStatistics expertise is also very important. Stats may be less sexy today than "data science", but they were doing "model explainability" before it was even a thing!
- nonbel 8y ago>"but they were doing "model explainability" before it was even a thing!" Do you have an example?
- roman030 8y agoLinear Regression
- nonbel 8y agoCheck this: https://news.ycombinator.com/item?id=19353204 https://news.ycombinator.com/item?id=19353204
- mturmon 8y agoTo an extent, CART. But more classically, any structured statistical model with (eg) terms for variable interactions, measurement noise, and hierarchy.
- tel 8y agoPrior to model explainability was experimental design.
- bradleyjg 8y agoI don't understand what it can possibly be without statistics expertise. If you take away that, what's left? a A not-very-good programmer? I always took data scientist to be a rebranding of statistician. (To be clear there's nothing wrong with rebranding.)
- bloomer 8y agoWell the common description of a data scientist as "better at stats than a programmer and better at programming than a statistician" can also be reframed as "worse at stats than a statistician and worse at programming than a programmer" in most cases. I think it's typically a rebranding of PhD in a science field realized that market for that educational track is very small so rebranded as "hottest job of the next millennium"
- deleted 8y ago[deleted]
- Novashi 8y agoHonestly what should your salary be if you have domain expertise and are good with data science tools (and tools surrounding them like git, CI/CD, etc.)?
- hhs 8y agoI wonder if data scientists engage in a triangulation* of their sources and models to explain things? *I refer to this sense of triangulation: https://en.wikipedia.org/wiki/Triangulation_(social_science) https://en.wikipedia.org/wiki/Triangulation_(social_science)
- wrnr 8y agoLook into the provenance research that has come out of the semantic web community. They have developed a model where triple statements behave like short english sentences: :subject :object :predicate . ex: :alice :loves :bob . This statement by itself tells you nothing, did Alice say she loves Bob, or was it Bob who said it is Alice who loves him, or did Carol see the way Alice looked at Bob and concluded that she must love him, and what exactly is the quantitive difference between this love she feels for Bob and my love for chocolate. To support this kind of information each term is annotated with meta-data like document of origin, author, time and place. ex: :s1 rdfs:type rdfs:Statement ; rdf:subject :alice ; rdf:object :loves ; rdf:predicate :bob . :a1 rdfs:type prov:Activity prov:wasAssociatedWith :alice ; prov:hadMember s1 . :alice rdfs:type prov:Agent . In computer science they call this process reification and it could be the first step into creating a model that takes the source into account.
- hhs 8y agoThis is very useful. I'll check out this vocabulary, wrnr, thank you!
- valgaze 8y agoThis is interesting: A Kaggle competition to predict loan defaults gives an extreme example. This competition had 100s of raw features. For privacy reasons, the features had names like f1, f2, f3 rather than common English names. This simulated a scenario where you have little intuition about the raw data. One competitor found that the difference between two of the features, specifically f527 — f528, created a very powerful new feature. Models including that difference as a feature were far better than models without it. But how might you think of creating this variable when you start with hundreds of variables? The techniques you’ll learn in [OP's cours] would make it transparent that f527 and f528 are important features, and that their role is tightly entangled. This will direct you to consider transformations of these two variables, and likely find the “golden feature” of f527 — f528.
- deleted 8y ago[deleted]
- mhuffman 8y agoAn interesting thought experiment is what if it turns out that the "golden feature" of f527 -- f528 is race or gender? Here is a talk about times when that happens exactly: https://www.chrisstucchio.com/pubs/slides/crunchconf_2018/slides.pdf https://www.chrisstucchio.com/pubs/slides/crunchconf_2018/sl...
- tomnipotent 8y agoThis. This so hard. Data captures existing biases, so models will likely lead to more of the same outcomes. We see this already with crime statistics in the U.S., not to mention college acceptance rates, home & auto interest rates, even job offers. We may have gone to the moon, but by and large scientists of today are just as susceptible to mistakes as prior generations that had "solid evidence" like how European ancestry correlated with higher IQ. As good as it feels to wrap ourselves in the warm fuzzy blankets of "being woke" and thinking we're somehow different, I'm going to keep remaining skeptical of the results until I know exactly how they were reached (and the black box of deep learning makes that a little tricky).
- 8y ago
- wwarner 8y agoI think it's more exciting than author admits. Sure, it'll increase confidence and adoption, but it will really mean that the machine can be queried like an expert on whatever corpus it was trained on.
- boxfire 8y agoSensitivity analysis is the next data science buzzword rewrite of classically important skills in model building.
- hprotagonist 8y agoJust you wait until someone redefines hysterisis, too!
- crankylinuxuser 8y agoIts also model justification and defense. And even unprotected variables can be shown to synthesize protected variables. If I was in the legal profession , that's how I'd attack that. Again, data is toxic. Even if you think you're doing something safe, there's a chance youre not.
- curiousgal 8y ago..and the convergence of Data Science to Statistics continues.
- minimaxir 8y agoOddly this post doesn't bring up linear/logistic regressions, which are still used extremely often today despite the rise of ML/DL because the models are 100% explainable, and can therefore make business decisions as a result. Sure, it may not win you Kaggle competitions, but there are more important things in life. Another "compromise" is using variable importance outputs from tree-based models (e.g. xgboost), which IMO is a crutch; it says which variables are important, but not whether it's a positive or negative impact.
- nonbel 8y ago> "the models are 100% explainable" In my experience this is largely illusory. People think they understand what the model is saying but forget that everything is based on assuming the model is a correct description of reality. Your typical case of linear regression isn't even close to a correct description of reality, what gets included is largely arbitrary and due to convenience. As a result, different people with different types of data can get very different estimates for any features common to both models. Also, stuff like this: "I don't even think simple linear models are actually explainable. They just seem to be. Eg, try this in R: set.seed(12345) treatment = c(rep(1, 4), rep(0, 4)) gender1 = rep(c(1, 0), 4) gender2 = rep(c(0, 1), 4) result = rnorm(8) summary(lm(result ~ treatment*gender1)) summary(lm(result ~ treatment*gender2)) Your average user will think coefficient for treatment tells you something like "the effect of the treatment on the result in this population when controlling for gender". I get a treatment effect of 1.17 in the first case, but -0.38 in the second case, just by switching whether male = 0 and female = 1 or vice versa." https://news.ycombinator.com/item?id=16719754 https://news.ycombinator.com/item?id=16719754
- minimaxir 8y agoThat's fair. Explainable yet, but making sure the regression follows statistical assumptions is another can of worms, and one ignored by the rise of "easier" ML/DL.
- nonbel 8y agoThe thing is the ML models don't try to be interpretable. Instead people try to find models that make correct predictions. AFAICT, that is also the appropriate role for simpler methods like linear regression, unless you have reason to believe your model is robust to any missing "unknown" features and any included "spurious" ones. In that case you are probably dealing with a model based on domain knowledge and not even running a standard linear regression. People (over-)interpret the coefficients all the time though, I actually think this is a huge problem.
- nonbel 8y ago>"What features in the data did the model think are most important?" This is going to be more misleading than informative, since the feature importance hierarchy can change drastically based on what other features are included. > "For any single prediction from a model, how did each feature in the data affect that particular prediction" Ok, but I don't see how this will be comprehended as anything other than a black box for the typical ML model. > "What interactions between features have the biggest effects on a model’s predictions" If there are still big "interaction" effects that means you should have done better feature generation/selection.
- iguy 8y ago> since the feature importance hierarchy can change drastically based on what other features are included This sentence describes a disturbingly large proportion of social science papers. After controlling for X,Y,Z (naturally!) then we find that A is the leading case of B.
- nonbel 8y agoYes, precisely. It is a common practice in medical research as well.
- solidasparagus 8y agoYou don't need to explain how the model works, you just need to explain what factors the model cared about when making a certain decision. A strong example is a classifier that distinguishes between wolves and dogs. When you look at which pixels were the most impactful in 'Wolf' predictions, you can see that it is actually pixels of snow that are leading to predictions of 'Wolf'. It's still a bit of a black box, but now it's a black box with an obvious, measurable flaw that you can work to address.
- nonbel 8y ago>"A strong example is a classifier that distinguishes between wolves and dogs. When you look at which pixels were the most impactful in 'Wolf' predictions, you can see that it is actually pixels of snow that are leading to predictions of 'Wolf'." Yes, but how common is a situation like this? I mean your classifier is then not much better than making a histogram of pixel intensities.
- deleted 8y ago[deleted]
- drieddust 8y agoCorrelation is not the causation is the common wisdom. But this is exactly we do in DL. Yet this is exactly DL techniques seems to be doing. I am amazed that it works so well. I am not the expert. Can someone explain like ELI5?
- Nasrudith 8y agoWell the thing is that correlations can be used but they need a correction in the loop somewhere. The job of the human is to laugh and correct it. One example was teaching computers to draw cats based on reference images - many were lolcats of varying languages and it associated the floating fonts with cat features resulting in a blurry eerie gibberish mess. https://mobile.twitter.com/goodfellow_ian/status/937406530743287808?lang=en https://mobile.twitter.com/goodfellow_ian/status/93740653074... The two fixes are to either exclude the captioned examples from the set or to add in a filter "this is caption text - ignore it".