4 ms·
One of my pet peeves in ML/stats/data science is people who hardly look at their data. Unless there are privacy reasons not to, then you really need to look at
by 6gvONxR4sf7o 7y ago
One of my pet peeves in ML/stats/data science is people who hardly look at their data. Unless there are privacy reasons not to, then you really need to look at some data. You'll learn so much more from looking at a few hundred samples than you will from different metrics. You'll get a feel for how complex the problem is, or whether something simple will do. Check your assumptions. You might even realize that your images are sideways.
- anoncareer0212 7y agoThe article explains why it's practically impossible to see that (check your assumptions! :P )
- m12k 7y agoWell, after you've read in your data, read them back out and display them on the screen.
- toxik 7y agoThis actually is generally true for any computer programming debugging. You need to look at the data as it appears to your program, to catch bugs in your preprocessing pipelines.
- BubRoss 7y agoI've tried working like that and it makes a massive difference. To be able to visualize all the intermediate stages is valuable because they are basically never right until you are able to isolate and refine them individually. https://github.com/LiveAsynchronousVisualizedArchitecture/lava https://github.com/LiveAsynchronousVisualizedArchitecture/la...
- ken 7y agoIt’s easy to print out the dimensions of the image when it’s loaded. I always log the size (pixels, rows, etc) of any data I load.
- dimatura 7y agoA critical piece of wisdom I find myself repeating to younger students, handed down from an advisor: "always look at the pixels". (Sometimes replacing pixels with voxels/lidar points/meshes, etc.)
- heyyyouu 7y agoYes, thank you. This can't be stated enough or loudly enough! You absolutely have to manually check, then check it again -- even with privacy issues, find a way to be non-identifiable and check it. Data is such a huge factor in whether your project will fail or not, yet too many people don't give it the respect (and dare I say love) it deserves.
- avip 7y agoThat’s what my first numeric mentor taught me. You have to look at raw data. The first q he’d ask any programmer was “did you look at the data, the actual data?”. He was a PhD in Physics and his approach really sticked with me. But it’s not always straightforward to “look at data”
- bonoboTP 7y agoAnd also at the results and the intermediate steps. Visualize everything. Don't just look at the final evaluation metric but dive in and see where things go wrong. Do just a few bad predictions skew your score? What does the best prediction look like? What does the worst look like? Are all your results just shifted by 2 pixels to the left due to some bug? Are there mislabeled examples in the test set? Etc. etc.
- barry-cotter 7y agoStuck. Everything else is perfect idiomatic English.
- avip 7y agoThanks!!
- AceJohnny2 7y agosomehow, the Allegory of the Cave comes to mind... ;)
- mlthoughts2018 7y agoJust looking at data and descriptive statistics is one of the first things a person is taught in machine learning, data science and statistics coursework. It’s a major skill in the field that is emphasized all the time. Practitioners frequently do cursory data analysis and data exploration to gain insight into the data, corner cases and which modeling approaches are plausible. Just to give some examples, Bayesian Data Analysis (Gelman et al), Data Analysis Using Regression and Multilevel/Hierarchical Models (Gelman and Hill), Doing Bayesian Data Analysis (Kruschke), Deep Learning (Goodfellow, Bengio, Courville), Pattern Recognition and Machine Learning (Bishop) and the excellent practical blog post [0] by Karpathy all list graphical data checking, graphical goodness of fit investigation, descriptive statistics and basic data exploration as critical parts of the model building workflow. If you are seeing people produce models without this, it’s likely because companies try to have engineers do this work, or hire from bootcamps or other sources that don’t produce professional statisticians with grounding in proper professional approach to these problems. When people mistakenly think models are commodities you can copy paste from some tutorials, and don’t require real professional specialization, then yes, you get this kind of over-engineered outcome with tons of “modeling” that’s disconnected from the actual data or stakeholder problem at hand. [0]: https://karpathy.github.io/2019/04/25/recipe/ https://karpathy.github.io/2019/04/25/recipe/
- faceshapeapp 7y agoFor one of my projects, I looked and manually labeled thousands of pictures, yet this issue never came up until people were actually using pictures coming from their smartphones.
- gbrown 7y agoParallel issue: people who fit complex models when simple descriptive statistics would be more useful/a better fit.
- mlthoughts2018 7y agoThis is a byproduct of hiring bootcamp grads or tasking a modeling project to engineers who read some tutorials. People think they can scan a few Jupyter notebooks and then professionally solve statistics problems. People wonder why it’s hard and expensive to hire ML engineers... because they actually solve these problems with craft. Meaning, they systematically grow understanding of the data, start with simple models, and have well articulated reasons explaining cases when complexity is justified.
- ma2rten 7y agoIt's the machine learning version of people who use hadoop if they could have used command-line utilities.
- pinouchon 7y agoAs Karpaty said: 1. Become one with the data. The first step to training a neural net is to not touch any neural net code at all and instead begin by thoroughly inspecting your data. This step is critical. I like to spend copious amount of time (measured in units of hours) scanning through thousands of examples, understanding their distribution and looking for patterns. Luckily, your brain is pretty good at this. One time I discovered that the data contained duplicate examples. Another time I found corrupted images / labels. I look for data imbalances and biases. I will typically also pay attention to my own process for classifying the data, which hints at the kinds of architectures we’ll eventually explore. As an example - are very local features enough or do we need global context? How much variation is there and what form does it take? What variation is spurious and could be preprocessed out? Does spatial position matter or do we want to average pool it out? How much does detail matter and how far could we afford to downsample the images? How noisy are the labels? source: http://karpathy.github.io/2019/04/25/recipe/ http://karpathy.github.io/2019/04/25/recipe/
- rajacombinator 7y ago100%. One of the most common mistakes too.
- Izkata 7y agoIt's not directly related to this topic, but my favorite example of this sentiment is Anscombe's Quartet [0], four sets of datapoints that have (almost) the same statistical values, but a very obviously different layout when simply viewed together on a graph. Plus an animated version that includes a T-Rex [1]. [0] https://en.m.wikipedia.org/wiki/Anscombe%27s_quartet https://en.m.wikipedia.org/wiki/Anscombe%27s_quartet [1] https://www.autodeskresearch.com/publications/samestats https://www.autodeskresearch.com/publications/samestats
- JorgeGT 7y agoOff topic, but each time I accidentally open the Wikipedia mobile layout in the desktop I'm amazed at how much better it is. Is there a way to always redirect to the mobile version?
- Skeime 7y agoIf you are logged in, the following will do it, I think: Go to Preferences, then Appearance and change the skin to “MinervaNeue”.
- m3at 7y agoYes, if you're on firefox I've been using that extension for quite a while and it works just fine: https://addons.mozilla.org/en-US/firefox/addon/mobile-wikipedia-webextension/ https://addons.mozilla.org/en-US/firefox/addon/mobile-wikipe...
- tomaskafka 7y agoYes. It's a method I use: Did you look at the actual data? No. Then they are wrong. Find a way to cheaply visualize data/state and call me back when you're done.