6 ms·
One of the ML mistakes that will hurt most: not collecting the right features
- uoaei 4y agoOne of the lessons I'm learning in my work right now is: Anyone can throw a mess of features into a huge hog of a model and get some sufficiently accurate input. But with feature engineering, you essentially absolve the computational model from needing the vast majority of the capacity in the naive case, instead "pre-computing" those transforms which would otherwise be learned by the model. With the reduced need for capacity, the ability to refine the model for interpretability's sake (and optimization stability) also improves. Of course part of success is also collecting the right data, but focusing on the right features is a slightly more nuanced and focused way to look at it IMO.
- sumy23 4y agoThis doesn’t seem like good advice. As a human, the data might not be engineerable by you. The data might seem like it is, but you may end up losing information through your feature engineering. The combinatorial power of NNs is so great that it’s almost always better to give the model as much data as possible and let it figure out which parts of the data are important.
- uoaei 4y agoYou're right, my advice may be biased. I work on physical systems operating under tight control and near equilibrium most of the time.
- melling 4y agoFunny he would say that because a few days ago in one of his tweets he was calling the stock market bottom for this year. I replied that he was using the wrong inputs: https://twitter.com/melling/status/1537562264387166209?s=21&t=YssDz7ElZAdzg-4jNB_zLA https://twitter.com/melling/status/1537562264387166209?s=21&... People make the same mistake in their decision process. I was going to add that his model wasn’t explainable either but only so much geek humor…
- mxwsn 4y ago"One of the ML mistakes that you will hurt you the most: not collecting the right features. If the information you need isn't in your data, you won't recover it with a better model." The word 'information' is doing some heavy lifting here. There is nuance - what information could be learned or computed from the data? We can only really guess. When we're wrong about this, then we will design models imperfectly. Is there enough information in internet text scrapes to solve unseen high school mathematics word problems? The feature engineers of olde would have said no. But there is, to a surprising degree.
- kazinator 4y agoThe way of speaking about "data" being an undifferentiated mass of bits, from which you get "useful information" is pretty well-worn, widely observed piece of rhetoric. (If it's wrong, the scope of the mistake reaches far beyond this article.)
- npalli 4y agoThis approach of data maximalism seems to work well primarily for NLP and CV. However, there are lot of domains where throwing every feature without some consideration of information will not get you good results. To take a simple case, you are asked to model (forecast) the sales of an item at a retail store. Jamming in features like - color of people shirts, average length of their nails, the duration of traffic signals in front of the mall, the length of the store name etc.. will not lead to good results. In contrast, seasonality, prices, promotions, weather etc. will most likely lead to good results. Why does the second set of features work while the first doesn't? I like to think of the second set having a model of causality, to figure that out you need some domain expertise and experience. Without that expertise, the feature set of non-causal factors is almost infinite.
- bloaf 4y agoI think the real biggest "gotcha" is not understanding where the control loops are in your data. The classic example is the "air conditioned room" thought experiment. Imagine feeding an AI model a dataset with a bunch of datapoints like this Room hot | AC on full blast Room cold | AC off Room warm | AC on 50% Then asking the AI how to make a hot room cold. The AI is very likely to say "turn off the AC" because "AC off" is a feature of cold rooms. Maybe we think that if we give the AI more information, it could do better. Lets imagine that our room has a window and an oversized AC. We'll now be feeding it info about the source of heat as well as the source of cold, so surely it will get the right answer this time: Window open | AC on full blast | Room cold Window closed | AC off | Room cold Window 50% | AC on 50% | Room cold This dataset is going to have the AI thinking that neither the window nor the AC impact room temperature, and that you might be able to close a window by turning off the AC. The key thing to understand is that control loops change the "physics" of your system and that knowledge about the control-physics does not transfer to the uncontrolled-physics.
- superb-owl 4y agoI've spent some of my career doing Data Science consulting, and easily 80% of my time is spent on finding, cleaning, formatting, and studying data. Building the model is almost trivial if you take the time to get your data in the right shape (i.e. removing noise and unsalient features, and making sure you have every usable bit you can get)
- morelandjs 4y agoStrongly agree with the author. A logistic regression built on top of perfectly crafted features will out perform more advanced approaches with crude data.
- jmyeet 4y agoIsn't this just "garbage in, garbage out"?
- hatmatrix 4y agoYes, but the statement provides more precision on what garbage is.
- monkeybutton 4y agoThe best datascience projects are ones that started collecting good data 3 months ago. The hardest thing I've had to do is convince engineers to involve DS people in the design phase of their work. Initially they just don't see us, or see us as an annoying extra stakeholder. But it's more annoying to go back to already deployed code and add data collection to it and then waiting, so they learn fast. Anyways, like the OP's link says, there's no magic modelling work that can bring back those missing features and data.
- usehackernews 4y agoI’m a product manager - The company (medium sized public company) I was at previously asked me how we could better utilize machine learning in our products. I spent a month writing and implementing a new company playbook on a building products with data science. It boils down to including data science early during discovery (In part to help identify what data we have available up front), and viewing them as a squad stakeholder rather than a team for one-off work. It also pushed everyone to start thinking about the reliability of data for every product we release. Some of the databases I’ve seen at companies is just an absolute abomination.
- master_yoda_1 4y agoI think it’s too much simplifying ML. If features are so easy to find yes you can write a rule based system and it works. But most of the real life problem is not easy.
- whatever1 4y agoThe scary thing about ML in practice is that it seems that it works until it does not work at all. Then people freak out and dismantle the entire ML team.
- SMAAART 4y agoThe weakest link is always the human factor: 1. data 2. algo
- TrackerFF 4y agoWith the explosion of deep learning, I don't want to say that feature engineering is a lost art - no at all - but many of the benefits of DL, such as automatic feature extraction, is certainly something to be considered before just jumping into a project. I've seen examples of where good and careful data collection in advance would have solved the problem, but instead, someone decided to transform the (easiest) collected data into a format suitable for DL - and spent months trying to optimize and tweak the DL model. Because their rationale was that it's much easier to just fine tune an out-of-box DL / transfer learning model, than spending ages on extra planning, data collection, feature engineering, and what not.