4 ms·
Best Practices for Applying Deep Learning to Novel Applications
- comicjk 10y ago> Let’s say you want to improve on a complex process where the physics is highly approximated (i.e., a “spherical cow” situation); you have a choice to input the data into a deep network that will (hopefully) output the desired result or you can train the network to find the correction in the approximate result. The latter method will almost certainly outperform the former. This aligns with my experience (computational chem PhD). When applying a strong, general-purpose mathematical patch to an existing model, use as much of the existing model as possible. Otherwise the patch will have a hard time fitting, and maybe be worse than what you started with. Philosophically, this also comports with my thinking (it's the modeling equivalent of Chesterton's Fence https://en.wikipedia.org/wiki/Wikipedia:Chesterton%27s_fence https://en.wikipedia.org/wiki/Wikipedia:Chesterton%27s_fence).
- gwern 10y agoLooks like good advice to me. Like a more DL-focused version of "A Few Useful Things to Know about Machine Learning" http://www.datascienceassn.org/sites/default/files/A%20Few%20Useful%20Things%20to%20Know%20about%20Machine%20Learning.pdf http://www.datascienceassn.org/sites/default/files/A%20Few%2... , Domingos 2012
- DrNuke 10y agoPhysics is deterministic in states and always tends towards an equilibrium, so novel results from DL may still fit some continuum math model without being stable or even real. Domain expertise, on the other hand, helps prepare data for ML algos in such a way that results will come (or not) within the boundaries of reality and hopefully stability. I am trying both approaches for some materials science goals of mine and am curious to see what happens, now that powerful hardware is cheap enough on the cloud to put some ideas to work. Side point is all this was just impossible five years ago, so I am grateful and excited to have this opportunity.
- AndrewKemendo 10y agoIt's good general advice, but frankly I think it doesn't address some of the major pitfalls - namely the top one: personnel. In the very beginning it's stated: "In this report I assume you are (or have access to) a subject matter expert for your application." In my experience this is where it goes off the rails for most of the crowd that she is addressing. Not because they don't have someone, but because who they have isn't really a "subject matter expert." It's a muddy term anyway especially in the field of Machine Learning. Excluding for a moment the huksters and bold faced liars, within ML there is WIDE variance in competence, domain specificity and application specific capability within the field. The biggest capability gap that I have encountered when working with fantastic ML folks is that the ones that understand the mechanisms/algorithms/approaches best, are actually pretty terrible at delivering production code. That's not for lack of capability, it's simply because the bulk of their time has been spent in research - so they approach things very differently than application focused engineers. This is extremely relevant in this case because this is an application specific paper. There are a plethora of mine-fields in applications of ML, some of which are outlined here from a systems approach, but the majority of which are personnel issues in my experience, and "culture" issues - not to be confused with "culture fit" problems that exist elsewhere.
- bagrow 10y agoI think the author means a non-ML expert interest in using ML. For example a geologist who wants to apply ANNs her field data
- AndrewKemendo 10y agoWell that's my point. A non-ML expert (your geologist) would need to find a SME to apply ML to their problem. The author seems to distinguish between these two roles rather than pushing the non-ML person into becoming a ML SME. Said geologist, would likely run into the problems I describe above if looking for that ML SME.
- mindcrime 10y agoI think everybody is kinda talking around each other here. In this context, the geologist is the SME. The "subject matter" is the geological application, not Machine Learning. The whole point of this document is advice for people who are SME's in some domain, but not ML experts, but who are working on applying ML in their domain. This report is targeted to groups who are subject matter experts in their application but deep learning novices. Of course it goes without saying that, in a ideal world, you'd have both domain SME's and ML experts working on the problem together, but given that we don't live in an ideal world, the point here - at least as I read it - is to help the non-ML-expert folks get started applying Deep Learning. And for all the stuff we can say about the difficulty of applying DL and the need for experts, the reality is that you don't actually always need a "deep learning expert". Let's say your problem just happens to be similar to, say, handwritten digit recognition. Given that anybody can download DL4J, go through the tutorials, and get a network going in an hour or two that gives something like 98%+ accuracy, there's a good chance that our "geologist who's a DL novice" can create a network that will yield useful results. Maybe getting the last couple of percentage points of accuracy out will require a "real" DL expert, but hell, depending on the scenario, that might not even be needed.
- bluetwo 10y agoWas kind of hoping for some examples of novel applications no on has thought of. :-)
- lngnmn 10y agoSurprisingly good and sane paper, without all that hipster's bullshit. The emphasis on the quality of the training data and, most importantly, on the evaluation and careful choice of heuristics on which the model to be build upon, is what makes the paper sane. There is no shortage of disconnected from reality models based on dogmas, while, it seems, there is acute shortage of the models properly reflecting some particular aspects of reality. Data and proper, reality-supported heuristics (domain knowledge) are the main factors of success. Technical details and particular frameworks are of the least importance. This, BTW, is why it is almost impossible to compete with megacorps - they have the data (all your social networks) and they have the resources, including domain experts, without whom designing and evaluating a model is a hopeless task.