8 ms·

Why is machine learning ‘hard’?

327 points by Dawny33 10y ago

du_bing 10y ago
That's right, machine learning requires knowledges of so many fields, so if any problem occurs, developer has to do so so many check to find the problem, and optimize it.
- curiousgal 10y ago
  Exactly why I despise "ML Crash course" and "Learn ML in x hours" kind of courses.
  - dzhiurgis 10y ago
    It gets worse. There's YouTube channel with uploads like: Build a Neural Net in 4 Minutes Build an Antivirus in 5 Min Build a Self Driving Car in 5 Min
    - walrus 10y ago
      Have you watched any of them? I just watched "Build a Self Driving Car in 5 Min" and the content was good. My only complaint is that his demo used end-to-end learning, which isn't how most self-driving cars actually work (but he acknowledges this).
      - Drdrdrq 10y ago
        Link: https://www.youtube.com/watch?v=hBedCdzCoWM https://www.youtube.com/watch?v=hBedCdzCoWM
  - taneq 10y ago
    Cue a link to the essay, "Teach Yourself Programming In Ten Years".
peatfreak 10y ago
Why would you expect it to be easy?
- Kirth 10y ago
  I've met some people whom believe that the more modern tools (such as Tensorflow) magically require little human input and make it so that you do not need to know/understand the mathematics and statistics. Not sure where they get this idea. Everyone wants to do machine learning, but nobody seems to want to learn statistics.
  - an_account 10y ago
    How would you recommend learning ML? I've been interested but don't know where to start.
    - kevinwang 10y ago
      Depending on your situation, this advice may be useless, but I'd say for me personally, taking an intro to machine learning class at my university is a wonderful and easy way to learn about the basics. So I'd recommend taking a university class in-person, or online.
      - Scea91 10y ago
        This is the best answer. In the end nothing is better than a degree in the field.
    - adamnemecek 10y ago
      I put together a list of resources I found useful: https://news.ycombinator.com/item?id=12900448 https://news.ycombinator.com/item?id=12900448
    - blahi 10y ago
      Not who you asked, but: Statistics in Plan English. Regression Analysis by Gilman Elements of Statistical Learning. You need to throw some matrix algebra and calc 1 & 2 somewhere in between. Certainly before ESL. It would also require You can't simply read the books and go through the examples. You will be stuck at a concept at many occasions and you will battle it out until after much googling and reading additional papers, you finally get. After those 3 books, you've got the basics.
godmodus 10y ago
because of preprocessing and needing to choose the functions that'll do the approximation - the process itself is semi-automatic, not fully automatic. ANN's inner nodes are specific functions that need parameter tweaking (after choosing the right ones that is) - vector machines have different kinds for different data, etc. and those two things are very domain specific so you need to do a lot of homework first, and debugging later.
tmptmp 10y ago
Beautifully written and insightful. >>After much trial and error I eventually learned that this is often the case of a training set that has not been correctly randomized and is a problem when you are using stochastic gradient algorithms that process the data in small batches. Take this single term from the above sentence: "stochastic gradient algorithms", they represent three key areas: statistics, calculus and CS. These three things, even when they are to be studied in isolation are much complex. For ML, you must be able to juggle these 3 fireballs effectively. No surprise, it's much, much more difficult than many other software engineering problems.
- ttub 10y ago
  No offense intended, but what is your background? I'm a researcher with a physics/stats PhD, and if a colleague approached me and said "stochastic gradient algorithms" entails three highly complex areas of scientific knowledge, I would have been stunned and assumed an undergrad with an English major had stumbled into our lab. Just because you find something extremely challenging, doesn't mean it is inherently challenging. Considering what a lot of people in my field is struggling with, your example is absolutely trivial. You might want to adjust your ego downwards a bit.
  - DavidSJ 10y ago
    He didn't say he found it personally challenging. He said this part of optimization finds itself at the intersection of three deep fields of human knowledge. That is undeniably correct.
    - partykid92 10y ago
      Thats like saying bubblesort lies at the intersection of combinatorics, topology and complexity theory. True, but pretentious
      - Al-Khwarizmi 10y ago
        I don't think it was pretentious, and your analogy is quite off. You don't need to understand any of those fields to grok bubblesort (algorithmics is enough). You do need to know some statistics, calculus and algorithmics to understand stochastic gradient descent.
blt 10y ago
I think to some extent, this kind of difficulty occurs with any numerical programming. With typical software engineering the bugs are usually in logic. All but the hardest problems can be diagnosed with a visual debugger. With numerical code, though, you usually can't look at the state of a few discrete-valued variables and a stack trace to figure out what went wrong. "Why is this matrix singular?" can take days to answer. You spend a lot of time staring at the code, comparing it to the math on paper, trying to visualize high-dimensional intermediate data, etc. Continuous math can be a lot harder to reason about than discrete.
- simonbarker87 10y ago
  This was it for me. I don't have a CS background but with PhD in electronics/semiconductors I can usually brute force my way through things and get a good grasp. I got half way through Andrew Ng's ML course and felt I fully understood the concepts but spent all of my time battling with matrices and the foundation mathematics rather than building on ML understanding. If I had a better grasp of the maths then I think I'd get on better with it.
- nabla9 10y ago
  It's even worse than that. Talking from personal experience in the private sector: You go trough the whole process and present your results for a customer. They make the required changes, see few percent improvement in the bottom line and are happy. Year later you pick up the same code for another project and discover small error in data collection script that completely invalidates everything. Garbage in, garbage out errors can have internal consistency that survive cross validation and provide similar results with many different algorithms and models. Random changes in the real word can produce actual gains. Standard machine learning datasets rarely suffer from this problem because results and semantics of the problem are known beforehand. If you have a original black-box problem, it's possible to do random search and improve by accident.
- boulos 10y ago
  tl;dr: Agreed, but the problems and models ML folks play with are more complex than most physics-like systems. Having spent a lot of time doing Monte Carlo rendering, I can say that "why is this pixel messed up" can be a painful affair (and that domain is at least visual!). Bad input geometry, NaNs or Infs because you forgot to handle some 1/r^2 term, etc. occur in all numerical computing. Some of this is actually obviated in TensorFlow the same way it is in much of numerical computing: frameworks that have been battle hardened through years of bug fixes. This lets you at least focus on "Did I describe my model correctly?" rather than "Did I implement gradient descent correctly too?". Again, rendering (and all physical simulation) has the benefit of producing a picture and the inputs are effectively "physical". The challenge for machine learning is that you're trying to optimize for figuring out the underlying model rather than relying on the laws of physics for your problem. While this is just as true in any numerical modeling problem ("Should this be a 4th degree polynomial or just cubic?"), the models that ML folks put together are effectively quite complex and the input datasets are both very large and usually unstructured. The visualization of the network being trained in the TF Playground (http://playground.tensorflow.org http://playground.tensorflow.org) helps with the model but not the original space.
partykid92 10y ago
one big dimension here, the "implementation error" can be easily be debugged. Gradients can be checked numerically. The model can be checked to work by looking at the optimality conditions (not just the loss function go down). This shouldn't be an issue for anyone from a traditional coding background.
edblarney 10y ago
I've worked in computational linguistics: 1) It's 'hard' because you need a lot of 'training data' in order to train models etc.. It's hard to get. 2) 'AI' type interfaces represent a whole new kind of UI challenge. For 'predictive typing' for example, you can optimize an algorithm so that it does better for 90% of the US population, but then it gets 'worse' for the remaining 10%. So it's a paradox. This can have weird effects. For example, if you have an app in the app-store, you may leave the settings so that it's 'broadly optimal'. You get ok stars. If you then make it 'better' for those 90%, you might get a little boost in ratings, but you get 1 and 0 star ratings from the 10% for whom it's a sub-par experience. This can destroy your product. Anyhow - 'there is no right answer' often in AI, and setting expectations can be extremely difficult. And all of that has nothing even to do with CS.
- Kenji 10y ago
  That reminds me of Stephen Hawking's computer. His old computer gave deterministic dictionary predictions and after some time, he had memorized all the predictions that came up, so he could be much quicker because he knew exactly how much of the word to type and then to select the proper word - a purely mechanical process that doesn't require feedback anymore. However, when they gave him a new computer, it had adaptive predictions (I think) and Hawkins could not memorize them anymore. Then he always had to type letter for letter and inspect the predictions before he could select one - a huge step down. Stuff like that is absolutely crucial but is often forgotten by engineers.
  - kixiQu 10y ago
    Seconded, and yet it's madness to me that engineers don't pick up on this. I can't be the only one who's let someone use my computer for basic internet browsing and then been deeply irritated when their newly visited sites break my intuitive knowledge of how many characters I need to type in the URL bar before hitting enter...
- mingyeow 10y ago
  "And all of that has nothing even to do with CS." -> therein lies the kicker! Consumer level (including sass) apps require very real user and business tradeoffs. That tradeoff is rarely discussed, which i honestly find very curious in and of itself.
ramblenode 10y ago
The diagrams are pretty misleading. One can craft a space with any number of arbitrary dimensions, but that's kind of meaningless until the space is populated with data. Certainly the likelihood of a bug is not uniformly distributed across the space, and certainly the density of bugs within a space varies greatly depending on the problem. I imagine the average kernel developer's 2-D space is both very dense and has greater spread than the 4-D space of many ML engineers.
atomical 10y ago
I'm getting a 404.
- a_bonobo 10y ago
  Me too, the whole blog 404s Here's the cached version: https://webcache.googleusercontent.com/search?q=cache:https://ai.stanford.edu/~zayd/why-is-machine-learning-hard.html https://webcache.googleusercontent.com/search?q=cache:https:...
- dgellow 10y ago
  try the HTTP (non SSL) version. I had to disable HTTPSEverywhere
jupp0r 10y ago
There are also lots of architectural problems with machine learning components to consider, beautifully summarized in "Machine Learning: The High Interest Credit Card of Technical Debt" by Sculley et al. http://research.google.com/pubs/pub43146.html http://research.google.com/pubs/pub43146.html
DrNuke 10y ago
Generally speaking, being hot in the media does not help: walking the walk is way harder than talking the talk.
Matthias247 10y ago
I think once you have the need to go deep enough into a topic they all get hard. Debugging and testing is also hard in all things that are somehow related to realtime or concurrency. E.g. OS development, embedded firmware, network stacks, etc. For these things you often also need to know about Math, Physics, Statistics, Electronics, Hardware and Software Architecture, etc. Game engine development is also hard because you should also know about most of this stuff to really find the most efficient solutions.
norswap 10y ago
> Machine learning often boils down to the art of developing an intuition for where something went wrong (or could work better) when there are many dimensions of things that could go wrong (or work better). I'm not a practitioner, but I always thought this was the main challenge. Uses of ML are rarely "right" or "wrong" per se, but they rely on intuition to get a model that "works" in a practical sense. There is no royal way to machine learning: you can't decide you are going to make an algorithm that detects bad comments (as determined by human consensus) and then just go make an implementation that you can reason out to be correct, the way you could prove a graph algorithm correct. Trial-and-error and hard-to-transcribe intuition are baked into the process. (I'd love to get some insider insight on this comment!)
Dzugaru 10y ago
Even if you develop an "intuition" for known tasks (like classification) - there are so many problems that are not tackled yet and no one has any "intuition" yet. Common sense very often doesn't work there (in high-dimensional spaces ;)). For example I've only recently stumbled upon an "Explaining and harnessing adversarial examples" article - and that completely changed my perception about my current work in computer vision.
js8 10y ago
I think it is hard because of https://en.wikipedia.org/wiki/No_free_lunch_theorem https://en.wikipedia.org/wiki/No_free_lunch_theorem That follows there is no single "good" algorithm, and you need to have and exploit domain knowledge in order to succeed.
jorgemf 10y ago
In software you can track the program and detect what instruction is not doing what it is suppose to do. In machine learning there is no program to track, it is not a set of instructions with a purpose, the whole thing either works or not. In order to discover what could be failing, you need to have a deep knowledge about a lot of stuff (maths, statistics, CS) to figure out what is wrong. And sometimes the answer is that the problem doesn't have a solution.
- fnl 10y ago
  It is very much possible to track your calculations and quite literally debug your models. But like in computer science, it is hard to find the data scientist who can actually do that and not just copy-paste tutorial code from some blog post and then wonder what isn't working...
deleted 10y ago
[deleted]
aflam 10y ago
The long and convoluted debugging cycle for machine learning really hurts my faith in ML models. This issue - with some practical advice - was the center of this interview (disclaimer: author). https://shapescience.xyz/blog/interview-data-science-methodology/ https://shapescience.xyz/blog/interview-data-science-methodo... I'm convinced we lack decent tools for ML debugging: what could they be?
mattlondon 10y ago
I think the problem is that we don't really understand ML properly yet. Like picking hyperparamters - time and time again I've asked experts/trainers/colleagues: "How do I know what type of model to use? How many layers? How many nodes per layer? Dropout or not?" etc etc And the answer is always along the lines of "just try a load of stuff and pick the one that works best". To me, that feels weird and worrying. Its like we don't yet understand ML properly yet to definitively say, for a given data set, what sort of model we'll need. This can lead us down the debugging black-hole TFA talks about since we appear to have zero-clue about why we chose something, so debugging something ultimately might just be "opps - we chose 3 layers of 10, 15, and 11 nodes, instead of 3 layers of 10, 15 and 12 nodes! D'oh! Lets start training again!" It really grates me to think about this considering how much maths and proofs and algorithms get thrown at you when being taught ML, then to be told when it comes to actually doing something its all down to "intuition" (guessing). And yeah as others have said - data :-)
- jps359 10y ago
  Well, generally, we know why certain models work on certain types of data. But picking a proper model and hyperparameters requires some insight/knowledge about the properties of the data itself, which is the whole purpose of doing ML experiments in the first place (to gain knowledge/insight about the data)! However I do agree that we don't fully understand the area yet or the reach of its applications.
- makmanalp 10y ago
  > just try a load of stuff and pick the one that works best Well, yeah, that's easier to say than "read this entire book to understand how machine learning models work". The truth is that different models work with different success for different kinds of data. How many predictors do I have? How many observations vs how many predictors? How much data overall? Do I care about interpretability? Am I doing classification or regression? If doing regression, what is my tolerance for error? How much time am I willing to spend training and predicting? If doing classification, are my classes balanced or imbalanced? How are the classes spread? Do I care more about overall errors or false positives / false negatives (think of a cancer screening test - you might rather have false positives than miss something with false negatives). All of these factors, and more, go into play when selecting a model. There's other bits too. Selecting variables: including unneeded or highly correlated variables can actually worsen the result. Feature engineering: Turns out you might be able to process your variables in such a way that the algorithm picks up on the details you need much more easily. Cross validation: You don't want to "overfit" your training data, i.e. get a model that is super specific to your training data such that when encountering actual data points, it has worse accuracy because it's not general enough. Hyperparameter tuning: Usually a lot of these models have tuning parameters that you can tweak, that are hard to know what the values should be from the get go, and you have to try a bunch and look at the response curve of how the accuracy changes. So yeah, machine learning is not magic. Turns out there are different tools for different problems. We do have some models like Random Forests and SVMs that work fairly well out of the box on a wide variety of problems, and some kinds of neural networks also do well but often need more data and processing time to get decent results. It's all a tradeoff :)
dschiptsov 10y ago
Because most of the models are flawed or wrong.
strictfp 10y ago
People don't use it simply because it's not what they signed up for. It's not obvious that a software engineer will enjoy being a data scientist. I for one think it's tedious, and I don' t enjoy spending time and effort collecting the necessary data to solve my problems the ML way.
bitL 10y ago
I will be a little sarcastic - using sub-optimal/locally-optimal algorithms everywhere in ML due to time complexity, why would you expect it would bring nice/predictable results? It's more like a miracle if you find something that works, otherwise you will be hitting the usual hard problems from optimization and end up in a "catch-as-you-can" situation where even Monte Carlo randomness is a good guess. And way too many people assume ML is just applied statistics, always keeping their minds in this frame and missing out on the large data ML capabilities where statistics is irrelevant and you can directly ask and find answers to many fundamental questions in your dataset.
BickNowstrom 10y ago
I actually think Machine Learning is relatively easy. There are a lot of resources, the community is very open, state-of-the-art tools are available, and all it needs to get incrementally better is trying out more stuff on different data sets. I worked in SEO before, which had far more elements of "black magic". Perhaps SEO helps with the transition to ML, because you are basically reverse engineering a model (Google's search engine) / crafting input to get a higher ranked output. It's feature engineering, experimentation, and debugging all-in-one. And front-end development of the old days... debugging old javascript or IE6 render bugs makes ML debugging pale in comparison. You had to make a broken model work, without being able to repair it. As for the long debugging cycles in ML. John Langford coined "sub-linear debugging": Output enough intermediate information to quickly know if you introduced a major bug or hit upon a significant improvement [1]. Machine learning competitions are not so much won by skill, but by the teams iterating faster and more efficiently: Those who try more (failed) experiments hit upon more successful experiments. No Neural Net researcher should let all nets finish training, before drawing conclusions/estimates on learning process. Sure, the ML field is relatively new, and computer programming has a longer history of proper debugging and testing. It is difficult to do monitoring on feedback-looped models running in production, yet no more difficult than control theory ;). And proper practices are being developed as we speak [2]. The author will probably write a randomization script to avoid malordered samples automatically in the future. [1] http://www.machinedlearnings.com/2013/06/productivity-is-about-not-waiting.html http://www.machinedlearnings.com/2013/06/productivity-is-abo... [2] http://research.google.com/pubs/pub43146.html http://research.google.com/pubs/pub43146.html
eva1984 10y ago
So there is a greedy strategy to approach problem with ML 1.Starts with a VANILLA model, a proven one. To establish a baseline you can fall back on. For example, in deep learning, starts with fully-connected nets, then vanilla CNN, adding BNs and ReLus, then residual connections, etc. 2.Do not spend too much time tuning hyperparmeters, especially in the field of deep learning, once you change the your algorithm, a.k.a network structure, everything changes. 3.Adding complexity as you go. It is important once you established some solid baseline, then you can start add more fancy ideas into your stack, and you will find, fancy ideas are improvement over the already working ideas, and it is not that hard to add it. 4.One important tip to remind, once you change your algorithm, as time goes, those changes might not happy with each other. So reduction is also very important. Rethink your approach from time to time, take away stuff that didn't fit anymore. 5.Look At Your Data. Garbage in, Garbage out. Cannot be more true. Really. Look at your data, maybe a sample of it, see whether youself, as the most intelligent being as of yet, can make sense of it or not. If you cannot, then you need to probably improve the quality. Anyway, ML is a very complex field and developing like crazy, but I didn't feel the methodology to tackle it is any different from any other complex problems. It is a iterative processes starting from simple prove solutions to something greater, piece by piece. Watch and think then improve.
fucking_idiot 10y ago
mostly because the implementation is really tough - mostly lots of matricies and calculus. i recommend using sklearn.

327 points by Dawny33 10y ago

That's right, machine learning requires knowledges of so many fields, so if any problem occurs, developer has to do so so many check to find the problem, and optimize it.

curiousgal 10y ago
Exactly why I despise "ML Crash course" and "Learn ML in x hours" kind of courses.
- dzhiurgis 10y ago
  It gets worse. There's YouTube channel with uploads like: Build a Neural Net in 4 Minutes Build an Antivirus in 5 Min Build a Self Driving Car in 5 Min
  - walrus 10y ago
    Have you watched any of them? I just watched "Build a Self Driving Car in 5 Min" and the content was good. My only complaint is that his demo used end-to-end learning, which isn't how most self-driving cars actually work (but he acknowledges this).
    - Drdrdrq 10y ago
      Link: https://www.youtube.com/watch?v=hBedCdzCoWM https://www.youtube.com/watch?v=hBedCdzCoWM
- taneq 10y ago
  Cue a link to the essay, "Teach Yourself Programming In Ten Years".

peatfreak 10y ago

Why would you expect it to be easy?

Kirth 10y ago
I've met some people whom believe that the more modern tools (such as Tensorflow) magically require little human input and make it so that you do not need to know/understand the mathematics and statistics. Not sure where they get this idea. Everyone wants to do machine learning, but nobody seems to want to learn statistics.
- an_account 10y ago
  How would you recommend learning ML? I've been interested but don't know where to start.
  - kevinwang 10y ago
    Depending on your situation, this advice may be useless, but I'd say for me personally, taking an intro to machine learning class at my university is a wonderful and easy way to learn about the basics. So I'd recommend taking a university class in-person, or online.
    - Scea91 10y ago
      This is the best answer. In the end nothing is better than a degree in the field.
  - adamnemecek 10y ago
    I put together a list of resources I found useful: https://news.ycombinator.com/item?id=12900448 https://news.ycombinator.com/item?id=12900448
  - blahi 10y ago
    Not who you asked, but: Statistics in Plan English. Regression Analysis by Gilman Elements of Statistical Learning. You need to throw some matrix algebra and calc 1 & 2 somewhere in between. Certainly before ESL. It would also require You can't simply read the books and go through the examples. You will be stuck at a concept at many occasions and you will battle it out until after much googling and reading additional papers, you finally get. After those 3 books, you've got the basics.

godmodus 10y ago

because of preprocessing and needing to choose the functions that'll do the approximation - the process itself is semi-automatic, not fully automatic. ANN's inner nodes are specific functions that need parameter tweaking (after choosing the right ones that is) - vector machines have different kinds for different data, etc. and those two things are very domain specific so you need to do a lot of homework first, and debugging later.

tmptmp 10y ago

Beautifully written and insightful. >>After much trial and error I eventually learned that this is often the case of a training set that has not been correctly randomized and is a problem when you are using stochastic gradient algorithms that process the data in small batches. Take this single term from the above sentence: "stochastic gradient algorithms", they represent three key areas: statistics, calculus and CS. These three things, even when they are to be studied in isolation are much complex. For ML, you must be able to juggle these 3 fireballs effectively. No surprise, it's much, much more difficult than many other software engineering problems.

ttub 10y ago
No offense intended, but what is your background? I'm a researcher with a physics/stats PhD, and if a colleague approached me and said "stochastic gradient algorithms" entails three highly complex areas of scientific knowledge, I would have been stunned and assumed an undergrad with an English major had stumbled into our lab. Just because you find something extremely challenging, doesn't mean it is inherently challenging. Considering what a lot of people in my field is struggling with, your example is absolutely trivial. You might want to adjust your ego downwards a bit.
- DavidSJ 10y ago
  He didn't say he found it personally challenging. He said this part of optimization finds itself at the intersection of three deep fields of human knowledge. That is undeniably correct.
  - partykid92 10y ago
    Thats like saying bubblesort lies at the intersection of combinatorics, topology and complexity theory. True, but pretentious
    - Al-Khwarizmi 10y ago
      I don't think it was pretentious, and your analogy is quite off. You don't need to understand any of those fields to grok bubblesort (algorithmics is enough). You do need to know some statistics, calculus and algorithmics to understand stochastic gradient descent.

blt 10y ago

I think to some extent, this kind of difficulty occurs with any numerical programming. With typical software engineering the bugs are usually in logic. All but the hardest problems can be diagnosed with a visual debugger. With numerical code, though, you usually can't look at the state of a few discrete-valued variables and a stack trace to figure out what went wrong. "Why is this matrix singular?" can take days to answer. You spend a lot of time staring at the code, comparing it to the math on paper, trying to visualize high-dimensional intermediate data, etc. Continuous math can be a lot harder to reason about than discrete.

simonbarker87 10y ago
This was it for me. I don't have a CS background but with PhD in electronics/semiconductors I can usually brute force my way through things and get a good grasp. I got half way through Andrew Ng's ML course and felt I fully understood the concepts but spent all of my time battling with matrices and the foundation mathematics rather than building on ML understanding. If I had a better grasp of the maths then I think I'd get on better with it.
nabla9 10y ago
It's even worse than that. Talking from personal experience in the private sector: You go trough the whole process and present your results for a customer. They make the required changes, see few percent improvement in the bottom line and are happy. Year later you pick up the same code for another project and discover small error in data collection script that completely invalidates everything. Garbage in, garbage out errors can have internal consistency that survive cross validation and provide similar results with many different algorithms and models. Random changes in the real word can produce actual gains. Standard machine learning datasets rarely suffer from this problem because results and semantics of the problem are known beforehand. If you have a original black-box problem, it's possible to do random search and improve by accident.
boulos 10y ago
tl;dr: Agreed, but the problems and models ML folks play with are more complex than most physics-like systems. Having spent a lot of time doing Monte Carlo rendering, I can say that "why is this pixel messed up" can be a painful affair (and that domain is at least visual!). Bad input geometry, NaNs or Infs because you forgot to handle some 1/r^2 term, etc. occur in all numerical computing. Some of this is actually obviated in TensorFlow the same way it is in much of numerical computing: frameworks that have been battle hardened through years of bug fixes. This lets you at least focus on "Did I describe my model correctly?" rather than "Did I implement gradient descent correctly too?". Again, rendering (and all physical simulation) has the benefit of producing a picture and the inputs are effectively "physical". The challenge for machine learning is that you're trying to optimize for figuring out the underlying model rather than relying on the laws of physics for your problem. While this is just as true in any numerical modeling problem ("Should this be a 4th degree polynomial or just cubic?"), the models that ML folks put together are effectively quite complex and the input datasets are both very large and usually unstructured. The visualization of the network being trained in the TF Playground (http://playground.tensorflow.org http://playground.tensorflow.org) helps with the model but not the original space.

partykid92 10y ago

one big dimension here, the "implementation error" can be easily be debugged. Gradients can be checked numerically. The model can be checked to work by looking at the optimality conditions (not just the loss function go down). This shouldn't be an issue for anyone from a traditional coding background.

edblarney 10y ago

I've worked in computational linguistics: 1) It's 'hard' because you need a lot of 'training data' in order to train models etc.. It's hard to get. 2) 'AI' type interfaces represent a whole new kind of UI challenge. For 'predictive typing' for example, you can optimize an algorithm so that it does better for 90% of the US population, but then it gets 'worse' for the remaining 10%. So it's a paradox. This can have weird effects. For example, if you have an app in the app-store, you may leave the settings so that it's 'broadly optimal'. You get ok stars. If you then make it 'better' for those 90%, you might get a little boost in ratings, but you get 1 and 0 star ratings from the 10% for whom it's a sub-par experience. This can destroy your product. Anyhow - 'there is no right answer' often in AI, and setting expectations can be extremely difficult. And all of that has nothing even to do with CS.

Kenji 10y ago
That reminds me of Stephen Hawking's computer. His old computer gave deterministic dictionary predictions and after some time, he had memorized all the predictions that came up, so he could be much quicker because he knew exactly how much of the word to type and then to select the proper word - a purely mechanical process that doesn't require feedback anymore. However, when they gave him a new computer, it had adaptive predictions (I think) and Hawkins could not memorize them anymore. Then he always had to type letter for letter and inspect the predictions before he could select one - a huge step down. Stuff like that is absolutely crucial but is often forgotten by engineers.
- kixiQu 10y ago
  Seconded, and yet it's madness to me that engineers don't pick up on this. I can't be the only one who's let someone use my computer for basic internet browsing and then been deeply irritated when their newly visited sites break my intuitive knowledge of how many characters I need to type in the URL bar before hitting enter...
mingyeow 10y ago
"And all of that has nothing even to do with CS." -> therein lies the kicker! Consumer level (including sass) apps require very real user and business tradeoffs. That tradeoff is rarely discussed, which i honestly find very curious in and of itself.

ramblenode 10y ago

The diagrams are pretty misleading. One can craft a space with any number of arbitrary dimensions, but that's kind of meaningless until the space is populated with data. Certainly the likelihood of a bug is not uniformly distributed across the space, and certainly the density of bugs within a space varies greatly depending on the problem. I imagine the average kernel developer's 2-D space is both very dense and has greater spread than the 4-D space of many ML engineers.

atomical 10y ago

I'm getting a 404.

a_bonobo 10y ago
Me too, the whole blog 404s Here's the cached version: https://webcache.googleusercontent.com/search?q=cache:https://ai.stanford.edu/~zayd/why-is-machine-learning-hard.html https://webcache.googleusercontent.com/search?q=cache:https:...
dgellow 10y ago
try the HTTP (non SSL) version. I had to disable HTTPSEverywhere

jupp0r 10y ago

There are also lots of architectural problems with machine learning components to consider, beautifully summarized in "Machine Learning: The High Interest Credit Card of Technical Debt" by Sculley et al. http://research.google.com/pubs/pub43146.html http://research.google.com/pubs/pub43146.html

DrNuke 10y ago

Generally speaking, being hot in the media does not help: walking the walk is way harder than talking the talk.

Matthias247 10y ago

I think once you have the need to go deep enough into a topic they all get hard. Debugging and testing is also hard in all things that are somehow related to realtime or concurrency. E.g. OS development, embedded firmware, network stacks, etc. For these things you often also need to know about Math, Physics, Statistics, Electronics, Hardware and Software Architecture, etc. Game engine development is also hard because you should also know about most of this stuff to really find the most efficient solutions.

norswap 10y ago

> Machine learning often boils down to the art of developing an intuition for where something went wrong (or could work better) when there are many dimensions of things that could go wrong (or work better). I'm not a practitioner, but I always thought this was the main challenge. Uses of ML are rarely "right" or "wrong" per se, but they rely on intuition to get a model that "works" in a practical sense. There is no royal way to machine learning: you can't decide you are going to make an algorithm that detects bad comments (as determined by human consensus) and then just go make an implementation that you can reason out to be correct, the way you could prove a graph algorithm correct. Trial-and-error and hard-to-transcribe intuition are baked into the process. (I'd love to get some insider insight on this comment!)

Dzugaru 10y ago

Even if you develop an "intuition" for known tasks (like classification) - there are so many problems that are not tackled yet and no one has any "intuition" yet. Common sense very often doesn't work there (in high-dimensional spaces ;)). For example I've only recently stumbled upon an "Explaining and harnessing adversarial examples" article - and that completely changed my perception about my current work in computer vision.

js8 10y ago

I think it is hard because of https://en.wikipedia.org/wiki/No_free_lunch_theorem https://en.wikipedia.org/wiki/No_free_lunch_theorem That follows there is no single "good" algorithm, and you need to have and exploit domain knowledge in order to succeed.

jorgemf 10y ago

In software you can track the program and detect what instruction is not doing what it is suppose to do. In machine learning there is no program to track, it is not a set of instructions with a purpose, the whole thing either works or not. In order to discover what could be failing, you need to have a deep knowledge about a lot of stuff (maths, statistics, CS) to figure out what is wrong. And sometimes the answer is that the problem doesn't have a solution.

fnl 10y ago
It is very much possible to track your calculations and quite literally debug your models. But like in computer science, it is hard to find the data scientist who can actually do that and not just copy-paste tutorial code from some blog post and then wonder what isn't working...

deleted 10y ago

[deleted]

aflam 10y ago

The long and convoluted debugging cycle for machine learning really hurts my faith in ML models. This issue - with some practical advice - was the center of this interview (disclaimer: author). https://shapescience.xyz/blog/interview-data-science-methodology/ https://shapescience.xyz/blog/interview-data-science-methodo... I'm convinced we lack decent tools for ML debugging: what could they be?

mattlondon 10y ago

I think the problem is that we don't really understand ML properly yet. Like picking hyperparamters - time and time again I've asked experts/trainers/colleagues: "How do I know what type of model to use? How many layers? How many nodes per layer? Dropout or not?" etc etc And the answer is always along the lines of "just try a load of stuff and pick the one that works best". To me, that feels weird and worrying. Its like we don't yet understand ML properly yet to definitively say, for a given data set, what sort of model we'll need. This can lead us down the debugging black-hole TFA talks about since we appear to have zero-clue about why we chose something, so debugging something ultimately might just be "opps - we chose 3 layers of 10, 15, and 11 nodes, instead of 3 layers of 10, 15 and 12 nodes! D'oh! Lets start training again!" It really grates me to think about this considering how much maths and proofs and algorithms get thrown at you when being taught ML, then to be told when it comes to actually doing something its all down to "intuition" (guessing). And yeah as others have said - data :-)

jps359 10y ago
Well, generally, we know why certain models work on certain types of data. But picking a proper model and hyperparameters requires some insight/knowledge about the properties of the data itself, which is the whole purpose of doing ML experiments in the first place (to gain knowledge/insight about the data)! However I do agree that we don't fully understand the area yet or the reach of its applications.
makmanalp 10y ago
> just try a load of stuff and pick the one that works best Well, yeah, that's easier to say than "read this entire book to understand how machine learning models work". The truth is that different models work with different success for different kinds of data. How many predictors do I have? How many observations vs how many predictors? How much data overall? Do I care about interpretability? Am I doing classification or regression? If doing regression, what is my tolerance for error? How much time am I willing to spend training and predicting? If doing classification, are my classes balanced or imbalanced? How are the classes spread? Do I care more about overall errors or false positives / false negatives (think of a cancer screening test - you might rather have false positives than miss something with false negatives). All of these factors, and more, go into play when selecting a model. There's other bits too. Selecting variables: including unneeded or highly correlated variables can actually worsen the result. Feature engineering: Turns out you might be able to process your variables in such a way that the algorithm picks up on the details you need much more easily. Cross validation: You don't want to "overfit" your training data, i.e. get a model that is super specific to your training data such that when encountering actual data points, it has worse accuracy because it's not general enough. Hyperparameter tuning: Usually a lot of these models have tuning parameters that you can tweak, that are hard to know what the values should be from the get go, and you have to try a bunch and look at the response curve of how the accuracy changes. So yeah, machine learning is not magic. Turns out there are different tools for different problems. We do have some models like Random Forests and SVMs that work fairly well out of the box on a wide variety of problems, and some kinds of neural networks also do well but often need more data and processing time to get decent results. It's all a tradeoff :)

dschiptsov 10y ago

Because most of the models are flawed or wrong.

strictfp 10y ago

People don't use it simply because it's not what they signed up for. It's not obvious that a software engineer will enjoy being a data scientist. I for one think it's tedious, and I don' t enjoy spending time and effort collecting the necessary data to solve my problems the ML way.

bitL 10y ago

I will be a little sarcastic - using sub-optimal/locally-optimal algorithms everywhere in ML due to time complexity, why would you expect it would bring nice/predictable results? It's more like a miracle if you find something that works, otherwise you will be hitting the usual hard problems from optimization and end up in a "catch-as-you-can" situation where even Monte Carlo randomness is a good guess. And way too many people assume ML is just applied statistics, always keeping their minds in this frame and missing out on the large data ML capabilities where statistics is irrelevant and you can directly ask and find answers to many fundamental questions in your dataset.

BickNowstrom 10y ago

I actually think Machine Learning is relatively easy. There are a lot of resources, the community is very open, state-of-the-art tools are available, and all it needs to get incrementally better is trying out more stuff on different data sets. I worked in SEO before, which had far more elements of "black magic". Perhaps SEO helps with the transition to ML, because you are basically reverse engineering a model (Google's search engine) / crafting input to get a higher ranked output. It's feature engineering, experimentation, and debugging all-in-one. And front-end development of the old days... debugging old javascript or IE6 render bugs makes ML debugging pale in comparison. You had to make a broken model work, without being able to repair it. As for the long debugging cycles in ML. John Langford coined "sub-linear debugging": Output enough intermediate information to quickly know if you introduced a major bug or hit upon a significant improvement [1]. Machine learning competitions are not so much won by skill, but by the teams iterating faster and more efficiently: Those who try more (failed) experiments hit upon more successful experiments. No Neural Net researcher should let all nets finish training, before drawing conclusions/estimates on learning process. Sure, the ML field is relatively new, and computer programming has a longer history of proper debugging and testing. It is difficult to do monitoring on feedback-looped models running in production, yet no more difficult than control theory ;). And proper practices are being developed as we speak [2]. The author will probably write a randomization script to avoid malordered samples automatically in the future. [1] http://www.machinedlearnings.com/2013/06/productivity-is-about-not-waiting.html http://www.machinedlearnings.com/2013/06/productivity-is-abo... [2] http://research.google.com/pubs/pub43146.html http://research.google.com/pubs/pub43146.html

eva1984 10y ago

So there is a greedy strategy to approach problem with ML 1.Starts with a VANILLA model, a proven one. To establish a baseline you can fall back on. For example, in deep learning, starts with fully-connected nets, then vanilla CNN, adding BNs and ReLus, then residual connections, etc. 2.Do not spend too much time tuning hyperparmeters, especially in the field of deep learning, once you change the your algorithm, a.k.a network structure, everything changes. 3.Adding complexity as you go. It is important once you established some solid baseline, then you can start add more fancy ideas into your stack, and you will find, fancy ideas are improvement over the already working ideas, and it is not that hard to add it. 4.One important tip to remind, once you change your algorithm, as time goes, those changes might not happy with each other. So reduction is also very important. Rethink your approach from time to time, take away stuff that didn't fit anymore. 5.Look At Your Data. Garbage in, Garbage out. Cannot be more true. Really. Look at your data, maybe a sample of it, see whether youself, as the most intelligent being as of yet, can make sense of it or not. If you cannot, then you need to probably improve the quality. Anyway, ML is a very complex field and developing like crazy, but I didn't feel the methodology to tackle it is any different from any other complex problems. It is a iterative processes starting from simple prove solutions to something greater, piece by piece. Watch and think then improve.

fucking_idiot 10y ago

mostly because the implementation is really tough - mostly lots of matricies and calculus. i recommend using sklearn.