3 ms·
I'm skeptical of the claim that it's easier to train a neural net than to hand-code a set of heuristics _when the heuristics are already known_. For the time be
by YeGoblynQueenne 7y ago
I'm skeptical of the claim that it's easier to train a neural net than to hand-code a set of heuristics _when the heuristics are already known_. For the time being, optimal results with neural nets need more data and more computing power ("more" because it's never enough) and are primarily useful when a hand-coded solution is not possible.
I also don't understand how it is possible for a neural net (or any approximator, really) to approximate a "precision heuristic" faster than a hand-coded heuristic and without a gross loss of well, precision in the order that would make the results unusable for engineering or scientific tasks. Could you elaborate?
- richk449 7y agoI’m also skeptical, but after reading the explanation above, I am intrigued. Say I have a cube with 100 x 100 x 100 mesh cells inside, and ports on opposing faces. Given enough time, I can literally run through every possible combination of PEC and air for every cell and solve the FD form of maxwells equations, then save the results. Now, a user can ask my solver for any of those cases, and I simply pull the presolved result, and give the user the answer with orders of magnitude reduction in time. Obviously, the presolving approach doesn’t scale. More materials, more mesh cells, eventually it is impractical to presolve every case. But the beauty of neural networks is that they can be very good at generalizing from a partial sample of the problem space. In effect, they can give results close enough to the presolve solution with drastically reduced numbers of computations.
- YeGoblynQueenne 7y ago>> But the beauty of neural networks is that they can be very good at generalizing from a partial sample of the problem space. That is really not the case. Neural nets generalise very poorly, hence the need for ever larger amounts of data: to overcome their lack of generalisation by attempting to cover as many "cases" as possible. Edit: when this subject comes up I cite the following article, by François Chollet, maintainer of Keras: The limitations of deep learning https://blog.keras.io/the-limitations-of-deep-learning.html https://blog.keras.io/the-limitations-of-deep-learning.html I quote from the article: This stands in sharp contrast with what deep nets do, which I would call "local generalization": the mapping from inputs to outputs performed by deep nets quickly stops making sense if new inputs differ even slightly from what they saw at training time. Consider, for instance, the problem of learning the appropriate launch parameters to get a rocket to land on the moon. If you were to use a deep net for this task, whether training using supervised learning or reinforcement learning, you would need to feed it with thousands or even millions of launch trials, i.e. you would need to expose it to a dense sampling of the input space, in order to learn a reliable mapping from input space to output space.
- richk449 7y agoIt doesn’t need to generalize, just do sophisticated interpolation. Basing the results on a dense sampling of the input space is exactly what I was suggesting.
- YeGoblynQueenne 7y agoApologies for the misunderstanding. You said "generalizing from a partial sample of the problem space" and I thought you meant generalisation to unseen data from few examples, which is generally what we would all like to get from machine learnig models (but don't). But, if a neural net can't _extrapolate_ to unseen instances, I don't see how it can solve problems like the one you describe with any useful precision, again unless it's trained with gigantic amounts of examples (which you say is not required). And how is this reducing computational costs with respect to hand-coded solvers?
- richk449 7y agoTo be clear - I have absolutely no experience in this domain. I'm just speculating. In the example I gave, everyone agrees that if you had long enough and enough processing power, you could solve every possible configuration, and store the results. Then you could instantaneously "solve" any problem. Unfortunately, the problem I describe is a toy problem (too simple to be useful), and yet it would still take way way too long to solve all the possible configurations. What if you solved some tiny fraction of the configurations though? That would be a sampling of the configuration space. Then a neural network could use that sampling to interpolate to the cases not solved. That would provide a significant speedup over actually solving the problem. So the real question is what density you need to pre-solve the configuration space to make it work? It definitely depends on what accuracy you need in the solution, as well as how good you can do with the interpolation. If I said previously that gigantic numbers of examples are not needed, then I misspoke. I am sure they would be needed. Gigantic is vague though - is it the kind of number that can be rented from AWS, or is it the kind of number that would require civilization resources? I have no idea if the math actually works out to make it a useful approach. All I am saying is that conceptually I can see that in some cases, it could be possible.
- allovernow 7y agoWell, there are pretty convincing examples in other domains: try hardcoding rules to classify animals or objects in photos, especially an algorithm which can handle thousands of different categories. Totally impractical - but if we appropriately design the net and structure the training data, you can train a pretty accurate net on a mid-range GPU in a matter of hours to do what would take far, far longer to hardcode! Perhaps not quite appropriate to call them heuristics in this context, but the principle is the same - you are leveraging joint probabilities of pixels to generate some conditional output. Similar principle in ML accelerated modeling.
- YeGoblynQueenne 7y agoI think I understand what you meant by heuristics. I agree that it's impractical to try and hand-code image recognition rules and all attempts to do that in the past have failed as they have in similarly complex domains (like machine translation, say). My concern is particularly about the use of neural networks (or in general machine learning models that learn to approximate a function) in domains where precision is normally required, like engineering. I mean, I know there's plenty of approximation in engineering already but of course we're not talking about computing integrals here (er, I think?). Anyway I was especially trying to understand the OP's comment about speedup using a neural network. I'm still a bit confused about that. But thanks for the conversation.