4 ms·

I used to do electromagnetic modeling using finite element methods (though now a product manager for AI software infra) and it would to take me on the order of

I used to do electromagnetic modeling using finite element methods (though now a product manager for AI software infra) and it would to take me on the order of hours to days or weeks to model wave interaction with real-world objects. A machine learning model trained to understand Maxwell's Equations can in principle be used perform said simulations, resulting in probably an order or more of magnitude increase in simulation speed. Getting this to work well will reduce the time (and cost) it takes to design optical sensors, radar for autonomous vehicles, smartphone antennas, MRI machines, and more. Having said that, it would require a lot of heaving lifting to pull this off to achieve near-physical accuracy for real-world physics problems. A cursory search on Google for "arxiv deep learning electromagnetics" returns results of proofs of concept in this direction.

allovernow 7y ago
You're on the right track. A lot of this tech is a potential goldmine and I'm sure there are many players developing in secret and not publishing yet (or ever).
YeGoblynQueenne 7y ago
Were would the speedup come from? I don't understand. If I understand your comment correctly, essentially you have a hand-crafted simulator for some physical process and then you train a neural net model to approximate the simulator. Why would the approximated simulator have "an order or more of magnitude increase in simulation speed"? Unless the approximation has massive losses in accuracy, of course. Honestly asking and really interested to know what you mean.
- allovernow 7y ago
  It's all about precision heuristics, derived from joint probabilities of inputs and outputs. That, by and large, is how I am increasingly coming to understand the power of neural networks. Imagine you are given a picture of a candle, overlaid with a grid, and asked to fill in, with colored pencils, colors for the air surrounding the candle representing relative temperature. Of course a human utilizes intuition to rapidly assign high temperature to the flame and decreasing temperature with increasing distance. A "dumb" finite method would need, even for such a relatively simple problem (for a human), to perform calculations for a series of time steps in each grid until some steady state condition to arrive at a much more precise but still overall similar coloring of the grid cells. You can do the same task much more quickly because you have developed intuition of the physics, which is to say you have learned heuristics which capture the general trends of the problem (air is hot close to a flame and cold far away). Neural nets take the best of both worlds - by effectively learning probability relationships between input and output pixels, they internalize heuristic approaches to produce outputs approaching finite method accuracies at a fraction of the computation. There's a lot of waste that can be optimized out of finite computation by hardcoding rules (heuristics), but doing so for real problems is impractical. Neural nets learn these rules through training - a far simpler task is organizing the data to teach the net the right trends; much like designing lessons for a child to teach a predictive ability.
  - YeGoblynQueenne 7y ago
    I'm skeptical of the claim that it's easier to train a neural net than to hand-code a set of heuristics _when the heuristics are already known_. For the time being, optimal results with neural nets need more data and more computing power ("more" because it's never enough) and are primarily useful when a hand-coded solution is not possible. I also don't understand how it is possible for a neural net (or any approximator, really) to approximate a "precision heuristic" faster than a hand-coded heuristic and without a gross loss of well, precision in the order that would make the results unusable for engineering or scientific tasks. Could you elaborate?
    - richk449 7y ago
      I’m also skeptical, but after reading the explanation above, I am intrigued. Say I have a cube with 100 x 100 x 100 mesh cells inside, and ports on opposing faces. Given enough time, I can literally run through every possible combination of PEC and air for every cell and solve the FD form of maxwells equations, then save the results. Now, a user can ask my solver for any of those cases, and I simply pull the presolved result, and give the user the answer with orders of magnitude reduction in time. Obviously, the presolving approach doesn’t scale. More materials, more mesh cells, eventually it is impractical to presolve every case. But the beauty of neural networks is that they can be very good at generalizing from a partial sample of the problem space. In effect, they can give results close enough to the presolve solution with drastically reduced numbers of computations.
      - YeGoblynQueenne 7y ago
        >> But the beauty of neural networks is that they can be very good at generalizing from a partial sample of the problem space. That is really not the case. Neural nets generalise very poorly, hence the need for ever larger amounts of data: to overcome their lack of generalisation by attempting to cover as many "cases" as possible. Edit: when this subject comes up I cite the following article, by François Chollet, maintainer of Keras: The limitations of deep learning https://blog.keras.io/the-limitations-of-deep-learning.html https://blog.keras.io/the-limitations-of-deep-learning.html I quote from the article: This stands in sharp contrast with what deep nets do, which I would call "local generalization": the mapping from inputs to outputs performed by deep nets quickly stops making sense if new inputs differ even slightly from what they saw at training time. Consider, for instance, the problem of learning the appropriate launch parameters to get a rocket to land on the moon. If you were to use a deep net for this task, whether training using supervised learning or reinforcement learning, you would need to feed it with thousands or even millions of launch trials, i.e. you would need to expose it to a dense sampling of the input space, in order to learn a reliable mapping from input space to output space.

allovernow 7y ago

You're on the right track. A lot of this tech is a potential goldmine and I'm sure there are many players developing in secret and not publishing yet (or ever).

YeGoblynQueenne 7y ago

Were would the speedup come from? I don't understand. If I understand your comment correctly, essentially you have a hand-crafted simulator for some physical process and then you train a neural net model to approximate the simulator. Why would the approximated simulator have "an order or more of magnitude increase in simulation speed"? Unless the approximation has massive losses in accuracy, of course. Honestly asking and really interested to know what you mean.

allovernow 7y ago
It's all about precision heuristics, derived from joint probabilities of inputs and outputs. That, by and large, is how I am increasingly coming to understand the power of neural networks. Imagine you are given a picture of a candle, overlaid with a grid, and asked to fill in, with colored pencils, colors for the air surrounding the candle representing relative temperature. Of course a human utilizes intuition to rapidly assign high temperature to the flame and decreasing temperature with increasing distance. A "dumb" finite method would need, even for such a relatively simple problem (for a human), to perform calculations for a series of time steps in each grid until some steady state condition to arrive at a much more precise but still overall similar coloring of the grid cells. You can do the same task much more quickly because you have developed intuition of the physics, which is to say you have learned heuristics which capture the general trends of the problem (air is hot close to a flame and cold far away). Neural nets take the best of both worlds - by effectively learning probability relationships between input and output pixels, they internalize heuristic approaches to produce outputs approaching finite method accuracies at a fraction of the computation. There's a lot of waste that can be optimized out of finite computation by hardcoding rules (heuristics), but doing so for real problems is impractical. Neural nets learn these rules through training - a far simpler task is organizing the data to teach the net the right trends; much like designing lessons for a child to teach a predictive ability.
- YeGoblynQueenne 7y ago
  I'm skeptical of the claim that it's easier to train a neural net than to hand-code a set of heuristics _when the heuristics are already known_. For the time being, optimal results with neural nets need more data and more computing power ("more" because it's never enough) and are primarily useful when a hand-coded solution is not possible. I also don't understand how it is possible for a neural net (or any approximator, really) to approximate a "precision heuristic" faster than a hand-coded heuristic and without a gross loss of well, precision in the order that would make the results unusable for engineering or scientific tasks. Could you elaborate?
  - richk449 7y ago
    I’m also skeptical, but after reading the explanation above, I am intrigued. Say I have a cube with 100 x 100 x 100 mesh cells inside, and ports on opposing faces. Given enough time, I can literally run through every possible combination of PEC and air for every cell and solve the FD form of maxwells equations, then save the results. Now, a user can ask my solver for any of those cases, and I simply pull the presolved result, and give the user the answer with orders of magnitude reduction in time. Obviously, the presolving approach doesn’t scale. More materials, more mesh cells, eventually it is impractical to presolve every case. But the beauty of neural networks is that they can be very good at generalizing from a partial sample of the problem space. In effect, they can give results close enough to the presolve solution with drastically reduced numbers of computations.
    - YeGoblynQueenne 7y ago
      >> But the beauty of neural networks is that they can be very good at generalizing from a partial sample of the problem space. That is really not the case. Neural nets generalise very poorly, hence the need for ever larger amounts of data: to overcome their lack of generalisation by attempting to cover as many "cases" as possible. Edit: when this subject comes up I cite the following article, by François Chollet, maintainer of Keras: The limitations of deep learning https://blog.keras.io/the-limitations-of-deep-learning.html https://blog.keras.io/the-limitations-of-deep-learning.html I quote from the article: This stands in sharp contrast with what deep nets do, which I would call "local generalization": the mapping from inputs to outputs performed by deep nets quickly stops making sense if new inputs differ even slightly from what they saw at training time. Consider, for instance, the problem of learning the appropriate launch parameters to get a rocket to land on the moon. If you were to use a deep net for this task, whether training using supervised learning or reinforcement learning, you would need to feed it with thousands or even millions of launch trials, i.e. you would need to expose it to a dense sampling of the input space, in order to learn a reliable mapping from input space to output space.