4 ms·

I get more than a little annoyed with the advertising-like tone in this and several other articles around this general class of method/trick for speeding up com

by jofer 7y ago

I get more than a little annoyed with the advertising-like tone in this and several other articles around this general class of method/trick for speeding up complex simulations. It's not that it's not a useful method (it is). It's that it misrepresents the utility of the general "build an exact simulation, then train a regressor on it to make fast approximations" approach. It's very useful in certain situations (repeated calculations on similar parameters) and completely useless in others. The key issue is that most of the slow models you'd want to use this on are highly non-linear. In certain regions of the parameter space, very small changes in input result in very large changes in output. This is fine, so long as you know where all of these regions are and can capture them in your training data. That's easier for some problems than others. Even assuming you do know how/where to collect dense training data, this approach is only useful within the bounds of the training data you collect. It's relatively uncommon (but not super rare) that you want to repeatedly run a complex simulation within the same parameter space. This method is great when you do want/need to do that, and useless otherwise. You have to understand that what you've trained is little more than a look up table. Anything that claims it can actually learn highly non-linear and irregular behavior well outside of the training dataset's bounds is snake oil. Understand where this general class of technique is useful and where it isn't and ignore overblown claims. The entire abstract here is overblown hogwash. The actual paper is relatively interesting, but I really wish folks would drop the absurd advertising language and focus on what distinguishes this from the hundreds of very similar studies/methods on this topic.

jofer 7y ago
Sorry for the rant... I just feel like abstracts are the most important part of a paper and get annoyed with poor ones. A better abstract would arguably be something they have a few paragraphs into their introduction: "To construct high fidelity emulators with limited training data, the machine learning models need to have a good prior on the simulation models. Most work to date in building emulators using random forests, Gaussian Processes,or other machine learning models, do not fully capture the correlation among the output points,limiting their accuracy in emulating simulations with one, two, or three-dimensional output signals. On the other hand, convolutional neural network (CNN) have shown to have a good prior on natural signals, making them suitable for processing natural n-dimensional signals. However, as the CNN priors inherently rely on their architectures,8one has to find an architecture that gives the suitable prior of a given problem. Manually searching for the right architecture can take a significant amount of time and domain-specific expertise and often produces sub-optimal results. Here we propose to solve this problem by employing efficient neural architecture search to simultaneously find the neural network architecture that is best suited for a given case and train it. With the efficient neural architecture search and a novel super-architecture presented in this work, the algorithm can find and train fast emulators for a wide range of applications while offering major improvements in terms of accuracy compared with other techniques, even when the training data is limited."
marmaduke 7y ago
Without being cynical, I think the authors surely know this. Writing this way is exploiting the DL fad for career stepping stones. This is simply how science works these days: a lot of good science is done but those climbing latter boast about it or lose out.

by jofer 7y ago

jofer 7y ago

Sorry for the rant... I just feel like abstracts are the most important part of a paper and get annoyed with poor ones. A better abstract would arguably be something they have a few paragraphs into their introduction: "To construct high fidelity emulators with limited training data, the machine learning models need to have a good prior on the simulation models. Most work to date in building emulators using random forests, Gaussian Processes,or other machine learning models, do not fully capture the correlation among the output points,limiting their accuracy in emulating simulations with one, two, or three-dimensional output signals. On the other hand, convolutional neural network (CNN) have shown to have a good prior on natural signals, making them suitable for processing natural n-dimensional signals. However, as the CNN priors inherently rely on their architectures,8one has to find an architecture that gives the suitable prior of a given problem. Manually searching for the right architecture can take a significant amount of time and domain-specific expertise and often produces sub-optimal results. Here we propose to solve this problem by employing efficient neural architecture search to simultaneously find the neural network architecture that is best suited for a given case and train it. With the efficient neural architecture search and a novel super-architecture presented in this work, the algorithm can find and train fast emulators for a wide range of applications while offering major improvements in terms of accuracy compared with other techniques, even when the training data is limited."

marmaduke 7y ago

Without being cynical, I think the authors surely know this. Writing this way is exploiting the DL fad for career stepping stones. This is simply how science works these days: a lot of good science is done but those climbing latter boast about it or lose out.