5 ms·

The last time I was reminded of the bitter lesson was when I read about Guidance & Control Networks, after seeing them used in an autonomous drone that beat the

The last time I was reminded of the bitter lesson was when I read about Guidance & Control Networks, after seeing them used in an autonomous drone that beat the best human FPV pilots [0]. Basically it's using a small MLP (Multi Layer Perceptron) on the order of 200 parameters, and using the drone's state as input and controlling the motors directly with the output. We have all kinds of fancy control theory like MCP (Model Predictive Control), but it turns out that the best solution might be to train a relatively tiny NN using a mix of simulation and collected sensor data instead. It's not better because of huge computation resources, it's actually more computationally efficient than some classic alternatives, but it is more general. [0] https://www.tudelft.nl/en/2025/lr/autonomous-drone-from-tu-delft-defeats-human-champions-in-historic-racing-first https://www.tudelft.nl/en/2025/lr/autonomous-drone-from-tu-d... https://www.nature.com/articles/s41586-023-06419-4 https://www.nature.com/articles/s41586-023-06419-4 https://arxiv.org/abs/2305.13078 https://arxiv.org/abs/2305.13078 https://arxiv.org/abs/2305.02705 https://arxiv.org/abs/2305.02705

logicchains 1y ago
>It's not better because of huge computation resources, it's actually more computationally efficient than some classic alternatives It's similar with options pricing. The most sophisticated models like multivariate stochastic volatility are computationally expensive to approximate with classical approaches (and have no closed form solution), so just training a small NN on the output of a vast number of simulations of the underlying processes ends up producing a more efficient model than traditional approaches. Same with stuff like trinomial trees.
- William_BB 1y ago
  Interesting. Are these models the SOTA in the options trading industry (e.g. MM) nowadays?
- cactusfrog 1y ago
  This is really interesting. I think force fields in molecular dynamics have underwent a similar NN revolution. You train your NN on the output of expensive calculations to replace the expensive function with a cheap one. Could you train a small language model with a big one?
  - lossolo 1y ago
    > Could you train a small language model with a big one? Yes, it's called distillation.
- jwjohnson314 1y ago
  Can you elaborate on this? I am curious what processes are being simulated to feed the network.
bsoles 1y ago
But I have also seen people trying to use deep networks to identify rotating machinery faults, like bearings, from raw accelerometer data collected a high frequencies like 40 kHz. Whereas the spectrum of the data from running FFT on the signal contains fault information much more obviously and clearly. Throwing a deep network on a problem without some physical insight into the problem has also its disadvantages it seems.
- jvanderbot 1y ago
  Yeah, we're shouting into the wind here. I have had people tell me directly that my ideas from old school state estimation were irrelevant in the era of deep learning. They may produce (in this case) worse results, but the long game I'm assured is superior. The specific scenario was estimating the orientation of a stationary semi trailer. An objectively measurable number and it was consistently off by 30 deg, yet I was the jerk for suggesting we move from end to end DL to trad Bar Shalom techniques. That scene isn't for me anymore.
  - AnonymousPlanet 1y ago
    > That scene isn't for me anymore. They will learn. At least when the competition beats their solution with a hybrid approach they can't begin to understand.
- physicsguy 1y ago
  I’m working on this sort of thing right now in a SaaS product that previously didn’t have support for vibration data. One competitor is all ML-d up to the hilt but customers don’t like the black box and keep finding it gives false positives with no explanation. I think one problem is those buying not understanding problem; they just want to plug in a sensor and insights to happen, but without information about the machine that’s never going to be able to provide useful insights
ssivark 1y ago
> an autonomous drone that beat the best human FPV pilots Doesn’t any such claim come with huge caveats — pre specified track/course, no random objects flying between, etc…? ie. train & test distributions are ensured same by ensuring test time can never be more complicated than training data. Also presumably better sensing than raw visual input.
bsoles 1y ago
There are also extremely misleading research articles out there promising good results with deep networks in the area of anomaly detection, without adequate comparison with more classical techniques. This well-known critical paper shows examples of AI articles/techniques applied to popular datasets with good-looking results. But, it also demonstrates that, literally, a single line of MATLAB code can outperform some of these techniques: https://arxiv.org/pdf/2009.13807 https://arxiv.org/pdf/2009.13807
chrisweekly 1y ago
> "MCP (Model Predictive Control)" ^ that's MPC. (MCP = Model Context Protocol)

logicchains 1y ago

>It's not better because of huge computation resources, it's actually more computationally efficient than some classic alternatives It's similar with options pricing. The most sophisticated models like multivariate stochastic volatility are computationally expensive to approximate with classical approaches (and have no closed form solution), so just training a small NN on the output of a vast number of simulations of the underlying processes ends up producing a more efficient model than traditional approaches. Same with stuff like trinomial trees.

William_BB 1y ago
Interesting. Are these models the SOTA in the options trading industry (e.g. MM) nowadays?
cactusfrog 1y ago
This is really interesting. I think force fields in molecular dynamics have underwent a similar NN revolution. You train your NN on the output of expensive calculations to replace the expensive function with a cheap one. Could you train a small language model with a big one?
- lossolo 1y ago
  > Could you train a small language model with a big one? Yes, it's called distillation.
jwjohnson314 1y ago
Can you elaborate on this? I am curious what processes are being simulated to feed the network.

bsoles 1y ago

But I have also seen people trying to use deep networks to identify rotating machinery faults, like bearings, from raw accelerometer data collected a high frequencies like 40 kHz. Whereas the spectrum of the data from running FFT on the signal contains fault information much more obviously and clearly. Throwing a deep network on a problem without some physical insight into the problem has also its disadvantages it seems.

jvanderbot 1y ago
Yeah, we're shouting into the wind here. I have had people tell me directly that my ideas from old school state estimation were irrelevant in the era of deep learning. They may produce (in this case) worse results, but the long game I'm assured is superior. The specific scenario was estimating the orientation of a stationary semi trailer. An objectively measurable number and it was consistently off by 30 deg, yet I was the jerk for suggesting we move from end to end DL to trad Bar Shalom techniques. That scene isn't for me anymore.
- AnonymousPlanet 1y ago
  > That scene isn't for me anymore. They will learn. At least when the competition beats their solution with a hybrid approach they can't begin to understand.
physicsguy 1y ago
I’m working on this sort of thing right now in a SaaS product that previously didn’t have support for vibration data. One competitor is all ML-d up to the hilt but customers don’t like the black box and keep finding it gives false positives with no explanation. I think one problem is those buying not understanding problem; they just want to plug in a sensor and insights to happen, but without information about the machine that’s never going to be able to provide useful insights

ssivark 1y ago

> an autonomous drone that beat the best human FPV pilots Doesn’t any such claim come with huge caveats — pre specified track/course, no random objects flying between, etc…? ie. train & test distributions are ensured same by ensuring test time can never be more complicated than training data. Also presumably better sensing than raw visual input.

bsoles 1y ago

There are also extremely misleading research articles out there promising good results with deep networks in the area of anomaly detection, without adequate comparison with more classical techniques. This well-known critical paper shows examples of AI articles/techniques applied to popular datasets with good-looking results. But, it also demonstrates that, literally, a single line of MATLAB code can outperform some of these techniques: https://arxiv.org/pdf/2009.13807 https://arxiv.org/pdf/2009.13807

chrisweekly 1y ago

> "MCP (Model Predictive Control)" ^ that's MPC. (MCP = Model Context Protocol)