4 ms·

I'll be the odd man out. No and on. 1) One of the key things in control systems is to deal with uncertainty. If you have a hundred trains, none of them should

by blagie 3y ago

I'll be the odd man out. No and on. 1) One of the key things in control systems is to deal with uncertainty. If you have a hundred trains, none of them should crash. You want a control system which you can understand to work even in corner cases. ML is bad at that. 2) You need to train an ML. Unless you have a very good model of the thing you're controlling (generating which is usually harder than designing an optimal control system), training an RL means crashing airplanes, burning out amplifiers, or destroying self-driving cars as the system "learns." Even so, when moving from model to reality, it's likely you didn't capture some oddball part of the system. 3) In many cases, there is some give-and-take between control system design and system design. In an extreme case, the PID can be purely mechanical or electrical (e.g. an op-amp). In a less extreme case, if you can design the mechanical system to avoid having e.g. a right-half-plane singularity, controls becomes easier. RL isn't at the level where you can do that kind of codesign. 4) It's a lot more complex. So "in general," no. In some specific cases, yes. RL is mostly beneficial for rather complex systems, such as walking robots or anytime a word like "nonholonomic" comes up. For relatively linear, relatively low-dimensional systems, classical methods will give better results for less effort.

blagie 3y ago
One more -- and this comes up a lot -- anything where latency or a delay matters. An analog circuit can have response times in the tens or hundreds of picoseconds. A digital PID on something like an FPGA in the nanoseconds. One on a basic microcontroller (the kind without an OS -- PIC, AVR, ESP32, etc.), in the microseconds. An RL algorithm running on a general-purpose computer, you'll be lucky to hit tens of milliseconds. That makes a big difference. For those unfamiliar with control systems, delay matters a lot. Something like an audio amplifier (where you can't hear anything <20kHz) will generally have a bandwidth in the tens of megahertz, and more is better. A rule-of-thumb is: https://en.wikipedia.org/wiki/Gain%E2%80%93bandwidth_product https://en.wikipedia.org/wiki/Gain%E2%80%93bandwidth_product In other words, if you'd like to bring down error by a factor of 10,000, your loop will generally like to be around 10,000x faster than whatever it's controlling. If something has a time constant of 1 second, you still might want to be able to react at the speed of microseconds. That's not a hard-and-fast rule per se (I can give all sorts of ways to work around it), but what is true is the more general principle that minimizing delays can be very, very helpful (even for things which otherwise seem pretty slow).

by blagie 3y ago

blagie 3y ago

One more -- and this comes up a lot -- anything where latency or a delay matters. An analog circuit can have response times in the tens or hundreds of picoseconds. A digital PID on something like an FPGA in the nanoseconds. One on a basic microcontroller (the kind without an OS -- PIC, AVR, ESP32, etc.), in the microseconds. An RL algorithm running on a general-purpose computer, you'll be lucky to hit tens of milliseconds. That makes a big difference. For those unfamiliar with control systems, delay matters a lot. Something like an audio amplifier (where you can't hear anything <20kHz) will generally have a bandwidth in the tens of megahertz, and more is better. A rule-of-thumb is: https://en.wikipedia.org/wiki/Gain%E2%80%93bandwidth_product https://en.wikipedia.org/wiki/Gain%E2%80%93bandwidth_product In other words, if you'd like to bring down error by a factor of 10,000, your loop will generally like to be around 10,000x faster than whatever it's controlling. If something has a time constant of 1 second, you still might want to be able to react at the speed of microseconds. That's not a hard-and-fast rule per se (I can give all sorts of ways to work around it), but what is true is the more general principle that minimizing delays can be very, very helpful (even for things which otherwise seem pretty slow).