3 ms·

is it possible to replace PID with reinforcement learning and get better results in general?

by tw1984 3y ago

317070 3y ago
Generally yes and yes. But you will need _a lot_ more data and compute for reinforcement learning. So RL is usually too cumbersome for relatively little gain. Source: I'm a roboticist working with RL.
- moffkalast 3y ago
  Yeah I doubt there's much point without insane amounts of data, and if you do have the extra onboard compute a particle filter can probably leverage it more effectively.
HPsquared 3y ago
Better results in general, with the occasional unexplainable glitch.
- elteto 3y ago
  “Control rods go brrrrrr”
blagie 3y ago
I'll be the odd man out. No and on. 1) One of the key things in control systems is to deal with uncertainty. If you have a hundred trains, none of them should crash. You want a control system which you can understand to work even in corner cases. ML is bad at that. 2) You need to train an ML. Unless you have a very good model of the thing you're controlling (generating which is usually harder than designing an optimal control system), training an RL means crashing airplanes, burning out amplifiers, or destroying self-driving cars as the system "learns." Even so, when moving from model to reality, it's likely you didn't capture some oddball part of the system. 3) In many cases, there is some give-and-take between control system design and system design. In an extreme case, the PID can be purely mechanical or electrical (e.g. an op-amp). In a less extreme case, if you can design the mechanical system to avoid having e.g. a right-half-plane singularity, controls becomes easier. RL isn't at the level where you can do that kind of codesign. 4) It's a lot more complex. So "in general," no. In some specific cases, yes. RL is mostly beneficial for rather complex systems, such as walking robots or anytime a word like "nonholonomic" comes up. For relatively linear, relatively low-dimensional systems, classical methods will give better results for less effort.
- blagie 3y ago
  One more -- and this comes up a lot -- anything where latency or a delay matters. An analog circuit can have response times in the tens or hundreds of picoseconds. A digital PID on something like an FPGA in the nanoseconds. One on a basic microcontroller (the kind without an OS -- PIC, AVR, ESP32, etc.), in the microseconds. An RL algorithm running on a general-purpose computer, you'll be lucky to hit tens of milliseconds. That makes a big difference. For those unfamiliar with control systems, delay matters a lot. Something like an audio amplifier (where you can't hear anything <20kHz) will generally have a bandwidth in the tens of megahertz, and more is better. A rule-of-thumb is: https://en.wikipedia.org/wiki/Gain%E2%80%93bandwidth_product https://en.wikipedia.org/wiki/Gain%E2%80%93bandwidth_product In other words, if you'd like to bring down error by a factor of 10,000, your loop will generally like to be around 10,000x faster than whatever it's controlling. If something has a time constant of 1 second, you still might want to be able to react at the speed of microseconds. That's not a hard-and-fast rule per se (I can give all sorts of ways to work around it), but what is true is the more general principle that minimizing delays can be very, very helpful (even for things which otherwise seem pretty slow).
greggsy 3y ago
It was interesting, and while I will never be in a situation where I’ll have to actually do the maths, I think it’s important to understand that it exists. The whole concept never really ‘clicked’ until I watched some demonstrations, like this [1] video. After seeing that one in particular, I realised that it’s everywhere… not everything is just a robotic on/off, or a directly proportional response based on feedback from some input. [1] https://youtu.be/qKy98Cbcltw https://youtu.be/qKy98Cbcltw
fumplethumb 3y ago
One thing I see missing from some other replies to your comment is a discussion of determinism. That is given the same input, always produce the same output. Determinism is often valued in safety critical systems. The system can't work _sometimes_. The system can't even work correctly _eventually_. It must produce the desired result in the time allotted. Actuating the thing 20ms too late could be as bad doing it incorrectly depending on the application. I'm no ML expert, but that seems to be a problem for those types of solutions in some spaces.

is it possible to replace PID with reinforcement learning and get better results in general?

by tw1984 3y ago

is it possible to replace PID with reinforcement learning and get better results in general?

Generally yes and yes. But you will need _a lot_ more data and compute for reinforcement learning. So RL is usually too cumbersome for relatively little gain. Source: I'm a roboticist working with RL.

moffkalast 3y ago
Yeah I doubt there's much point without insane amounts of data, and if you do have the extra onboard compute a particle filter can probably leverage it more effectively.

HPsquared 3y ago

Better results in general, with the occasional unexplainable glitch.

elteto 3y ago
“Control rods go brrrrrr”

blagie 3y ago

I'll be the odd man out. No and on. 1) One of the key things in control systems is to deal with uncertainty. If you have a hundred trains, none of them should crash. You want a control system which you can understand to work even in corner cases. ML is bad at that. 2) You need to train an ML. Unless you have a very good model of the thing you're controlling (generating which is usually harder than designing an optimal control system), training an RL means crashing airplanes, burning out amplifiers, or destroying self-driving cars as the system "learns." Even so, when moving from model to reality, it's likely you didn't capture some oddball part of the system. 3) In many cases, there is some give-and-take between control system design and system design. In an extreme case, the PID can be purely mechanical or electrical (e.g. an op-amp). In a less extreme case, if you can design the mechanical system to avoid having e.g. a right-half-plane singularity, controls becomes easier. RL isn't at the level where you can do that kind of codesign. 4) It's a lot more complex. So "in general," no. In some specific cases, yes. RL is mostly beneficial for rather complex systems, such as walking robots or anytime a word like "nonholonomic" comes up. For relatively linear, relatively low-dimensional systems, classical methods will give better results for less effort.

blagie 3y ago
One more -- and this comes up a lot -- anything where latency or a delay matters. An analog circuit can have response times in the tens or hundreds of picoseconds. A digital PID on something like an FPGA in the nanoseconds. One on a basic microcontroller (the kind without an OS -- PIC, AVR, ESP32, etc.), in the microseconds. An RL algorithm running on a general-purpose computer, you'll be lucky to hit tens of milliseconds. That makes a big difference. For those unfamiliar with control systems, delay matters a lot. Something like an audio amplifier (where you can't hear anything <20kHz) will generally have a bandwidth in the tens of megahertz, and more is better. A rule-of-thumb is: https://en.wikipedia.org/wiki/Gain%E2%80%93bandwidth_product https://en.wikipedia.org/wiki/Gain%E2%80%93bandwidth_product In other words, if you'd like to bring down error by a factor of 10,000, your loop will generally like to be around 10,000x faster than whatever it's controlling. If something has a time constant of 1 second, you still might want to be able to react at the speed of microseconds. That's not a hard-and-fast rule per se (I can give all sorts of ways to work around it), but what is true is the more general principle that minimizing delays can be very, very helpful (even for things which otherwise seem pretty slow).

greggsy 3y ago

It was interesting, and while I will never be in a situation where I’ll have to actually do the maths, I think it’s important to understand that it exists. The whole concept never really ‘clicked’ until I watched some demonstrations, like this [1] video. After seeing that one in particular, I realised that it’s everywhere… not everything is just a robotic on/off, or a directly proportional response based on feedback from some input. [1] https://youtu.be/qKy98Cbcltw https://youtu.be/qKy98Cbcltw

fumplethumb 3y ago

One thing I see missing from some other replies to your comment is a discussion of determinism. That is given the same input, always produce the same output. Determinism is often valued in safety critical systems. The system can't work _sometimes_. The system can't even work correctly _eventually_. It must produce the desired result in the time allotted. Actuating the thing 20ms too late could be as bad doing it incorrectly depending on the application. I'm no ML expert, but that seems to be a problem for those types of solutions in some spaces.