The only difference in power consumption is in the reduced toggling of the IO pins and the reduced effort of the CPU (but that's assuming the CPU will downclock

The only difference in power consumption is in the reduced toggling of the IO pins and the reduced effort of the CPU (but that's assuming the CPU will downclock or downvolt automatically, which is unlikely for a microcontroller.) In an LED panel, the vast majority of power consumption is in lighting up the LEDs.