Is there really performance to gain? I doubt a well configured OS is slowing the CPU down. The alternative could be GPU like devices: A PCI-E card with sockets

Is there really performance to gain? I doubt a well configured OS is slowing the CPU down. The alternative could be GPU like devices: A PCI-E card with sockets for the chip and some RAM running it's own firmware. Intel tried that with it's Xeon Phi line and they opted to run Linux (called uOS) on the accelerator board, so apparently that was never an issue.