Caddy doesn't support CGI out of the box. You need to use a module (which requires a custom build of it and the hope the module author continues to maintain it

Caddy doesn't support CGI out of the box. You need to use a module (which requires a custom build of it and the hope the module author continues to maintain it), or you need to run something like fcgiwrap to convert FCGI (what Caddy does support) to CGI (and now you need a process manager to make sure your fcgiwrap and caddy processes are kept running, restarted on failure, etc).