The biggest reason I can think of not to close stderr (or any other of the standard handles) is that the next open(2) call is likely to get that same descriptor

The biggest reason I can think of not to close stderr (or any other of the standard handles) is that the next open(2) call is likely to get that same descriptor recycled. So now joe random open file is going to receive all the error messages from random libraries, or perhaps even from unrelated programs you may have forked with that file as fd 2.