Sure but then the file protections also apply to the programs used by the agent. If you can strictly limit the agent to only modifying files in the source code

Sure but then the file protections also apply to the programs used by the agent. If you can strictly limit the agent to only modifying files in the source code directory of a single repository, that greatly limits the blast radius of damage.