Considering that Agent Mode saves me a lot of hassle doing refactoring ("move the handler to file X and review imports", "check where this constant is used and

Considering that Agent Mode saves me a lot of hassle doing refactoring ("move the handler to file X and review imports", "check where this constant is used and replace it with <another> for <these cases>", etc.), I'd say you are missing the point... I actually flip things - I do the breakdown myself in a SPEC.md file and then have the agent work through it. Markdown checklists work great, and the agent can usually update/expand them as it goes.