I think an interesting way to address this impact, is to start adding more time to each task and address it as technical debt, which is both informative and act

I think an interesting way to address this impact, is to start adding more time to each task and address it as technical debt, which is both informative and actually true. For example after a few months of no refactoring, you start adding 10% to all estimations based on technical debt. It grows more the more refactoring is kept at bay. Similar to how we add a certain percentage to each task for communication, bug tracking, qa, scope changes or anything else. You can keep an arbitrary count, and perhaps even have a "technical debt" reduction amount for certain backlog tasks. I haven't ever tried this though. But it's an interesting way to communicate technical debt (and is very close to the metaphor).