Why would major refactoring be insane when collaborating? We do this regularly in my team in early stages of projects and only when really required once project

Why would major refactoring be insane when collaborating? We do this regularly in my team in early stages of projects and only when really required once projects become stable. But these projects are usually not about crypto, so I might miss the difference here.