Refactor AS you continue ahead when you notice specifically bad code/design issues. Also if you find performance issues when profiling. Don't let refactoring

Refactor AS you continue ahead when you notice specifically bad code/design issues. Also if you find performance issues when profiling. Don't let refactoring be your primary task, keep it on a secondary track but running along side your primary task.