Wizards are physically weak, have finite spells to use and require solitude and rest to regain them. Living in a tower gives them substantial physical security

Wizards are physically weak, have finite spells to use and require solitude and rest to regain them. Living in a tower gives them substantial physical security. You can have one point of entrance at ground level that is significantly guarded. A typical house has numerous points of entry. Not as big of a deal for a fighter that can defend him/herself at a moment’s notice but much riskier for a wizard.