I am a total newbie in economics. Stupid question: But what would hinder to put a very high tax for any habitable property if not the primary owner living in i

I am a total newbie in economics. Stupid question: But what would hinder to put a very high tax for any habitable property if not the primary owner living in it as their primary home? Of course people can have secondary homes, maybe to a lower tax and rented appartment would benefit from a reduction if rental price moderate. Wouldn’t this just solve any speculation and make investments from « unequal« parties uninteresting?