I don't think there are $1m houses in their area. My guess is that the value of a $2m house in that area is land $1.5m, structure $500k. Even if they downsize

I don't think there are $1m houses in their area. My guess is that the value of a $2m house in that area is land $1.5m, structure $500k. Even if they downsize the house cost by half, and buy in a smaller lot, the minimum house cost is still $1.5m. I think the one element that the OP is missing is that after the transaction, he is better off by about $130k in tax due to the increase in basis of his home. If he eventually sells and doesn't buy another home in that area, he will realize an additional $130k in profits on the second home that would be lost to tax if he had not sold the first house.