I don't even calculate primary residence into net worth. Don't think it's relevant compared to other assets that can be liquidated.
I don't either for the same reason as you. Like having one car, if I get rid of the roof over my head or my main mode of transportation then I'll still need to find another one. I know others see them as investments and that's fine with me, but I just don't see it the same.
Now....people who McMansion themselves with extravagant possessions are a whole 'nother story but it's their money and I have no say in what they do with their money and vise versa.