The preferential treatment is kind of annoying TBH. I've seen sites disappear from Google's listing for months as a result of simple issues such as a change of

The preferential treatment is kind of annoying TBH. I've seen sites disappear from Google's listing for months as a result of simple issues such as a change of domain name (with all the required redirects in place). Yet here, a website owner is purposely blocking the crawler and they jump with solutions to try to fix the problem. Sigh.