From the point of view of a little web site owner, Google certainly has a weird behavior. It comes several times per day (sometimes 20 times per day) even if a

From the point of view of a little web site owner, Google certainly has a weird behavior. It comes several times per day (sometimes 20 times per day) even if a robot.txt tells to come only every 7 days (it's a blog with news every four days on average). Petalsearch (Huawei) indexing rate is half of Google, Apple and Bing half of Petalsearch, Yandex half of Bing. The other search engines respect the robot.txt. I wonder how much energy and Internet capacity is wasted because of Google and PetalSearch indexing.