Yes, and robots.txt was reasonable because it was created in the early days of the web. But the others don’t have any excuse (and they were warned about it as s

Yes, and robots.txt was reasonable because it was created in the early days of the web. But the others don’t have any excuse (and they were warned about it as soon as they were announced and ignored people).