Yes, search engines were not hiding, as it was website owner interest involved here as well - without those search bots their sites would not be indexed and sea

Yes, search engines were not hiding, as it was website owner interest involved here as well - without those search bots their sites would not be indexed and searchable in the Internet. So there was kind of win-win situation, in most typical cases at least, as for instance publishers complained about deep links, etc. because their ads revenue was hurt. AI scrapping bots provide zero value for sites owners.