> Robots.txt allows website owners to block or permit bots on a case-by-case basis. While there’s no legal obligation for bots to adhere. I don't know how long

> Robots.txt allows website owners to block or permit bots on a case-by-case basis. While there’s no legal obligation for bots to adhere. I don't know how long it's going to work like that, internet is basically the farewest. If you can't paywall your data, someone is free to take it. Beside I bet most article written on these website used artificial intelligence to do so.