> "a fair bit of work" for them to read the standard robots.txt for "boring technical reasons". that's embarrassing. What other reasons should there ever be?

> "a fair bit of work" for them to read the standard robots.txt for "boring technical reasons". that's embarrassing. What other reasons should there ever be? Stop taking their money, stop using their shit. Turn your attention to where it's beneficial.