3 ms·

Meanwhile... > For robots.txt, we do follow the same restrictions applied to googlebot, otherwise Google benefits from its dominant position. https://communit

Meanwhile... > For robots.txt, we do follow the same restrictions applied to googlebot, otherwise Google benefits from its dominant position. https://community.brave.com/t/stop-website-being-shown-in-brave-search/495774/3 https://community.brave.com/t/stop-website-being-shown-in-br...

scrum-treats 3y ago
It makes sense. It's not great but it does make sense. Also, do big crawlers even observe robots.txt?
- deleted 3y ago
  [deleted]
GhostWhisperer 3y ago
ianal but this reads like unauthorised access would it fall under the cfaa?