3 ms·
I hope you leave it on the WAF. If they're only just deciding to respect robots.txt, which has been internet infrastructure forever, then it's probably still in
by BLKNSLVR 6mo ago
I hope you leave it on the WAF. If they're only just deciding to respect robots.txt, which has been internet infrastructure forever, then it's probably still incredibly amateur software with 'Amazon-priorities' rather than 'responsible internet traffic' priorities.
- tardedmeme 6mo agoThe responsible internet is dead. Every big actor on the internet is selfish now that there's money involved. And has been for 20 years. Google only respected it because blocking Google from crawling your site used to hurt you more than it hurt Google.
- BLKNSLVR 6mo agoTime to switch to allow lists instead of block lists...