I run an above board crawler, but in general crawler traps are relatively easy to work around, especially coming from a big target like Cloudflare, where it rea

I run an above board crawler, but in general crawler traps are relatively easy to work around, especially coming from a big target like Cloudflare, where it really pays off to build a specialized workaround that fingerprints and avoids the trap. Cloudflare's strength is arguably that they have enough traffic data they don't have to rely on stuff like this, they can gather statistics and identify bot patterns in ways smaller actors can not. It's trickier when you have 10,000 different webmasters inventing their own solutions to do sabotage crawlers, where the juice isn't worth the squeeze when it comes to implementing individual workarounds.