3 ms·
This. Bytedance's official spider has a clear User-Agent tagged Bytespider, but OP didn't mention what they're seeing.
by teractiveodular 2y ago
This. Bytedance's official spider has a clear User-Agent tagged Bytespider, but OP didn't mention what they're seeing.
- jsheard 2y agoThis isn't spider traffic though, the traffic pattern indicates that it's a special-purpose bot designed to hit Cobalts internal API in particular. A generic spider probably wouldn't even be able to find the API endpoints that are only referenced by Javascript, nevermind consistently hit the API with a valid video URL from a residential proxy then switch to a different IP address to download the result every time.