3 ms·

@uwukko's full thread for those who don't have a Twitter account: earlier today i noticed very elevated traffic to cobalt api that looked a lot like ddos. it t

@uwukko's full thread for those who don't have a Twitter account: earlier today i noticed very elevated traffic to cobalt api that looked a lot like ddos. it turned out to be bytedance! we can't tell what videos they were downloading or where the original request comes from as it's built to go around all limiters, but there's still a pattern first request: json post with content url & settings from a residential proxy second request: tunnel with pseudo microsoft edge on windows user agent & youtube origin/referer, from byteplus ip third request: same tunnel with aria2 user agent & no referer, also from byteplus ip cobalt is a media downloader, mostly known for supporting youtube even at worst times. cobalt's tunnel is either a proxy stream or ffmpeg live render considering all of this, i can safely assume that bytedance was scraping youtube videos by abusing our private api with release of v10 we implemented cloudflare turnstile, but later disabled it due to access issues by a chunk of our users enabling it back brought the server load to normal levels and stopped bytedance from choking our servers cuz they didn't account for this (yet) before resorting to turnstile, i attempted using other cloudflare services, but none of them seemed to help much my theory is that bytedance's scraper was specifically built to go around cloudflare & other web security solutions, which is just genuinely evil this incident caused a few minutes of api unavailability, but taught me that cobalt (and probably anything else) can no longer exist without active bot/scraping protection im really glad that cloudflare turnstile exists because i don't know what i'd do without it here byteplus AS that was spamming requests is 150436 and last seen ip range was 207.166.160.0/21 the amount of unique users on cloudflare analytics rapidly increased by 2.25 times and didn't go down since, while web analytics (plausible) show no increase whatsoever

gnfargbl 2y ago
Sounds like they're using residential proxies for set-up in order to look like normal users, but then switching back to their own ASN for content because residential proxies are expensive. > im really glad that cloudflare turnstile exists because i don't know what i'd do without it here Why not just blackhole the byteplus ASN?
- sandworm101 2y ago
  And how many of those residential IPs belong to work-from-home bytedance employees running work laptops? Any large company these days has direct access to a pool of innocent residential IPs. The weaponization of that pool may be more evil than the actual scraping imho.
miki123211 2y ago
Bytedance seems to have increased its scraping efforts significantly. I've posted a canary token[1] URL as a Mastodon post, to check how scrape-resistant Mastodon actually is (it is not resistant at all), and have been getting quite a few hits from the ByteDance spider recently. Last hit is from 47.128.114.151, Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com) Edit: added missing footnote. [1] https://canarytokens.org https://canarytokens.org
- diggan 2y ago
  > to check how scrape-resistant Mastodon actually is (it is not resistant at all) That's expected, no? It's a social network that is explicitly designed to be as open as possible, as it's using ActivityPub. To be "scraping resisting" would be to go against the very goal of Mastodon.
  - Aachen 2y ago
    Exactly, this is how I want it to be. I post there because it's not another walled garden that profits from lock-in
  - miki123211 2y ago
    Yes and no. If you look at the technical side of things, you're absolutely right. If you look at the social side, however, there's a lot of talk on there about opting out of scraping, scrapers being bad, not wanting to be part of AI training and so on. Naming-and-shaming people who have been caught scraping is a routine practice. I think that many Mastodonians believe that defederating from scraper-friendly instances and blocking scraper-like requests on their own protects them, this was a way to show that this very much isn't true.
deleted 2y ago
[deleted]

gnfargbl 2y ago

Sounds like they're using residential proxies for set-up in order to look like normal users, but then switching back to their own ASN for content because residential proxies are expensive. > im really glad that cloudflare turnstile exists because i don't know what i'd do without it here Why not just blackhole the byteplus ASN?

sandworm101 2y ago
And how many of those residential IPs belong to work-from-home bytedance employees running work laptops? Any large company these days has direct access to a pool of innocent residential IPs. The weaponization of that pool may be more evil than the actual scraping imho.

miki123211 2y ago

Bytedance seems to have increased its scraping efforts significantly. I've posted a canary token[1] URL as a Mastodon post, to check how scrape-resistant Mastodon actually is (it is not resistant at all), and have been getting quite a few hits from the ByteDance spider recently. Last hit is from 47.128.114.151, Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com) Edit: added missing footnote. [1] https://canarytokens.org https://canarytokens.org

diggan 2y ago
> to check how scrape-resistant Mastodon actually is (it is not resistant at all) That's expected, no? It's a social network that is explicitly designed to be as open as possible, as it's using ActivityPub. To be "scraping resisting" would be to go against the very goal of Mastodon.
- Aachen 2y ago
  Exactly, this is how I want it to be. I post there because it's not another walled garden that profits from lock-in
- miki123211 2y ago
  Yes and no. If you look at the technical side of things, you're absolutely right. If you look at the social side, however, there's a lot of talk on there about opting out of scraping, scrapers being bad, not wanting to be part of AI training and so on. Naming-and-shaming people who have been caught scraping is a routine practice. I think that many Mastodonians believe that defederating from scraper-friendly instances and blocking scraper-like requests on their own protects them, this was a way to show that this very much isn't true.

deleted 2y ago

[deleted]