3 ms·

"Overall, we attempted to download 130M videos and achieved a link success rate of approximately 60%, resulting in 80M successfully retrieved videos with a tota

"Overall, we attempted to download 130M videos and achieved a link success rate of approximately 60%, resulting in 80M successfully retrieved videos with a total duration of 10M hours." I am astonished that the success rate is so high. How Youtube didn't block them, I don't know. But I think that this URL list won't age well because youtube will very quickly block any researcher trying to download these videos themselves.

ks2048 1mo ago
I believe they have the actual videos to download, "exclusively for academic and non-commercial research." and you have to request access, https://projects.laion.ai/bvd/bvd-raw-access.html https://projects.laion.ai/bvd/bvd-raw-access.html They say they used yt-dlp and "employ a residential proxy network". But, yeah, youtube seems to now more aggressively block yt-dlp.
- kevmo314 1mo ago
  That’s a pretty penny on the bandwidth bill alone!
- kennyadam 1mo ago
  Ha, is it just me or does a “residential proxy network” sound like a fancy way of saying “botnet”?
  - m00dy 1mo ago
    Nope, they’re different things. Proxy networks like Proxybase [0] use open-source clients and ask for the user’s consent before allowing them to join the network. [0] https://proxybase.xyz https://proxybase.xyz
  - MallocVoidstar 1mo ago
    Basically all of them are botnets, yes. They generally claim to have consent but I'm pretty sure 99% of it is "some app the user uses has it buried in a 200 page ToS"-style consent.
    - m00dy 1mo ago
      No, they are not. Just ask any AI.
      - MallocVoidstar 1mo ago
        The companies say they have consent because saying "yeah we'll give you access to a botnet" is illegal. https://cloud.google.com/blog/topics/threat-intelligence/disrupting-largest-residential-proxy-network https://cloud.google.com/blog/topics/threat-intelligence/dis... >These SDKs, which are offered to developers across multiple mobile and desktop platforms, surreptitiously enroll user devices into the IPIDEA network. If I install an app with a big ToS and buried in there is "we use third-party monetization SDKs" and buried in there is "welcome to our botnet" they do not actually have my consent.
  - stymaar 1mo ago
    It's the "legal version" of botnets, yes.
- maybewhenthesun 1mo ago
  > youtube seems to now more aggressively block yt-dlp. Another thing to thank AI for! /s
ks2048 1mo ago
I was wondering if youtube blocked them a lot and they got a lot of videos from other sources - but no, you can see on this HF page that 93.1% of URls are youtube.com, https://huggingface.co/datasets/laion/BVD-URLs https://huggingface.co/datasets/laion/BVD-URLs

ks2048 1mo ago

I believe they have the actual videos to download, "exclusively for academic and non-commercial research." and you have to request access, https://projects.laion.ai/bvd/bvd-raw-access.html https://projects.laion.ai/bvd/bvd-raw-access.html They say they used yt-dlp and "employ a residential proxy network". But, yeah, youtube seems to now more aggressively block yt-dlp.

kevmo314 1mo ago
That’s a pretty penny on the bandwidth bill alone!
kennyadam 1mo ago
Ha, is it just me or does a “residential proxy network” sound like a fancy way of saying “botnet”?
- m00dy 1mo ago
  Nope, they’re different things. Proxy networks like Proxybase [0] use open-source clients and ask for the user’s consent before allowing them to join the network. [0] https://proxybase.xyz https://proxybase.xyz
- MallocVoidstar 1mo ago
  Basically all of them are botnets, yes. They generally claim to have consent but I'm pretty sure 99% of it is "some app the user uses has it buried in a 200 page ToS"-style consent.
  - m00dy 1mo ago
    No, they are not. Just ask any AI.
    - MallocVoidstar 1mo ago
      The companies say they have consent because saying "yeah we'll give you access to a botnet" is illegal. https://cloud.google.com/blog/topics/threat-intelligence/disrupting-largest-residential-proxy-network https://cloud.google.com/blog/topics/threat-intelligence/dis... >These SDKs, which are offered to developers across multiple mobile and desktop platforms, surreptitiously enroll user devices into the IPIDEA network. If I install an app with a big ToS and buried in there is "we use third-party monetization SDKs" and buried in there is "welcome to our botnet" they do not actually have my consent.
- stymaar 1mo ago
  It's the "legal version" of botnets, yes.
maybewhenthesun 1mo ago
> youtube seems to now more aggressively block yt-dlp. Another thing to thank AI for! /s

ks2048 1mo ago

I was wondering if youtube blocked them a lot and they got a lot of videos from other sources - but no, you can see on this HF page that 93.1% of URls are youtube.com, https://huggingface.co/datasets/laion/BVD-URLs https://huggingface.co/datasets/laion/BVD-URLs