5 ms·

It’s really easy for them to block this method: return a random video for a certain percent of non existing identifiers. Throw in a bit of random for good measu

It’s really easy for them to block this method: return a random video for a certain percent of non existing identifiers. Throw in a bit of random for good measure This is the risk of explaining the method

bomewish 3y ago
Though if they didn’t say they were doing that we wouldn’t know the method was invalid. Further, that other video would have its own existing uid, so in theory we could know if they’d duplicated it to thwart these measures.
hn_throwaway_99 3y ago
My guess is that would be very, very difficult without breaking a ton of other invariant assumptions throughout the system, like: Video IDs are immutable A video can only be represented by a single unique video ID etc.
- bertil 3y ago
  You could keep that assumption and just serve a random video if some external IP dials a non-existent ID. Presumably, no one would do that except researchers trying to count videos (or randomly find hidden ones?). You can break the assumption of unicity (if an unassigned ID is later assigned) if you do that internally, although not sure that’d be common but it’s not an assumption that has to be strict for non-attributed ones, and you never use the fake ID.
  - tczMUFlmoNk 3y ago
    This is easy to detect, though, at least for public videos. Click through to the channel on a different IP and find the video link, or search for the video's title and description, or find a canonical link to the video by any other means. If the IDs don't match, it's been faked.
    - bertil 3y ago
      Maybe use a hash for consistency?
- j2kun 3y ago
  You misunderstand that this technique was executed by using YouTube _search_ to find videos, not by querying the exact URL. They can doctor search results however they like.
samstave 3y ago
Are the video IDs sequential in the available domain? or just all over the place? Is there anything in common with all known live video IDs that could make it easier to scan the quintillion possibilities?
dclowd9901 3y ago
I can’t imagine trying to debug production errors with a system like this.
chii 3y ago
> return a random video for a certain percent of non existing identifiers. if you got a video from a randomly generated ID, you can immediately query for it again, and see if the video was the same as before. If it's not the same, you disgard the result and assume the ID generated was actually non-existent. If it's the same, then you know it's a real ID. As long as youtube vidoe URLs are immutable, this method will stop the blocks you described.
- alex_young 3y ago
  Unless they cache the false information…
  - chii 3y ago
    but now you have to store state for some arbitrary url, share it across many different clusters/regions, record which IP has seen it etc... a lot of work, for very little gain.
  - danjc 3y ago
    You could use a hashing function that guarantees the same result every time.
samstave 3y ago
They could use a random redirect function. Throw the user a video URL which is random, but then redirects to the video that you want. So one could never count the space?
cavisne 3y ago
I think this would be easily detectable anyway as whatever IP’s they are using would be viewing videos that basically no one ever watches.
- oefrha 3y ago
  The paper says they made 1.8e10 attempts to produce 1e4 videos. TFA says they now have 2.5e4 videos, so they’ve made >4e10 attempts so far. No way in hell you can scrape YouTube like that without buying access to a big proxy IP pool (e.g. Luminati).
j2kun 3y ago
Or don't return any search results for searches for random hashes

bomewish 3y ago

Though if they didn’t say they were doing that we wouldn’t know the method was invalid. Further, that other video would have its own existing uid, so in theory we could know if they’d duplicated it to thwart these measures.

hn_throwaway_99 3y ago

My guess is that would be very, very difficult without breaking a ton of other invariant assumptions throughout the system, like: Video IDs are immutable A video can only be represented by a single unique video ID etc.

bertil 3y ago
You could keep that assumption and just serve a random video if some external IP dials a non-existent ID. Presumably, no one would do that except researchers trying to count videos (or randomly find hidden ones?). You can break the assumption of unicity (if an unassigned ID is later assigned) if you do that internally, although not sure that’d be common but it’s not an assumption that has to be strict for non-attributed ones, and you never use the fake ID.
- tczMUFlmoNk 3y ago
  This is easy to detect, though, at least for public videos. Click through to the channel on a different IP and find the video link, or search for the video's title and description, or find a canonical link to the video by any other means. If the IDs don't match, it's been faked.
  - bertil 3y ago
    Maybe use a hash for consistency?
j2kun 3y ago
You misunderstand that this technique was executed by using YouTube _search_ to find videos, not by querying the exact URL. They can doctor search results however they like.

samstave 3y ago

Are the video IDs sequential in the available domain? or just all over the place? Is there anything in common with all known live video IDs that could make it easier to scan the quintillion possibilities?

dclowd9901 3y ago

I can’t imagine trying to debug production errors with a system like this.

chii 3y ago

> return a random video for a certain percent of non existing identifiers. if you got a video from a randomly generated ID, you can immediately query for it again, and see if the video was the same as before. If it's not the same, you disgard the result and assume the ID generated was actually non-existent. If it's the same, then you know it's a real ID. As long as youtube vidoe URLs are immutable, this method will stop the blocks you described.

alex_young 3y ago
Unless they cache the false information…
- chii 3y ago
  but now you have to store state for some arbitrary url, share it across many different clusters/regions, record which IP has seen it etc... a lot of work, for very little gain.
- danjc 3y ago
  You could use a hashing function that guarantees the same result every time.

samstave 3y ago

They could use a random redirect function. Throw the user a video URL which is random, but then redirects to the video that you want. So one could never count the space?

cavisne 3y ago

I think this would be easily detectable anyway as whatever IP’s they are using would be viewing videos that basically no one ever watches.

oefrha 3y ago
The paper says they made 1.8e10 attempts to produce 1e4 videos. TFA says they now have 2.5e4 videos, so they’ve made >4e10 attempts so far. No way in hell you can scrape YouTube like that without buying access to a big proxy IP pool (e.g. Luminati).

j2kun 3y ago

Or don't return any search results for searches for random hashes