6 ms·

> Google probably only provides stats about growth (like "500 hours of video are uploaded to YouTube every minute") because the total number of videos is so lar

by spuz 5y ago

> Google probably only provides stats about growth (like "500 hours of video are uploaded to YouTube every minute") because the total number of videos is so large, it's an unknowable amount. I suppose you could sample random YouTube urls to find out how many of them link to public videos. Given the total number of possible URLs, it would give you an idea of what percentage of them have been used and therefore how many videos YouTube has in total. It would not tell you how many private videos or Google Drive / Photos videos exist of course.

warent 5y ago
It doesn't seem like this would work. I think you could sample trillions of YouTube IDs with a high likelihood of all of them being unused. They're supposed to be unique after all
- chrisseaton 5y ago
  What do you mean 'supposed to be unique'? How can an ID not be unique?
  - zakki 5y ago
    Maybe when they reach 62^11 + 1?
- rococode 5y ago
  Clicked into a couple random videos, looks like all of their video IDs are 11 characters, alphanumeric with cases. So 26+26+10 = 62 choices for each char, 62^11 = 5.2e+19 = 52 quintillion unique IDs (52 million trillions). So, yeah, sampling would be a mostly futile effort since you're looking to estimate about 8 to 10 decimal digits of precision. Though it's technically still possible since you'd expect about 1 in every 50 million - 5 billion IDs to work (assuming somewhere between a trillion and 10 billion videos). My statistics knowledge is rusty, but I guess if you could sample, say, 50 billion urls you could actually make a very coarse estimate with a reasonable confidence level. That's a lot but, ignoring rate limits, well within the range of usual web-scale stuff.
  - spuz 5y ago
    Thanks for doing the maths - it does seem the sampling method would not be feasible. Taking the statistic of "500 hours uploaded per minute" and assuming the average video length is 10 minutes, we can say about 1.5bn videos are uploaded to YouTube every year or 15bn every 10 years. So it seems likely that YouTube has less than 1tn videos in total.
  - dpatterbee 5y ago
    They also use "_" and "-" according to Tom Scott. https://www.youtube.com/watch?v=gocwRvLhDf8 https://www.youtube.com/watch?v=gocwRvLhDf8
    - _0ffh 5y ago
      Which would bring it up to a nice 64 choices, making it exactly 6 bits per character.
      - slver 5y ago
        It's a URL-friendly form of base64. 11 chars encode 66 bits, but actually 2 bits are likely not used and it's simply an int64 encoded to base64. Given everyone and their grandma is pushing 128-bit UUID for distributed entity PK, it's interesting to see YouTube keep it short and sweet. Int64 is my go to PK as well, if I have to, I make it hierarchical to distribute it, but I don't do UUID.
        littlestymaar 5y ago
        > Given everyone and their grandma is pushing 128-bit UUID for distributed entity PK, it's interesting to see YouTube keep it short and sweet. The trade-off you make when using short IDs is that you can't generate them at random. With 128-bit Id, you can't realistically have collisions, but with 64-bit ones, because of the birthday paradox, as soon as you have more than 2^32 elements, you're really likely to have collisions.
        5y ago
- espadrine 5y ago
  Let’s do the math. IDs are 64-bit integers. The number of tries before an event with probability P occurs is a geometric distribution. If V is the number of valid IDs (that have a video), the number of tries is 2^64÷V. Assuming 1 megatries per second, since we can parallelize it, we would find the first video in 20 seconds on average, with a conservative estimate of V = 10^12 (a hundred billion videos). To have a sample of ~100 videos, it’d take about half an hour.
  - warent 5y ago
    Formal math & probability is something I'm trained in. that being said, intuitively this sounds off to me... The youtube ID pool is closer to 7.4e19, not 1.8e19. I'm not a math expert and my probability is quite weak. If you assume generously that 1 trillion IDs have been taken, the percent of IDs that are in use is 1e12/7.4e19, then 0.0000000135% of available IDs have been taken. Getting a single video at 1 megatry per second would take something more like 2 hours, not 20 seconds.
    - warent 5y ago
      typo: I meant to say it's not something I'm trained in
NavinF 5y ago
Yeah that should work. If I do 1 request per second with 64 IP addresses, I'd expect to find ~110 videos after 1 year of random sampling if there are 1T videos on YouTube. 110=(64*3.154e+7)*(1,000,000,000,000/2^64) (The other thread that assumes a 62-character set is wrong because they forgot about '-' and '_'. I'm fairly certain a video ID is a urlsafe base64 encoded 64-bit int. 64^11==2^66)
jasoncartwright 5y ago
Surprised they don't also mention Nest, which I assume also has an interesting & significant video encoding operation.
slver 5y ago
> Google probably only provides stats about growth (like "500 hours of video are uploaded to YouTube every minute") because the total number of videos is so large, it's an unknowable amount. I can totally see it being non-trivial to count your videos, which is a funny problem to have. But I doubt it's unknowable. More like they don't care/want us to know that.
- nine_k 5y ago
  Quite likely they have a good approximate number. But knowing the exact number can indeed be hard. It would take stopping the entire uploading and deletion activity. Of course they may have counters of uploads and deletions on every node which handles them, but the notion of 'the same instant' is tricky in distributed systems, so the exact number still remains elusive.
  - iainmerrick 5y ago
    It’s not just tricky and elusive, I think it’s literally unknowable -- not a well-defined question. Like asking about the simultaneity of disconnected events in special relativity.
    - zodiac 5y ago
      You can modify the question to be well-defined and not suffer those measurement problems, eg "the total number of videos uploaded before midnight UTC on 2021-04-29"
      - suprfsat 5y ago
        Interesting, I wonder if a distributed database could be developed to consistently answer queries phrased in this way.
        hderms 5y ago
        Seems like it would either need to be append only or have some kind of snapshot isolation
kyrra 5y ago
Googler, opinions are my own. Youtube isn't the only platform where Google does video transcoding. I don't know them all, but here are a few other places where video plays a part: Meet - I'm guessing for participates that on are different devices (desktop, android, ios) and depending on their bandwidth will get different video feed quality. Though, the real-time nature of this may not work as well? Though, Meet has a live stream[0] feature for when your meeting is over 250 people, which gives you a youtube-like player, so this likely is transcoded. Duo - more video chat. Photos - when you watch a photos video stored at google (or share it with someone), it will likely be transcoded. Video Ads. I'd guess these are all pre-processed for every platform type for effecient delivery. While these are mainly on youtube, they show up on other platforms as well. Play Movies. Nest cameras. This is a 24-stream of data to the cloud that some people pay to have X days of video saved. [0] https://support.google.com/meet/answer/9308630?co=GENIE.Platform%3DDesktop&hl=en https://support.google.com/meet/answer/9308630?co=GENIE.Plat...
- jankeymeulen 5y ago
  Also Googler, you can add Stadia here as well. Needs fast and low-latency transcoding.
  - kyrra 5y ago
    You're right. I thought stadia did something special, but I guess not. So yes, Stadia. Another: Youtube TV EDIT: Stadia: I did some searching around the interwebs and found a year-old interview[0] that hints at something different. > It's basically you have the GPU. We've worked with AMD to build custom GPUs for Stadia. Our hardware--our specialized hardware--goes after the GPU. Think of it as it's a specialized ASIC. [0] https://www.techrepublic.com/article/how-google-stadia-encodes-big-data-for-real-time-use/ https://www.techrepublic.com/article/how-google-stadia-encod...

by spuz 5y ago

warent 5y ago

It doesn't seem like this would work. I think you could sample trillions of YouTube IDs with a high likelihood of all of them being unused. They're supposed to be unique after all

chrisseaton 5y ago
What do you mean 'supposed to be unique'? How can an ID not be unique?
- zakki 5y ago
  Maybe when they reach 62^11 + 1?
rococode 5y ago
Clicked into a couple random videos, looks like all of their video IDs are 11 characters, alphanumeric with cases. So 26+26+10 = 62 choices for each char, 62^11 = 5.2e+19 = 52 quintillion unique IDs (52 million trillions). So, yeah, sampling would be a mostly futile effort since you're looking to estimate about 8 to 10 decimal digits of precision. Though it's technically still possible since you'd expect about 1 in every 50 million - 5 billion IDs to work (assuming somewhere between a trillion and 10 billion videos). My statistics knowledge is rusty, but I guess if you could sample, say, 50 billion urls you could actually make a very coarse estimate with a reasonable confidence level. That's a lot but, ignoring rate limits, well within the range of usual web-scale stuff.
- spuz 5y ago
  Thanks for doing the maths - it does seem the sampling method would not be feasible. Taking the statistic of "500 hours uploaded per minute" and assuming the average video length is 10 minutes, we can say about 1.5bn videos are uploaded to YouTube every year or 15bn every 10 years. So it seems likely that YouTube has less than 1tn videos in total.
- dpatterbee 5y ago
  They also use "_" and "-" according to Tom Scott. https://www.youtube.com/watch?v=gocwRvLhDf8 https://www.youtube.com/watch?v=gocwRvLhDf8
  - _0ffh 5y ago
    Which would bring it up to a nice 64 choices, making it exactly 6 bits per character.
    - slver 5y ago
      It's a URL-friendly form of base64. 11 chars encode 66 bits, but actually 2 bits are likely not used and it's simply an int64 encoded to base64. Given everyone and their grandma is pushing 128-bit UUID for distributed entity PK, it's interesting to see YouTube keep it short and sweet. Int64 is my go to PK as well, if I have to, I make it hierarchical to distribute it, but I don't do UUID.
      - littlestymaar 5y ago
        > Given everyone and their grandma is pushing 128-bit UUID for distributed entity PK, it's interesting to see YouTube keep it short and sweet. The trade-off you make when using short IDs is that you can't generate them at random. With 128-bit Id, you can't realistically have collisions, but with 64-bit ones, because of the birthday paradox, as soon as you have more than 2^32 elements, you're really likely to have collisions.
        5y ago
espadrine 5y ago
Let’s do the math. IDs are 64-bit integers. The number of tries before an event with probability P occurs is a geometric distribution. If V is the number of valid IDs (that have a video), the number of tries is 2^64÷V. Assuming 1 megatries per second, since we can parallelize it, we would find the first video in 20 seconds on average, with a conservative estimate of V = 10^12 (a hundred billion videos). To have a sample of ~100 videos, it’d take about half an hour.
- warent 5y ago
  Formal math & probability is something I'm trained in. that being said, intuitively this sounds off to me... The youtube ID pool is closer to 7.4e19, not 1.8e19. I'm not a math expert and my probability is quite weak. If you assume generously that 1 trillion IDs have been taken, the percent of IDs that are in use is 1e12/7.4e19, then 0.0000000135% of available IDs have been taken. Getting a single video at 1 megatry per second would take something more like 2 hours, not 20 seconds.
  - warent 5y ago
    typo: I meant to say it's not something I'm trained in

NavinF 5y ago

Yeah that should work. If I do 1 request per second with 64 IP addresses, I'd expect to find ~110 videos after 1 year of random sampling if there are 1T videos on YouTube. 110=(64*3.154e+7)*(1,000,000,000,000/2^64) (The other thread that assumes a 62-character set is wrong because they forgot about '-' and '_'. I'm fairly certain a video ID is a urlsafe base64 encoded 64-bit int. 64^11==2^66)

jasoncartwright 5y ago

Surprised they don't also mention Nest, which I assume also has an interesting & significant video encoding operation.

slver 5y ago

> Google probably only provides stats about growth (like "500 hours of video are uploaded to YouTube every minute") because the total number of videos is so large, it's an unknowable amount. I can totally see it being non-trivial to count your videos, which is a funny problem to have. But I doubt it's unknowable. More like they don't care/want us to know that.

nine_k 5y ago
Quite likely they have a good approximate number. But knowing the exact number can indeed be hard. It would take stopping the entire uploading and deletion activity. Of course they may have counters of uploads and deletions on every node which handles them, but the notion of 'the same instant' is tricky in distributed systems, so the exact number still remains elusive.
- iainmerrick 5y ago
  It’s not just tricky and elusive, I think it’s literally unknowable -- not a well-defined question. Like asking about the simultaneity of disconnected events in special relativity.
  - zodiac 5y ago
    You can modify the question to be well-defined and not suffer those measurement problems, eg "the total number of videos uploaded before midnight UTC on 2021-04-29"
    - suprfsat 5y ago
      Interesting, I wonder if a distributed database could be developed to consistently answer queries phrased in this way.
      - hderms 5y ago
        Seems like it would either need to be append only or have some kind of snapshot isolation

kyrra 5y ago

Googler, opinions are my own. Youtube isn't the only platform where Google does video transcoding. I don't know them all, but here are a few other places where video plays a part: Meet - I'm guessing for participates that on are different devices (desktop, android, ios) and depending on their bandwidth will get different video feed quality. Though, the real-time nature of this may not work as well? Though, Meet has a live stream[0] feature for when your meeting is over 250 people, which gives you a youtube-like player, so this likely is transcoded. Duo - more video chat. Photos - when you watch a photos video stored at google (or share it with someone), it will likely be transcoded. Video Ads. I'd guess these are all pre-processed for every platform type for effecient delivery. While these are mainly on youtube, they show up on other platforms as well. Play Movies. Nest cameras. This is a 24-stream of data to the cloud that some people pay to have X days of video saved. [0] https://support.google.com/meet/answer/9308630?co=GENIE.Platform%3DDesktop&hl=en https://support.google.com/meet/answer/9308630?co=GENIE.Plat...

jankeymeulen 5y ago
Also Googler, you can add Stadia here as well. Needs fast and low-latency transcoding.
- kyrra 5y ago
  You're right. I thought stadia did something special, but I guess not. So yes, Stadia. Another: Youtube TV EDIT: Stadia: I did some searching around the interwebs and found a year-old interview[0] that hints at something different. > It's basically you have the GPU. We've worked with AMD to build custom GPUs for Stadia. Our hardware--our specialized hardware--goes after the GPU. Think of it as it's a specialized ASIC. [0] https://www.techrepublic.com/article/how-google-stadia-encodes-big-data-for-real-time-use/ https://www.techrepublic.com/article/how-google-stadia-encod...