11 ms·
How big is YouTube?
- judge2020 3y agoThe outcome of this article is this linked accompanying website: https://tubestats.org/ https://tubestats.org/
- paulsutter 3y agoIt’s really easy for them to block this method: return a random video for a certain percent of non existing identifiers. Throw in a bit of random for good measure This is the risk of explaining the method
- bomewish 3y agoThough if they didn’t say they were doing that we wouldn’t know the method was invalid. Further, that other video would have its own existing uid, so in theory we could know if they’d duplicated it to thwart these measures.
- hn_throwaway_99 3y agoMy guess is that would be very, very difficult without breaking a ton of other invariant assumptions throughout the system, like: Video IDs are immutable A video can only be represented by a single unique video ID etc.
- bertil 3y agoYou could keep that assumption and just serve a random video if some external IP dials a non-existent ID. Presumably, no one would do that except researchers trying to count videos (or randomly find hidden ones?). You can break the assumption of unicity (if an unassigned ID is later assigned) if you do that internally, although not sure that’d be common but it’s not an assumption that has to be strict for non-attributed ones, and you never use the fake ID.
- tczMUFlmoNk 3y agoThis is easy to detect, though, at least for public videos. Click through to the channel on a different IP and find the video link, or search for the video's title and description, or find a canonical link to the video by any other means. If the IDs don't match, it's been faked.
- bertil 3y agoMaybe use a hash for consistency?
- j2kun 3y agoYou misunderstand that this technique was executed by using YouTube _search_ to find videos, not by querying the exact URL. They can doctor search results however they like.
- samstave 3y agoAre the video IDs sequential in the available domain? or just all over the place? Is there anything in common with all known live video IDs that could make it easier to scan the quintillion possibilities?
- dclowd9901 3y agoI can’t imagine trying to debug production errors with a system like this.
- chii 3y ago> return a random video for a certain percent of non existing identifiers. if you got a video from a randomly generated ID, you can immediately query for it again, and see if the video was the same as before. If it's not the same, you disgard the result and assume the ID generated was actually non-existent. If it's the same, then you know it's a real ID. As long as youtube vidoe URLs are immutable, this method will stop the blocks you described.
- alex_young 3y agoUnless they cache the false information…
- samstave 3y agoThey could use a random redirect function. Throw the user a video URL which is random, but then redirects to the video that you want. So one could never count the space?
- cavisne 3y agoI think this would be easily detectable anyway as whatever IP’s they are using would be viewing videos that basically no one ever watches.
- oefrha 3y agoThe paper says they made 1.8e10 attempts to produce 1e4 videos. TFA says they now have 2.5e4 videos, so they’ve made >4e10 attempts so far. No way in hell you can scrape YouTube like that without buying access to a big proxy IP pool (e.g. Luminati).
- j2kun 3y agoOr don't return any search results for searches for random hashes
- zX41ZdbW 3y agoI recommend checking the dataset of "YouTube dislikes": https://clickhouse.com/docs/en/getting-started/example-datasets/youtube-dislikes https://clickhouse.com/docs/en/getting-started/example-datas... (it is named this way because it was an archival effort to collect the information before the dislike feature was removed) It can be used to find things like the most controversial videos; top videos with a description in a certain language, etc.
- chii 3y agoIt's important to know stats like this dislike counts, because youtube is such a large and public platform that it borderline a public utility. from the article: > It’s possible that YouTube will object to the existence of this resource or the methods we used to create it. Counterpoint: I believe that high level data like this should be published regularly for all large user-generated media platforms. These platforms are some of the most important parts of our digital public sphere, and we need far more information about what’s on them, who creates this content and who it reaches. The gov't ought to make it regulation to force platforms to expose stats like these, so that it can be collected by the statistics bureaus.
- tap-snap-or-nap 3y ago> youtube is such a large and public platform that it borderline a public utility. So has been the big banks, large corporations, land but they all feed off each other and the government. What we want as a community is usually quite different to what they decide to do.
- otteromkram 3y agoI disagree. Nobody is stopping users from selecting one of the many YouTube competitors out there (eg - Twitch, Facebook, Vimeo) to host their content. We could also argue that savvy marketers/influencers use multiple hosting platforms. YouTube's data is critical for YouTube and Google, which is basically an elaborate marketing company. Governments should only enforce oversight on matters such as user rights and privacy, anticompetitive practices, content subject matter, etc.
- japanman185 3y ago[dead]
- paxys 3y agoDoes YouTube generate video URLs completely randomly though? I'd be surprised if that was actually the case.
- timeagain 3y agoRandomly, probably not. But I bet that they go pretty uniformly through their URL hashing space.
- Dylan16807 3y agoAnything random-ish will make this estimation method work. On either youtube's side or the measuring side. Why would they not be random? Nobody has ever found a pattern that I'm aware of, and there are pretty solid claims of past PRNG use. And a leak of the PRNG seed was likely why they mass-privated all unlisted videos a couple years ago.
- jolyon 3y agoWait a second. If the IDs are all allocated in a contiguous block, and the author samples the whole space at random—then the estimate will converge to the correct value. If, on the other hand, the IDs are allocated at random, the estimate will also converge to the correct value. If you believe that there is some structure in YouTube video IDs, that would have no effect on this experiment. It would just reduce the fraction of the total address space that YouTube can use. This is a well-known property of "impure" names, and it means there is a good chance that the IDs have no structure. In other words, the video IDs would be "pure" names.
- treesciencebot 3y agoIs 32,000 a good enough number to estimate the entirety of the Youtube’s video space? It felt to little for what they are trying to accomplish (especially when they started doing year by year analysis)
- Dylan16807 3y agoWhen you're estimating a ratio between two outcomes, the rule of thumb is that you want at least 10-100 samples of each outcome, depending on how much precision you want. They got 10,000 samples of hits, and a huge number of samples of misses. Their result should be very accurate. (32,000 was a different number)
- arrozconcosas 3y agoThe fact that reports on "misinformation" don't look at the denominator when considering the volume of impressions is a great example of selectively reporting statistics to support a preconceived notion.
- sneak 3y agoThe videos they find also show view counts. This allows them to estimate (very roughly!) the view counts across all videos, because it allows them to see the distribution.
- im3w1l 3y agoThe issue with that is if there is one or a handful of videos that have a significant portion of all youtube views. Most likely they will not be in your sample which could lead to a big underestimate.
- junofan 3y agoNot to mention bot farms. The most-viewed source of “misinformation” in the linked 2020 report simply… doesn’t exist anymore. Have we just been hearing about crappy sites buying views these past few years?
- senderista 3y agoIt should be obvious, but "misinformation" is an arbitrary political designation and therefore a constantly moving target. "You should wear an N95 mask to avoid getting COVID" was misinformation in March 2020 but not 3 months later. "Vaccines may not prevent COVID transmission" was misinformation from 2020 until sometime in 2022. "COVID infection may confer significant immunity from future infection" was misinformation for about the same period. The "lab leak hypothesis" was a "racist conspiracy theory" from 2020 until 2023, when it was officially endorsed by YouTube's political sponsors. And so on.
- ta8645 3y ago[flagged]
- pvankessel 3y agoThis is such a clever way of sampling, kudos to the authors. Back when I was at Pew we tried to map YouTube using random walks through the API's "related videos" endpoint and it seemed like we hit a saturation point after a year, but the magnitude described here suggests there's a quite a long tail that flies under the radar. Google started locking down the API almost immediately after we published our study, I'm glad to see folks still pursuing research with good old-fashioned scraping. Our analysis was at the channel level and focused only on popular ones but it's interesting how some of the figures on TubeStats are pretty close to what we found (e.g. language distribution): https://www.pewresearch.org/internet/2019/07/25/a-week-in-the-life-of-popular-youtube-channels/ https://www.pewresearch.org/internet/2019/07/25/a-week-in-th...
- 0x1ceb00da 3y agoThis technique isn't new. Biologists use it to count the number of fish in a lake. (Catch 100 fish, tag them, wait a week, catch 100 fish again, count the number of tagged fishes in this batch)
- dclowd9901 3y agoI made the same connection but it’s still the first time I’ve seen it used for reverse looking up IDs.
- fergbrain 3y agoIsn’t this just a variation of the Monte Carlo method?
- deleted 3y ago[deleted]
- zellyn 3y agoDo you get the same 100 dumb fish?
- egeozcan 3y ago
- RajT88 3y agoI wonder how the wingnut in the comments arrived (and whether he read the article at all - wingnuts usually don't): > NOT big enough; that it can’t be wiped away from the surface earth; with a relatively cyber attack from the Alliance. In fact; Google/YouTube will just be one; of 63 communist controlled platforms; that will be completely erased from existence once the order is given. I have to wonder what this "Alliance" is supposed to be.
- nwatson 3y agoMy guess, "The Alliance" would be a QAnon trope, a confederation of select "White Hats" in the US military and parts of US government (and other select humans) and some alien "Arcturian" UFO contingent. They are trying to overthrow the Deep State (corrupt civil/intelligence/military service in USA and Western governments) that are colluding with "globalists" and exploitative capitalists to economically and socially oppress "good white people of European descent" and corrupt their morals and dilute their influence and moral strength. Presumably The Alliance chose Trump to become President in 2016 as part of an attempt to awaken the masses and usher in "The Great Reset / The Storm / The Awakening" where global corruption will be exposed, corrupt politicians, businesspeople, officials, cultural influencers will quickly be judged and executed. There are many variations on this theme, and though not widely held as a belief, elements seep into the wider discourse and thought. I'm not sure what the aliens' motivation would be in this scheme, and often they're not part of the story. EDIT: here's a random page that likely will bring more questions than answers but it references the elements described above ... https://divinecosmos.com/davids-blog/22143-moment-of-truth-q-anon-discloses-secret-space-program/2/ https://divinecosmos.com/davids-blog/22143-moment-of-truth-q... ... there are variations of this conspiracy theory tailored to many different tendencies: New Agers, Evangelical Christians, White Supremacists, Conservative Republicans, etc. For a further look, find interesting interviews with a certain Jan Halper Hayes and her supposed work with USA Space Force. Some of these promoters a true believers, others are grifters. Also ... grand corporatists and communists are usually conflated in these narratives.
- PaulDavisThe1st 3y agoIt's sort of a wonder that this screenplay has not yet been a major hollywood production. I mean, sure, anything QAnon is terribly uncool for us informed types /s but its hard to believe that this would not be a major success if done with high production values and non-cheesy script. Something vaguely related to next year's A24 release "Civil War", but much more QAnon-adjacent and informed. Even better if Q would denounce it all as an attempt to "innoculate people from the truth".
- goles 3y agoYoutube is in a fairly sad state from it's former self. Search has become, quite frankly, unusable. I recall searching for a <band> recently at the gym, and instead of just showing me <band>. Results showed 3-4 results then completely unrelated topics to <band>. Results are entirely off-topic, but related to other interests I have; tech, planes, trains, and automobiles, etc.
- jchw 3y agoJust FYI, in case you didn't notice, you can bypass the stupid thing that makes search useless by using a filter, like filtering by "Type: Video" or something. I'm sure that workaround will stop working some day randomly.
- goles 3y agoOh that's interesting! Thanks for that I'll try it. Much in the same way Google has "Site:X" "filetype:X".
- throwaway041207 3y agoThe "For You" section of my youtube search is awful, like terribly awful. It's decided for some reason I enjoy popping and weird skin infection videos. I think this is because I clicked through to an external link to a horse getting a hoof infection shaved once (did not enjoy and did not watch all of it). That's literally the only time I've watched anything like that and since then it's always at the bottom of my search. These videos are never recommended on my front page, though. There are some weird corners of YouTube.
- hyperhopper 3y agoThe one comment on this post is hilariously unhinged. For posterity: > NOT big enough; that it can’t be wiped away from the surface earth; with a relatively cyber attack from the Alliance. In fact; Google/YouTube will just be one; of 63 communist controlled platforms; that will be completely erased from existence once the order is given.
- steve_adams_86 3y agoI don’t know why but; many conspiracy theorists I’ve encountered; use semicolons; wrong
- im3w1l 3y agoSemi colons are a little understood and rarely used punctuation symbol, the mark of a true intellectual; it's only natural then, that those of us that know how the world really works use them more than others - and then there are those that think they know how they world works, but are mistaken: they similarly misuse the semicolons.
- jedberg 3y agoI actually was more interested in that comment than the article (not that the article wasn't interesting, but that comment was just wow). Who is the Alliance?? What are the 63 communist controlled platforms?? Who is giving the order to wipe them out? I must know these things now.
- noSyncCloud 3y ago
- ekianjo 3y ago[flagged]
- Sai_ 3y agoYou may have missed the first for the trees. This article is about figuring out 3.8 B views of misinformation videos out of how many total views. The question being pondered is that if the ratio is 0.5%, is disinformation really a problem?
- DharmaPolice 3y agoLooking at things in terms of ratios seems like an odd way of judging whether misinformation is a problem. If 0.5% of the sentences I say in a day are violent threats towards children that'd be a problem, right?
- Sai_ 3y ago“Won’t you think of the children?” type questions are not very useful, imho. In your case, no one (barring a few people employing child soldiers) think children are fair game to be hurt so the answer to your question is that yeah, 0.5% of statements which are violent threats to children is a problem. Covid misinformation isn’t as clear cut - reasonable people asked about the lab leak theory while yet others asked why a vax was being rolled out without holding manufacturers liable for adverse outcomes. We’ve had situations in the past when a new medicine caused children to be born with shortened arms and other birth defects. In this light, it is reasonable to wonder why we should trust a pharma company when they didn’t trust themselves. But the linked post is not even about the correctness of misinformation - given something has been classified as misinformation, how often is it viewed compared to non-misinformation videos is what they’re trying to answer.
- halfjoking 3y ago[flagged]
- verall 3y ago
- yyyfb 3y agoNon-view-weighted and non-impression-weighted stats are interesting but basically useless for disinformation research (which is what the post starts about). To a first approximation every viewpoint and its opposite is out there somewhere on YouTube, along with zero-value videos like security cam backups. The real questions of societal interest involve reach and volume.
- jeffbee 3y agoThe only thing I am suspicious of here is their estimated median view count being so high. I would expect the median view count to be zero.
- nick222226 3y agoCome on now, this isn't Twitter
- wavemode 3y agoIt would be interesting to see if some other method of estimating YouTube's size could be used to corroborate the accuracy of this estimate.
- alex_young 3y agoThis is an interesting way to attack mitigations to the German Tank Problem [0]. I expect the optimal solution is to increase the address space to prevent random samples from collecting enough data to arrive at a statistically significant conclusion. There are probably other good solutions which attempt to vary the distribution in different ways, but a truly random sample should limit that angle. [0] https://en.m.wikipedia.org/wiki/German_tank_problem https://en.m.wikipedia.org/wiki/German_tank_problem
- consp 3y agoI didn't read it in the article but this hinges on it being a discrete uniform distribution. Who knows what kind of shenanigans Google did to the identifiers.
- Maxion 3y agoWould be quite the challenge to use a skewed distribution of the address space that's skewed enough to mitigate this type of scraping while at the same time minimizing the risk of collisions.
- cbolton 3y agoActually the method works regardless of the distribution. It's an interesting and important feature of random sampling. Consider 1000 IDs assigned in the worst (most skewed) way: as a sequence from 0 to 999. If there are 20 videos they will have IDs 0 to 19. If you draw 500 independent random numbers between 0 and 999, each number will have 2% probability of being in the range 0 to 19. So on average you will find that 2% of your 500 attempts find an existing video. From that you conclude that 2% of the whole space of 1000 IDs are assigned. You find correctly that there are 0.02*1000 = 20 videos.
- pbhjpbhj 3y agoYou've assumed the conclusion, you don't know it's 2%; you don't know there are 1000 possible addresses.
- herval 3y agodoes that sampling function assume every "area code" contain the same number of usable numbers? In the case of some big sites out there (twitter, etc), it's common to have certain shards be much less dense when they hold data that's more requested - eg there'd be fewer numbers on the area code Justin Bieber is in, etc. That might skew things considerably..
- jaggederest 3y agoYeah that was my immediate response - what makes them think that videos are actually distributed randomly across the space? I would assume that the numbers are more like a UUID which has random sections but also meaningful sections as well, so it is quite likely that certain subsets have skewed or more/less density.
- deleted 3y ago[deleted]
- johnma2006 3y agoTheir estimator does not require any assumption of uniformity. Notice that P(random ID is valid) = total_valid_IDs / total_IDs is the same regardless of the distribution of valid IDs.
- gmoot 3y agoI don't think this is correct. In order to generate a random number at all you first have to assume a distribution. If they generate a series of random ids, and those random ids are uniformly distributed across the space of all possible ids - while the valid ids have clusters - then won't their method still give a skewed estimate?
- barrkel 3y agoNo. Think about it: how could the underlying distribution change the probability of a hit from a uniform random sample?
- parisisles 3y ago[flagged]
- parisisles 3y ago[dead]
- rglullis 3y agoDamn, I didn't know about this one. Maybe it can help with my work on https://fediverser.network https://fediverser.network
- drewtato 3y agoI was expecting to find out how much data YouTube has, but that number wasn't present. I've used the stats to roughly calculate that the average video is 500 seconds long. Then using a bitrate of 400 KB/s and 13 billion videos, that gives us 2.7 exabytes. I got 400KB/s from some FHD 24-30 fps videos I downloaded, but this is very approximate. YouTube will encode sections containing less perceptible information with less bitrate, and of course, videos come in all kinds of different resolutions and frame rates, with the distribution changing over the history of the site. If we assume every video is 4K with a bitrate of 1.5MB/s, that's 10 exabytes. This estimate is low for the amount of storage YouTube needs, since it would store popular videos in multiple datacenters, in both VP9 and AV1. It's possible YouTube compresses unpopular videos or transcodes them on-demand from some other format, which would make this estimate high, but I doubt it.
- staplers 3y agoI'm interested in the carbon footprint.
- mattlondon 3y agohttps://sustainability.google/operating-sustainably/net-zero-carbon/ https://sustainability.google/operating-sustainably/net-zero...
- staplers 3y agoThanks, 10 million metric tons per year. ~0.1% of global emissions. Pretty wild stuff.
- laluser 3y agoYou’re forgetting about replication and erasure coding overhead. 10 exabytes seems very low tbh. I’d put it closer to 50-100EB at this point.
- ksec 3y agoThat storage number is highly likely to be off by an order of magnitude. 400KB/s, or 3.2Mbps as we would commonly use in video encoding, is quite low for original quality upload in FHD or commonly known as 1080p. The 4K video number is just about right for average original upload. You then have to take into account YouTube at least compress those into 2 video codec, H.264 and VP9. Each codec to have all the resolution from 320P to 1080P or higher depending on the original upload quality. With many popular additional and 4K video also encoded in AV1 as well. Some even comes in HEVC for 360 surround video. Yes you read that right. H.265 HEVC on YouTube. And all of that doesn't even include replication or redundancy. I would not be surprised if the total easily exceed 100EB. Which is 100 (2020 ) Dropbox in size.
- chrismsimpson 3y agoA mountains? A sky?
- stochtastic 3y agoThis is a fun dataset. The paper leaves a slight misimpression about channel statistics: IIUC, they do not correct for sampling propensity to reweight when looking at subscriber counts (it should be weighted ~1/# of videos per channel since the probability of a given channel appearing is proportional to the number of public videos that channel has, as long as the sample is a small fraction of the population).
- bevan 3y agoI noticed that too. Seems very unlikely that 1,000,000 subscribers represents the 98th percentile and not the 99.999th.
- xwowsersx 3y agoPretty big
- bane 3y agoGoogle used to ask scaling questions about youtube for some positions. They often ended up in some big-O line of questions about syncing log-data across a growing an distributed infrastructure. The result was some ridiculous big-O(f(n)) where the function was almost impossible to even describe verbally. Fun fun. source interviewed by Google a few times
- Zenul_Abidin 3y agoToo big. Last night they issued a notification to my phone requiring me to update the YouTube app. The problem - it is the last version that runs on my phone. At least web still works, for now.
- maxhille 3y agoJust curious - what device and OS version are you on?
- rvba 3y ago> Kevin Zheng wrote a whole bunch of scripts to do the dialing, and over the course of several months, we collected more than 10,000 truly random Several is 3 to 5? So 4 months. 10 000 videos over 4 months. So 2500 per month. If we assume 20 work days.. that's 125 per day. Over 8 hours.. 15,6 per hour. Those scripts seem to be quite slow? Was this done by hand? A video every 3 minutes.
- defrost 3y agoThe context you omitted is: Turns out there are 2^64 possible YouTube addresses, an enormous number: 18.4 quintillion. There are lots of YouTube videos, but not that many. Let’s guess for a moment that there are 1 billion YouTube videos – if you picked URLs at random, you’d only get a valid address roughly once every 18.4 billion tries. I mean, sure, they did reduce that a little: Jason found a couple of cheats that makes the method roughly 32,000 times as efficient, meaning our “phone call” connects lots more often. With that in mind, how many attempts did they make to get a hit every three minutes? It's surprising that they weren't throttled back for making excessive requests.
- 0xDEADFED5 3y agothe cheat was using a botnet of 32k devices, lol
- spacebacon 3y agoSo if youtube is worth 180 billion, with 14 billion videos. Each video is worth $13. The average length of a youtube video is around 13 minutes. My napkin GPT math values video per minute at $35 per minute. My napkin GPT math values each video closer to $455. This leaves me with one unanswered question… Where’s the beef?
- hippich 3y agoThese calculations should be viewed as long term expected value. Long-term is probably 10-20 years in this case. So, based on your calculations, each video is about 2-4 bucks a year.
- TMWNN 3y ago>My napkin GPT math values each video closer to $455. I've read that the number of views / 100 in dollars is a good estimate of how much the channel receives.
- jart 3y ago> https://redditmap.social/ https://redditmap.social/ This is the best thing I've ever seen.
- 1vuio0pswjnm7 3y ago"We're starting some experiments to understand how the videos YouTube recommends differ from the "average" YouTube video - YouTube likes recommending videos with at least ten thousand views, while the median YouTube video has 39 views." In experiments by yours truly, it seems that quality goes down as views go up.
- deleted 3y ago[deleted]
- 38 3y agoI think the plot is more dome shaped. Big channels are crap, but many smaller ones are too
- anon_cow1111 3y agoGetting a 403 forbidden link from source right now, did we trip the poor guy's ddos protection or is it just me?
- Aachen 3y agoSame here. A computer decides I'm not going to see this and that's that, the info on page is 403 forbidden and the word nginx. No recourse or way to let anyone know the thing is broken. If my IP address or language setting were Russian or something I'd at least be able to see a reason for it, but no, Germany and English. On an Android! I know, very dangerous. Can anyone post what this site is? So I can tell whether this #1 story is worth it trying a different ISP or user agent string or whatnot
- nkurz 3y agoIt's the blog of a researcher at the University of Massachusetts-Amherst. He does research on the impact of social media on society. It's a Wordpress site, hosted by someone commercial. I know him, and I'm almost certain no blocking would be intentional. If you (or others) give me details about how your access is failing, I'll pass the info on to him so he can try to get it fixed. In the meantime, this archive page should work for you: https://web.archive.org/web/20231223131348/https://ethanzuckerman.com/2023/12/22/how-big-is-youtube/ https://web.archive.org/web/20231223131348/https://ethanzuck...
- Aachen 3y ago> you (or others) give me details about how your access is failing Well, I click the link and get the access denied page. Do you need an IP address (93.135.167.200), user agent string¹, timestamp, or what would help? Randomly noticed my accept-language header is super convoluted, mixing English-in-Germany, English, German in Germany, Dutch in the Netherlands, Dutch, and finally USA English, with decreasing priority from undefined to 0.9 to 0.4 in decrements of 0.1. I guess it takes this from my OS settings? Though I haven't configured it explicitly like that, particularly the en-US I'd not use because of the inconsistent date format and unfamiliar units system. Maybe the server finds it weird that six different locales are supported? Thanks for responding and relaying! ¹ Mozilla/5.0 (Linux; Android 10; Pixel Build/QP1A.190711.019; wv) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Mobile Safari/537.36
- SnowflakeOnIce 3y agoSurely someone at YouTube could answer this question definitively?
- bzbz 3y agoFor anyone who's wondering, their estimation method works like so: 1. Assume a range of values 2. Assume a fair probability function for sampling over the range of values The estimated size is the %-of-hits * the total range of values.
- Keyframe 3y agoI skimmed through the article but that's a lot of assumptions there if so. 1. So let's say that possible range of values is true (10 characters of specific range + 1). That would represent one big circle of possible area where videos might be. 2. Distribution of identifiers (valid videos) is everything. If Youtube did some contraints (or skewing) to IDs, that we don't know about, then actual existing video IDs might be a small(er) circle within that bigger circle of possibilities and not equally dispersed throughout, or there mught be clumping or whatever... So you'd need to sample the space by throwing darts in a way to get a silhouette of their skew or to see if it's random-ish, by I don't know let's say Poisson distribution. Only then one could estimate the size. So is this what they're doing? Also.. anyone bothered to you know, ask Youtube?
- remus 3y agoVideo IDs are generated by hashing a a secret identifier, so they should be uniformly distributed.
- cbolton 3y agoNo the distribution doesn't matter at all. I've given an extreme example here: https://news.ycombinator.com/item?id=38742735 https://news.ycombinator.com/item?id=38742735
- Keyframe 3y agoI see what you did there. So basically an overlapped proportion (or hits proportion) would be overlapping hits divided by samples run, and then an estimated total would be this proportion divided by total space of possibilities. That would work.
- Cypher 3y agovery big
- djtango 3y agowould be interesting to know how this was before/after Shorts and how it's changed (and how notShorts have performed since Shorts launched)
- verginer 3y agoThe author notes that they used "cheats". Depending on what these do the iid assumption of the samples being independent could be violated. If it is akin to snowball sampling it could have an "excessive" success rate thereby inflating the numbers. > Jason found a couple of cheats that makes the method roughly 32,000 times as efficient, meaning our “phone call” connects lots more often
- stravant 3y agoThe data probably wouldn't look so clean if it were skewed. If Google were doing something interesting it probably wouldn't be skewed only by a little bit.
- verginer 3y agoAdmittedly, I did not read the paper linked. But my point is not about google doing something funny. Even if we assume that ids are truly random and uniformly distributed this does not mean that the sampling method doesn't have to be iid. This problem is similar to density estimation where Rejection sampling is super inefficient but converges to the correct solution, but MCMC type approaches might need to run multiple times to be sure to have found the solution.
- mocamoca 3y agoI agree. Proving that using cheats and auto complete does not break sample independence and keeps sampling as random as possible would be needed here for stats beginners such as me! Drunk dialing but having a human operator that each time tries to help you connect with someone, even if you mistyped the number... Doesn't look random to me. However I did not read the 85 pages paper... Maybe it's addressed there.
- Mogzol 3y agoPage 9 & 10 of the paper [1] go into some detail: > By constructing a search query that joins together 32 randomly generated identifiers using the OR operator, the efficiency of each search increases by a factor of 32. To further increase search efficiency, randomly generated identifiers can take advantage of case insensitivity in YouTube’s search engine. A search for either "DQW4W9WGXCQ” or “dqw4w9wgxcq” will return an extant video with the ID “dQw4w9WgXcQ”. In effect, YouTube will search for every upper- and lowercase permutation of the search query, returning all matches. Each alphabetical character in positions 1 to 10 increases search efficiency by a factor of 2. Video identifiers with only alphabetical characters in positions 1 to 10 (valid characters for position 11 do not benefit from case-insensitivity) will maximize search efficiency, increasing search efficiency by a factor of 1024. By constructing search queries with 32 randomly generated alphabetical identifiers, each search can effectively search 32,768 valid video identifiers. They also mention some caveats to this method, namely, that it only includes publicly listed videos: > As our method uses YouTube search, our random set only includes public videos. While an alternative brute force method, involving entering video IDs directly without the case sensitivity shortcut that requires the search engine, would include unlisted videos, too, it still would not include private videos. If our method did include unlisted videos, we would have omitted them for ethical reasons anyway to respect users’ privacy through obscurity (Selinger & Hartzog, 2018). In addition to this limitation, there are considerations inherent in our use of the case insensitivity shortcut, which trusts the YouTube search engine to provide all matching results, and which oversamples IDs with letters, rather than numbers or symbols, in their first ten characters. We do not believe these factors meaningfully affect the quality of our data, and as noted above a more direct “brute force” method - even for the purpose of generating a purely random sample to compare to our sample - would not be computationally realistic. [1]: https://journalqd.org/article/view/4066 https://journalqd.org/article/view/4066
- padmasampath 3y ago[flagged]
- padmasampath 3y ago[flagged]
- snthpy 3y ago> start with Never Going to Give You Up, objectively the center of the internet, LOL
- hknmtt 3y agoAs a user, for me, YouTube has been shrinking ever since the first adpocalypse. By that I mean I rarely see new videos or relevant recommendations. It is all stale content and pages only couple of pages instead of going infinite like it used to. My most visited page is home page which shows recommended content(it used to be on a dedicated page) and I literally see the same content for days, rarely with new videos. And I have hundreds of subscribed channels. Youtube is a living corpse, I do not care what numbers they put out in the public.
- iamflimflam1 3y agoGo to the subscribed page and watch the channels you subscribed to. The home page is whatever YouTube is pushing you to watch - it’s not necessarily what you want to watch…
- internet101010 3y agoYeah some people never do that for some reason. I have a simple little tampermonkey script set up to automatically just redirect me to my subscriptions when I go to the home page on accident.
- falsaberN1 3y agoFunny, for me it's the opposite, I hardly ever set foot outside of my subcriptions page except for searching, which is how I find new channels to add to subs.
- realusername 3y agoThat's what I was about to comment, 13 billion videos and the same 10 videos recommended in a loop, what a product failure.
- Xiol32 3y agoYouTube also seems to forget what videos I've watched, so I end up having videos recommended to me that I've already seen.
- SeriousM 3y agoDoes anyone has experience with the autocomplete feature that was mentioned in the article? > generate a random 5 character long string where one of them is a dash, and Youtube will autocomplete the string if a matching video exists A random youtube video service would be interesting! Like Wikipedia random, but less educational XD
- starshadowx2 3y agoThere are multiple sites that do already do that, this is one I like - https://perchance.org/youtube-video https://perchance.org/youtube-video
- teekert 3y agoSkim the article, find graph with key piece of info that answers title, y axis label: Estimated size (billions). …
- Dowwie 3y agoGangnam Style has 4.9 Billion views now. I remember how big a deal it was when it broke 1 Billion!
- dvh1990 3y agoVery cool, kudos! It would be even cooler if it had a deeper view of categories. Right now the biggest category by far is People & Blogs but it's possible to get much more information if it was broken down into sub-categories.
- pvankessel 3y agoHere's a study I did a few years ago that broke down popular YT videos into more granular categories, might be of interest: https://www.pewresearch.org/internet/2019/07/25/childrens-content-content-featuring-children-and-video-games-were-among-the-most-viewed-videos-genres/ https://www.pewresearch.org/internet/2019/07/25/childrens-co...
- dadjoker 3y agoHeh, when he starts quoting reports on COVID “misinformation“, you know where this report is going…
- braunjohnson 3y ago[dead]
- golemotron 3y agoHall monitors are using math now. Oh no!
- countWSS 3y agofinding random videos is fairly easy these researchers are doing it the dumb brute-force way: 1.Block any famous accounts(1M+views): these poison your recommendations. 2.Like only old and lowest-view videos. 3.Only click on lowest-viewcount videos that are not marked as recent. in a few days your home feed and recommendations will consist of random low-ranked junk. Then, start blocking the highest views channels and repeat the 1.2.3. steps to get even more obscure content. No need to scrape millions of IDs or flood the servers with random requests.
- deleted 3y ago[deleted]
- mik09 3y agoalso its hard to replicate. like for reddit there's a lot of alternatives cause it's mainly texts but youtube videos require datacenters which are very expensive. perhaps some kind of distributed video storage? but seems hard.
- dostick 3y agoThe