6 ms·

>Why don't we just make a platform that crawls the internet for videos and build a recommendation algorithm. Therefore no matter where you host your videos you'

by jasode 6y ago

>Why don't we just make a platform that crawls the internet for videos and build a recommendation algorithm. Therefore no matter where you host your videos you'll be easily discover? Because there are fundamental tradeoffs of _information_ and _meta_information_ that a system has that factors into recommendations. People upload about ~500 hours of videos to Youtube every minute.[1] Yes, that means that centralization to that degree has negatives such as the guitar channel being deleted but it also has positives such as better (not foolproof -- but relatively better) detection of spam and mistitled content. Because Youtube servers actually have the real bytes of actual videos, they can scrub the content with machine learning algorithms augmented with some human oversight. If you only have a platform that crawls for video urls and doesn't actually ingest the petabytes of video like Youtube, the "recommendations-only" service will be susceptible to gaming such as people putting "Emma Watson and Scarlett Johansson nudes" as the titles but the actual video is an ad for somebody trying to sell a used car. Or the first few days of a video url has "Tips to deal with COVID" which is real but the video hoster later switches it out to be a video for that same url to be something else. Also, a centralized service like Youtube can measure actual user behavior such as "watch time" (because Youtube's servers know they are sending bytes to the client) to see if the video is actually engaging viewers and this factors into future recommendations. If web surfers are closing out a 10 minute video after 20 seconds, the videos are appropriately penalized because users are "voting" that the content is bad. I.e. the viewers "voted" without even having to press the "thumbs down" button. A new platform to recommend videos pointing to decentralized video hosting has advantages but also has unavoidable disadvantages that lower the quality of the recommendations. EDIT to add my previous comments about the financial incentives (ads) that help video content creators which many techies overlook: https://news.ycombinator.com/item?id=21506992 https://news.ycombinator.com/item?id=21506992 [1] https://www.google.com/search?q=upload+about+~500+hours+of+videos+to+Youtube+every+minute https://www.google.com/search?q=upload+about+~500+hours+of+v...

mIOAMDA 6y ago
What if instead we had the recommendations from the main sites YouTube, Vimeo, Dailymotion and if you want your personal video hosting site to be recommended you would fill in a form to add you sites to the collection of sites the recommendation system crawl through? This might be helpful to avoid bad quality recommendations?
tylerdinner 6y ago
I don't feel these unavoidable disadvantages are as black and white and you describe. To circumvent the fake title/bait & switch practices, you could allow users to flag videos as misleading, vote them as spam, etc. Put the number of flags next to the video, add the ability to not even see flagged videos. Further, scripts could be used to report video play statistics to the scrape site. Drawback is users would have to add the script to their own site, but adding analytics scripts is common practice.
- jasode 6y ago
  >To circumvent the fake title/bait & switch practices, you could allow users to flag videos as misleading, vote them as spam, etc. And now you've re-created the "Amazon 1-star ratings left by competitors instead of real buyers" and/or the "false DMCA takedown claims of public domain or original music". E.g. an honest person creates a "How to sew a mask at home to protect from COVID" which is the real content but a bunch of coronavirus deniers falsely flag that video as "child pornography". Game theory is hard and everybody has to think through all the future chess moves that adversaries will use.
  - tylerdinner 6y ago
    Damn you're totally right. You'd have to have a subscription fee and a team of mods to fix that.
- antocv 6y ago
  > To circumvent the fake title/bait & switch practices, you could allow users to flag videos as misleading, vote them as spam, etc. Put the number of flags next to the video, add the ability to not even see flagged videos. And who would flag the bad users? This just increases slightly the discomfort of bad guys, but does not prevent it at all. Instead they will flag competitive videos and such. Nobody has solved "decentralized trust" yet, if ever.
doh 6y ago
You bring some interesting points, but you are off the mark on what's the true challenge. We [0] actually indexed "all" content on YouTube and dozens of other platforms. Not just metadata, but the videos. We could easily run search engine like this. In fact, we have explored it in the past. We also came up with page rank-like algorithm that allows us to bootstrap the experience and eventually move to usage based sorting. The true challenge is in customer acquisition. Google has such a strong hold on the browser market [1], that there is just no way to convince people to "go to pex" and search for videos. You can see it with DDG which after more a decade its penetration [2] is a fraction of Google's. Even Microsoft, that plunged billions of dollars into Bing is not able to acquire more than just few % of the market [3]. Finally, there are literally no investors to back this. After the fall of Blekko [4], nobody will touch this market. We spoke to dozens of investors. Most have a default position to "never compete with Google". Up until this doesn't change, there is no way to bring it to the market. [0] https://pex.com https://pex.com [1] https://www.theverge.com/2020/7/1/21310591/apple-google-search-engine-safari-iphone-deal-billions-regulation-antitrust https://www.theverge.com/2020/7/1/21310591/apple-google-sear... [2] https://duckduckgo.com/traffic https://duckduckgo.com/traffic [3] https://gs.statcounter.com/search-engine-market-share https://gs.statcounter.com/search-engine-market-share [4] https://en.wikipedia.org/wiki/Blekko https://en.wikipedia.org/wiki/Blekko
- jasode 6y ago
  >Not just metadata, but the videos. But calculating the fingerprints of the videos rather than store on disk and serve all the exabytes video means you're still at a fundamental _information_ disadvantage. You already know the following but I'll spell it out for readers who may not see the distinction: web surfers don't watch fingerprints... they watch the actual video bytes. Youtube employees and researchers found out that user behavior such as actual watch time of the videos was a _stronger_ voting signal of quality than the user hitting the subscribe button. To replicate that measurement, how would Pex get similar usage data without actually storing the exabytes of videos or convincing millions of web surfers to install a browser plugin to spy on their youtube usage? >The true challenge is in customer acquisition. Google has such a strong hold on the browser market [1], that there is just no way to convince people to "go to pex" and search for videos. I'm not convinced of your claim that the Chrome browser is the competitive moat that prevents Pex from rising up. E.g. Tik Tok got very popular without Chrome browser help. As another example, I found out that a gardening expert[1] gets most of her views from Facebook-hosted videos instead of Youtube. She gets more than 3x the views on Facebook. I don't have a Facebook account so I watch her on Youtube but it turns out I'm actually in the minority of her audience. [1] https://www.youtube.com/c/gardenanswer/videos https://www.youtube.com/c/gardenanswer/videos

by jasode 6y ago

mIOAMDA 6y ago

What if instead we had the recommendations from the main sites YouTube, Vimeo, Dailymotion and if you want your personal video hosting site to be recommended you would fill in a form to add you sites to the collection of sites the recommendation system crawl through? This might be helpful to avoid bad quality recommendations?

tylerdinner 6y ago

I don't feel these unavoidable disadvantages are as black and white and you describe. To circumvent the fake title/bait & switch practices, you could allow users to flag videos as misleading, vote them as spam, etc. Put the number of flags next to the video, add the ability to not even see flagged videos. Further, scripts could be used to report video play statistics to the scrape site. Drawback is users would have to add the script to their own site, but adding analytics scripts is common practice.

jasode 6y ago
>To circumvent the fake title/bait & switch practices, you could allow users to flag videos as misleading, vote them as spam, etc. And now you've re-created the "Amazon 1-star ratings left by competitors instead of real buyers" and/or the "false DMCA takedown claims of public domain or original music". E.g. an honest person creates a "How to sew a mask at home to protect from COVID" which is the real content but a bunch of coronavirus deniers falsely flag that video as "child pornography". Game theory is hard and everybody has to think through all the future chess moves that adversaries will use.
- tylerdinner 6y ago
  Damn you're totally right. You'd have to have a subscription fee and a team of mods to fix that.
antocv 6y ago
> To circumvent the fake title/bait & switch practices, you could allow users to flag videos as misleading, vote them as spam, etc. Put the number of flags next to the video, add the ability to not even see flagged videos. And who would flag the bad users? This just increases slightly the discomfort of bad guys, but does not prevent it at all. Instead they will flag competitive videos and such. Nobody has solved "decentralized trust" yet, if ever.

doh 6y ago

You bring some interesting points, but you are off the mark on what's the true challenge. We [0] actually indexed "all" content on YouTube and dozens of other platforms. Not just metadata, but the videos. We could easily run search engine like this. In fact, we have explored it in the past. We also came up with page rank-like algorithm that allows us to bootstrap the experience and eventually move to usage based sorting. The true challenge is in customer acquisition. Google has such a strong hold on the browser market [1], that there is just no way to convince people to "go to pex" and search for videos. You can see it with DDG which after more a decade its penetration [2] is a fraction of Google's. Even Microsoft, that plunged billions of dollars into Bing is not able to acquire more than just few % of the market [3]. Finally, there are literally no investors to back this. After the fall of Blekko [4], nobody will touch this market. We spoke to dozens of investors. Most have a default position to "never compete with Google". Up until this doesn't change, there is no way to bring it to the market. [0] https://pex.com https://pex.com [1] https://www.theverge.com/2020/7/1/21310591/apple-google-search-engine-safari-iphone-deal-billions-regulation-antitrust https://www.theverge.com/2020/7/1/21310591/apple-google-sear... [2] https://duckduckgo.com/traffic https://duckduckgo.com/traffic [3] https://gs.statcounter.com/search-engine-market-share https://gs.statcounter.com/search-engine-market-share [4] https://en.wikipedia.org/wiki/Blekko https://en.wikipedia.org/wiki/Blekko

jasode 6y ago
>Not just metadata, but the videos. But calculating the fingerprints of the videos rather than store on disk and serve all the exabytes video means you're still at a fundamental _information_ disadvantage. You already know the following but I'll spell it out for readers who may not see the distinction: web surfers don't watch fingerprints... they watch the actual video bytes. Youtube employees and researchers found out that user behavior such as actual watch time of the videos was a _stronger_ voting signal of quality than the user hitting the subscribe button. To replicate that measurement, how would Pex get similar usage data without actually storing the exabytes of videos or convincing millions of web surfers to install a browser plugin to spy on their youtube usage? >The true challenge is in customer acquisition. Google has such a strong hold on the browser market [1], that there is just no way to convince people to "go to pex" and search for videos. I'm not convinced of your claim that the Chrome browser is the competitive moat that prevents Pex from rising up. E.g. Tik Tok got very popular without Chrome browser help. As another example, I found out that a gardening expert[1] gets most of her views from Facebook-hosted videos instead of Youtube. She gets more than 3x the views on Facebook. I don't have a Facebook account so I watch her on Youtube but it turns out I'm actually in the minority of her audience. [1] https://www.youtube.com/c/gardenanswer/videos https://www.youtube.com/c/gardenanswer/videos