5 ms·

I tested this with Facebook a while back. I put two videos up of various lengths and linked to them directly by IP in Facebook chat. I also included a restricti

I tested this with Facebook a while back. I put two videos up of various lengths and linked to them directly by IP in Facebook chat. I also included a restrictive robots.txt. In both cases Facebook downloaded the entire videos from my server. I repeated the experiment with several other providers and the results were varied. Skype, for example, does not download the entire video and seems to respect the robots.txt... Not sure if this is still the case for Skype but, I just tested on FB again and they pulled the whole video...

0x0 13y ago
Isn't that typically done to show thumbnail previews under the pasted links?
- choult 13y ago
  Facebook generally will present the OpenGraph data from pasted links; for sites lacking it they might present the first image on the page. It would be vast overkill to download a whole video just to present a thumbnail, and the user experience in the delay while the whole file is downloaded and processed would render the feature fairly useless. Which leaves the question of why Facebook would do this open... It might just be really bad practice and an under-/non-optimized crawler.
  - Cthulhu_ 13y ago
    Or they download the whole video to pull it through an analyzer to look for stuff prohibited to be republished via FB, such as child pornography. If I were FB, I'd screen everything published via my site.
    - pyre 13y ago
      Right, because we have automated software child pornography recognizers.
      - ceejayoz 13y ago
        It wouldn't be surprising. YouTube looks for video fingerprints of copyrighted data. Repurposing such technology to find copies of known CP videos would be pretty straightforward. Wouldn't help you with new material, of course.
        pyre 13y ago
        If you start filtering for child pornography, and people know about it, they will come at your hard if you miss anything. Even for a technology that gets all known child porn videos, people would be beating down the door to sue/bad mouth Facebook for missing the new ones. Mostly due to people being people, but also because the technology is 'magical' to far too many people in society so they feel like there are no limitations (or it's hard for them to grasp them).
      - runamok 13y ago
        They already have image algorithms to detect nudity with a good degree of accuracy so that's probably enough to forbid an image or video.
        pyre 13y ago
        Like that won't cause a shitstorm. A social networking site that won't let people trade possibly nude images privately?
        kamjam 13y ago
        I don't think he's talking about necessarily private, but they detect nudity on the "publicly" available images available in your albums which your friends an see... http://petapixel.com/2012/11/25/facebook-removes-risque-photograph-of-woman-showing-an-elbow/ http://petapixel.com/2012/11/25/facebook-removes-risque-phot...
        fnordfnordfnord 13y ago
        That sort of thing might also flag a benign "Baby's first bath" video as child porn.
    - artursapek 13y ago
      Child porn detection bot: add that to the list of projects I never want to work on.
      - jlgreco 13y ago
        I have to imagine they contract that work out. Unless doing that was my business, I wouldn't want to touch that training set with a thousand foot pole. Well I wouldn't regardless..., but you see what I'm saying about encapsulating and isolating that.
        rmc 13y ago
        I've always wondered about that. Surely possesing the 'training data' (basically child porn) is illegal. Only police can posses it for investigations etc. So how can you make something that can recongnise it?
        steveklabnik 13y ago
        It's humans. There was a story about the 'dark side of Google' within the last few months about how Google contracts this out, and they burn out, but I'm not searching for that story directly myself.
        Spooky23 13y ago
        There are companies who have special licenses, etc to do that type of work. I was supposed to meet with a Scandinavian vendor a few years ago that provides a service where you can match SHA hashes of files against known illegal content. I recall the pitch deck said that they had millions of records to match against, which I found incredibly depressing. I ended up bailing on the meeting... it's a subject I don't want to learn more about.
  - pbhjpbhj 13y ago
    >It might just be really bad practice and an under-/non-optimized crawler. // I know they've got lots of money but really, are they wasting bandwidth and memory so badly as to download terrabytes of video they don't need?
- uncoder0 13y ago
  That is certainly one possible explanation. For fun if you share a Google Drive video you can watch the FB user popup on the active user list and then disappear after a certain amount of time that varies based on video length and quality.
snowwrestler 13y ago
Try posting a link to a page on the Pirate Bay in Facebook comments and see what happens.

0x0 13y ago

Isn't that typically done to show thumbnail previews under the pasted links?

choult 13y ago
Facebook generally will present the OpenGraph data from pasted links; for sites lacking it they might present the first image on the page. It would be vast overkill to download a whole video just to present a thumbnail, and the user experience in the delay while the whole file is downloaded and processed would render the feature fairly useless. Which leaves the question of why Facebook would do this open... It might just be really bad practice and an under-/non-optimized crawler.
- Cthulhu_ 13y ago
  Or they download the whole video to pull it through an analyzer to look for stuff prohibited to be republished via FB, such as child pornography. If I were FB, I'd screen everything published via my site.
  - pyre 13y ago
    Right, because we have automated software child pornography recognizers.
    - ceejayoz 13y ago
      It wouldn't be surprising. YouTube looks for video fingerprints of copyrighted data. Repurposing such technology to find copies of known CP videos would be pretty straightforward. Wouldn't help you with new material, of course.
      - pyre 13y ago
        If you start filtering for child pornography, and people know about it, they will come at your hard if you miss anything. Even for a technology that gets all known child porn videos, people would be beating down the door to sue/bad mouth Facebook for missing the new ones. Mostly due to people being people, but also because the technology is 'magical' to far too many people in society so they feel like there are no limitations (or it's hard for them to grasp them).
    - runamok 13y ago
      They already have image algorithms to detect nudity with a good degree of accuracy so that's probably enough to forbid an image or video.
      - pyre 13y ago
        Like that won't cause a shitstorm. A social networking site that won't let people trade possibly nude images privately?
        kamjam 13y ago
        I don't think he's talking about necessarily private, but they detect nudity on the "publicly" available images available in your albums which your friends an see... http://petapixel.com/2012/11/25/facebook-removes-risque-photograph-of-woman-showing-an-elbow/ http://petapixel.com/2012/11/25/facebook-removes-risque-phot...
      - fnordfnordfnord 13y ago
        That sort of thing might also flag a benign "Baby's first bath" video as child porn.
  - artursapek 13y ago
    Child porn detection bot: add that to the list of projects I never want to work on.
    - jlgreco 13y ago
      I have to imagine they contract that work out. Unless doing that was my business, I wouldn't want to touch that training set with a thousand foot pole. Well I wouldn't regardless..., but you see what I'm saying about encapsulating and isolating that.
      - rmc 13y ago
        I've always wondered about that. Surely possesing the 'training data' (basically child porn) is illegal. Only police can posses it for investigations etc. So how can you make something that can recongnise it?
        steveklabnik 13y ago
        It's humans. There was a story about the 'dark side of Google' within the last few months about how Google contracts this out, and they burn out, but I'm not searching for that story directly myself.
        Spooky23 13y ago
        There are companies who have special licenses, etc to do that type of work. I was supposed to meet with a Scandinavian vendor a few years ago that provides a service where you can match SHA hashes of files against known illegal content. I recall the pitch deck said that they had millions of records to match against, which I found incredibly depressing. I ended up bailing on the meeting... it's a subject I don't want to learn more about.
- pbhjpbhj 13y ago
  >It might just be really bad practice and an under-/non-optimized crawler. // I know they've got lots of money but really, are they wasting bandwidth and memory so badly as to download terrabytes of video they don't need?
uncoder0 13y ago
That is certainly one possible explanation. For fun if you share a Google Drive video you can watch the FB user popup on the active user list and then disappear after a certain amount of time that varies based on video length and quality.

snowwrestler 13y ago

Try posting a link to a page on the Pirate Bay in Facebook comments and see what happens.