7 ms·

They're also forcing all adult-oriented content on Tumblr (which isn't limited to porn, although porn makes up the majority of their userbase) behind a logged-i

They're also forcing all adult-oriented content on Tumblr (which isn't limited to porn, although porn makes up the majority of their userbase) behind a logged-in-users-only wall, blocking it from non-Tumblr users and in turn, external search engine results. https://techcrunch.com/2017/06/20/tumblr-rolls-out-new-content-filtering-tools-with-launch-of-safe-mode/ https://techcrunch.com/2017/06/20/tumblr-rolls-out-new-conte...

deleted 9y ago
[deleted]
toomuchtodo 9y ago
Oi, I suppose archiving Tumblr is this weekend's project.
- fweespeech 9y ago
  Is archiving a bunch of porn really safe legally? I always had the impression archiving a large number of copyrighted images wasn't.
  - toomuchtodo 9y ago
    It goes into cold storage for a later date.
    - fweespeech 9y ago
      Thanks. :)
- fake-name 9y ago
  https://github.com/fake-name/xA-Scraper https://github.com/fake-name/xA-Scraper already supports tumblr. Their API has annoying limits, though. Note: It's a project of mine.
  - toomuchtodo 9y ago
    I use grab site, which is a spiritual successor to ArchiveTeam's ArchiveBot. I have to be careful to give my scraping methods away, as I've had digital targets attempt countermeasures.
    - fake-name 9y ago
      I've had similar problems. At this point, I basically run a botnet, albeit one I pay for. I have a rolling swarm of DigitalOcean and Vultr VMs that act as RPC clients for a custom system I wrote.
      - toomuchtodo 9y ago
        We should collaborate.
        fake-name 9y ago
        All my junk is on github already: https://github.com/fake-name/ReadableWebProxy https://github.com/fake-name/ReadableWebProxy https://github.com/fake-name/AutoTriever https://github.com/fake-name/AutoTriever https://github.com/fake-name/IntraArchiveDeduplicator https://github.com/fake-name/IntraArchiveDeduplicator https://github.com/fake-name/MangaCMS https://github.com/fake-name/MangaCMS https://github.com/fake-name/DanbooruScraper https://github.com/fake-name/DanbooruScraper https://github.com/fake-name/PatreonArchiver https://github.com/fake-name/PatreonArchiver https://github.com/fake-name/ChromeController https://github.com/fake-name/ChromeController I've been meaning to integrate wpull into my scraper structure, so I can also ship what I scrape off to the internet archive, but I've not had the motivation. https://github.com/fake-name https://github.com/fake-name
- Veratyr 9y ago
  They currently have 150B posts (https://www.tumblr.com/about https://www.tumblr.com/about) so I expect that to be quite challenging. If you seriously make any progress though, it'd be cool to see.
  - nickpsecurity 9y ago
    Step 1: Hack into a supercomputing center with at least 1Gbps line. Step 2: Massively-parallel downloading of all the sites using clustered nodes, compression of it, and resulting data stored into high-performance, clustered filesystem. Step 3: Move it off of there when traffic is low or overnight if system doesn't go offline overnight.
    - Veratyr 9y ago
      Step 0: Acquire at least ~1PB of storage to store all the data.
      - toomuchtodo 9y ago
        I have over 1PB of storage at my disposal.
        Veratyr 9y ago
        Business or personal? It's doable but it's a lot of money to buy all those drives.
        boulos 9y ago
        $6k for 600T fully assembled backblaze storage pod: https://www.backuppods.com https://www.backuppods.com (no affiliation, just pointing out that ~$12k isn't a massive amount).
        noisem4ker 9y ago
        Note that hard drives are not included. 1000TB / 8TB = 125 HDDs 125 * $200 = $25k
- voltagex_ 9y ago
  http://www.archiveteam.org/index.php?title=Tumblr http://www.archiveteam.org/index.php?title=Tumblr
tptacek 9y ago
Porn makes up the majority of Tumblr's userbase?
- waterphone 9y ago
  Tumblr has a huge porn community, and always has. I don't have actual statistics, but it's huge. In general, people on tumblr might have a regular, personal blog, and then a side blog where they reblog and/or comment on/add captions to pornographic images, gifs and videos. In that sense, they can sort of curate porn that interests and appeals to them. The addition of this personal touch and intellectual component appeals to a lot of people, including a lot of women. But there are also various sex advice blogs, etc., which will be affected by this change as well, artists who do non-pornographic nude artwork, and potentially even various LGBT communities where sex can at times be a topic. There are various levels of NSFW-flagging on Tumblr: users flagging their own site NSFW, Tumblr marking a blog NSFW and the user having no way to change that, or enough individual posts being flagged NSFW by users and AI that Tumblr decides to call the entire blog NSFW.
  - antoineMoPa 9y ago
    Maybe your vision of tumblr's userbase is biased by the kind of people you follow. Personally, I'm on the "computer science/glitch art/study/science" part of tumblr and I don't see much NSFW content (Apart from some bots who follow me which I block automatically).
    - waterphone 9y ago
      The areas are kept quite separate, but that doesn't mean that there isn't still a huge porn userbase. Many people on tumblr have multiple blogs on different subjects, and deliberately keep their porn blog(s) separate from the rest.
  - rmc 9y ago
    Sounds a lot like what happened with LiveJournal. About 10-15 years ago LJ was where all the LGBT and niche erotic interests where, like fanfic. It's been bought by a Russian company and they've been turning the screws on the LGBT users for years.
btym 9y ago
This is a direction they headed in shortly after Yahoo acquired them, it's not a new thing with the Verizon acquisition. Blogs marked "adult" and NSFW posts already won't show up in search engines[edit: I stand corrected], by default in searches on the website, or searches on the mobile app (even when logged in).
- waterphone 9y ago
  That is not true at all. All Tumblr blogs, including NSFW blogs, have always been viewable and indexable by external search engines through their public URL (blogname.tumblr.com) unless flagged as private/hidden (a choice of the blog owner). They have only been hidden from Tumblr's search engine if the user has their account set to not see adult content. You can verify this by searching for "tumblr porn" on Google. Five of the results on the first page are porn blogs on Tumblr.
  - btym 9y ago
    >All Tumblr blogs, including NSFW blogs, have always been viewable and indexable by external search engines through their public URL (blogname.tumblr.com) unless flagged as private/hidden (a choice of the blog owner). There is definitely something in account settings somewhere that removes you from search results, I may have mixed that up with the NSFW flag. >They have only been hidden from Tumblr's search engine if the user has their account set to not see adult content As I mentioned, this is default. And "adult" blogs do not show up at all in search results on mobile. My point stands, though. Yahoo already started making adult content harder to find (accidentally or intentionally), the momentum is already there.
    - waterphone 9y ago
      There are three visibility flags for Tumblr blogs: - Allow logged-out users to see this blog - Allow this blog to appear in search results - Flag this blog as adult-oriented Either of the first two flags would obscure the blog from search engines, but the latter one has never.

deleted 9y ago

[deleted]

toomuchtodo 9y ago

Oi, I suppose archiving Tumblr is this weekend's project.

fweespeech 9y ago
Is archiving a bunch of porn really safe legally? I always had the impression archiving a large number of copyrighted images wasn't.
- toomuchtodo 9y ago
  It goes into cold storage for a later date.
  - fweespeech 9y ago
    Thanks. :)
fake-name 9y ago
https://github.com/fake-name/xA-Scraper https://github.com/fake-name/xA-Scraper already supports tumblr. Their API has annoying limits, though. Note: It's a project of mine.
- toomuchtodo 9y ago
  I use grab site, which is a spiritual successor to ArchiveTeam's ArchiveBot. I have to be careful to give my scraping methods away, as I've had digital targets attempt countermeasures.
  - fake-name 9y ago
    I've had similar problems. At this point, I basically run a botnet, albeit one I pay for. I have a rolling swarm of DigitalOcean and Vultr VMs that act as RPC clients for a custom system I wrote.
    - toomuchtodo 9y ago
      We should collaborate.
      - fake-name 9y ago
        All my junk is on github already: https://github.com/fake-name/ReadableWebProxy https://github.com/fake-name/ReadableWebProxy https://github.com/fake-name/AutoTriever https://github.com/fake-name/AutoTriever https://github.com/fake-name/IntraArchiveDeduplicator https://github.com/fake-name/IntraArchiveDeduplicator https://github.com/fake-name/MangaCMS https://github.com/fake-name/MangaCMS https://github.com/fake-name/DanbooruScraper https://github.com/fake-name/DanbooruScraper https://github.com/fake-name/PatreonArchiver https://github.com/fake-name/PatreonArchiver https://github.com/fake-name/ChromeController https://github.com/fake-name/ChromeController I've been meaning to integrate wpull into my scraper structure, so I can also ship what I scrape off to the internet archive, but I've not had the motivation. https://github.com/fake-name https://github.com/fake-name
Veratyr 9y ago
They currently have 150B posts (https://www.tumblr.com/about https://www.tumblr.com/about) so I expect that to be quite challenging. If you seriously make any progress though, it'd be cool to see.
- nickpsecurity 9y ago
  Step 1: Hack into a supercomputing center with at least 1Gbps line. Step 2: Massively-parallel downloading of all the sites using clustered nodes, compression of it, and resulting data stored into high-performance, clustered filesystem. Step 3: Move it off of there when traffic is low or overnight if system doesn't go offline overnight.
  - Veratyr 9y ago
    Step 0: Acquire at least ~1PB of storage to store all the data.
    - toomuchtodo 9y ago
      I have over 1PB of storage at my disposal.
      - Veratyr 9y ago
        Business or personal? It's doable but it's a lot of money to buy all those drives.
        boulos 9y ago
        $6k for 600T fully assembled backblaze storage pod: https://www.backuppods.com https://www.backuppods.com (no affiliation, just pointing out that ~$12k isn't a massive amount).
        noisem4ker 9y ago
        Note that hard drives are not included. 1000TB / 8TB = 125 HDDs 125 * $200 = $25k
voltagex_ 9y ago
http://www.archiveteam.org/index.php?title=Tumblr http://www.archiveteam.org/index.php?title=Tumblr

tptacek 9y ago

Porn makes up the majority of Tumblr's userbase?

waterphone 9y ago
Tumblr has a huge porn community, and always has. I don't have actual statistics, but it's huge. In general, people on tumblr might have a regular, personal blog, and then a side blog where they reblog and/or comment on/add captions to pornographic images, gifs and videos. In that sense, they can sort of curate porn that interests and appeals to them. The addition of this personal touch and intellectual component appeals to a lot of people, including a lot of women. But there are also various sex advice blogs, etc., which will be affected by this change as well, artists who do non-pornographic nude artwork, and potentially even various LGBT communities where sex can at times be a topic. There are various levels of NSFW-flagging on Tumblr: users flagging their own site NSFW, Tumblr marking a blog NSFW and the user having no way to change that, or enough individual posts being flagged NSFW by users and AI that Tumblr decides to call the entire blog NSFW.
- antoineMoPa 9y ago
  Maybe your vision of tumblr's userbase is biased by the kind of people you follow. Personally, I'm on the "computer science/glitch art/study/science" part of tumblr and I don't see much NSFW content (Apart from some bots who follow me which I block automatically).
  - waterphone 9y ago
    The areas are kept quite separate, but that doesn't mean that there isn't still a huge porn userbase. Many people on tumblr have multiple blogs on different subjects, and deliberately keep their porn blog(s) separate from the rest.
- rmc 9y ago
  Sounds a lot like what happened with LiveJournal. About 10-15 years ago LJ was where all the LGBT and niche erotic interests where, like fanfic. It's been bought by a Russian company and they've been turning the screws on the LGBT users for years.

btym 9y ago

This is a direction they headed in shortly after Yahoo acquired them, it's not a new thing with the Verizon acquisition. Blogs marked "adult" and NSFW posts already won't show up in search engines[edit: I stand corrected], by default in searches on the website, or searches on the mobile app (even when logged in).

waterphone 9y ago
That is not true at all. All Tumblr blogs, including NSFW blogs, have always been viewable and indexable by external search engines through their public URL (blogname.tumblr.com) unless flagged as private/hidden (a choice of the blog owner). They have only been hidden from Tumblr's search engine if the user has their account set to not see adult content. You can verify this by searching for "tumblr porn" on Google. Five of the results on the first page are porn blogs on Tumblr.
- btym 9y ago
  >All Tumblr blogs, including NSFW blogs, have always been viewable and indexable by external search engines through their public URL (blogname.tumblr.com) unless flagged as private/hidden (a choice of the blog owner). There is definitely something in account settings somewhere that removes you from search results, I may have mixed that up with the NSFW flag. >They have only been hidden from Tumblr's search engine if the user has their account set to not see adult content As I mentioned, this is default. And "adult" blogs do not show up at all in search results on mobile. My point stands, though. Yahoo already started making adult content harder to find (accidentally or intentionally), the momentum is already there.
  - waterphone 9y ago
    There are three visibility flags for Tumblr blogs: - Allow logged-out users to see this blog - Allow this blog to appear in search results - Flag this blog as adult-oriented Either of the first two flags would obscure the blog from search engines, but the latter one has never.