4 ms·

https://github.com/fake-name/xA-Scraper https://github.com/fake-name/xA-Scraper already supports tumblr. Their API has annoying limits, though. Note: It's a pr

• toomuchtodo 9y ago
  I use grab site, which is a spiritual successor to ArchiveTeam's ArchiveBot. I have to be careful to give my scraping methods away, as I've had digital targets attempt countermeasures.
  • fake-name 9y ago
    I've had similar problems. At this point, I basically run a botnet, albeit one I pay for. I have a rolling swarm of DigitalOcean and Vultr VMs that act as RPC clients for a custom system I wrote.
    • toomuchtodo 9y ago
      We should collaborate.
      • fake-name 9y ago
        All my junk is on github already: https://github.com/fake-name/ReadableWebProxy https://github.com/fake-name/ReadableWebProxy https://github.com/fake-name/AutoTriever https://github.com/fake-name/AutoTriever https://github.com/fake-name/IntraArchiveDeduplicator https://github.com/fake-name/IntraArchiveDeduplicator https://github.com/fake-name/MangaCMS https://github.com/fake-name/MangaCMS https://github.com/fake-name/DanbooruScraper https://github.com/fake-name/DanbooruScraper https://github.com/fake-name/PatreonArchiver https://github.com/fake-name/PatreonArchiver https://github.com/fake-name/ChromeController https://github.com/fake-name/ChromeController I've been meaning to integrate wpull into my scraper structure, so I can also ship what I scrape off to the internet archive, but I've not had the motivation. https://github.com/fake-name https://github.com/fake-name