3 ms·

What an absolutely insufferable explanation from ArchiveTeam. What else do you expect from an organization aggressively crawling websites and bringing them down

What an absolutely insufferable explanation from ArchiveTeam. What else do you expect from an organization aggressively crawling websites and bringing them down to their knees because they couldn't care less?

rossng 7mo ago
I'm curious to hear about examples of where this has happened. Because ArchiveTeam also has an important role in rescuing cultural artefacts that have been taken into private hands and then negligently destroyed.
- tredre3 7mo ago
  Having a laudable goal doesn't absolve them from bad behavior.
  - Dylan16807 7mo ago
    It's a good reason to not worry about hypothetical bad behavior and wait for evidence of real bad behavior.
tech234a 7mo ago
That page was written by Jason Scott in 2011 and has barely been changed since then.
- textfiles 7mo ago
  Why mess with perfection?
wlonkly 7mo ago
ArchiveTeam (which is not the Internet Archive) aggressively crawls websites because they care a lot, because the website in question is about to go away. Heck, I'd say as caring goes, ArchiveTeam cares more than the owners of the website, because in the ideal shutdown, the owners provide the data instead of forcing people to scrape it if they want to retain it after the site shuts down.
- sunaookami 7mo ago
  They also crawl aggressively when the site is not in danger. They crawled my MediaWiki because someone else input the site in their bot and it overloaded the PHP process. I know that archiving is important but please, not like this.
  - 47282847 7mo ago
    “Their bot” is a software anyone can run.
    - sunaookami 7mo ago
      So it's... their bot
pabs3 7mo ago
ArchiveTeam definitely do not intend to kill websites with too fast crawling, but definitely have done that unintentionally and always will stop/slow the crawling when it happens. Even the distributed crawling system has monitoring and controls to ensure it doesn't kill sites.

rossng 7mo ago

I'm curious to hear about examples of where this has happened. Because ArchiveTeam also has an important role in rescuing cultural artefacts that have been taken into private hands and then negligently destroyed.

tredre3 7mo ago
Having a laudable goal doesn't absolve them from bad behavior.
- Dylan16807 7mo ago
  It's a good reason to not worry about hypothetical bad behavior and wait for evidence of real bad behavior.

tech234a 7mo ago

That page was written by Jason Scott in 2011 and has barely been changed since then.

textfiles 7mo ago
Why mess with perfection?

wlonkly 7mo ago

ArchiveTeam (which is not the Internet Archive) aggressively crawls websites because they care a lot, because the website in question is about to go away. Heck, I'd say as caring goes, ArchiveTeam cares more than the owners of the website, because in the ideal shutdown, the owners provide the data instead of forcing people to scrape it if they want to retain it after the site shuts down.

sunaookami 7mo ago
They also crawl aggressively when the site is not in danger. They crawled my MediaWiki because someone else input the site in their bot and it overloaded the PHP process. I know that archiving is important but please, not like this.
- 47282847 7mo ago
  “Their bot” is a software anyone can run.
  - sunaookami 7mo ago
    So it's... their bot

pabs3 7mo ago

ArchiveTeam definitely do not intend to kill websites with too fast crawling, but definitely have done that unintentionally and always will stop/slow the crawling when it happens. Even the distributed crawling system has monitoring and controls to ensure it doesn't kill sites.