3 ms·

Did you respect robots.txt?

• 333c 9y ago
  It sounds to me like an obvious no, if they have a large team to get around countermeasures.
• rk06 9y ago
  Considering the effort that went into it. I am pretty sure crawling robot.txt links was their P1 requirement.