4 ms·

Yes, and google has been observed several times to ignore the robots.txt file

• Google234 6y ago
  No, this is true in general. Maybe there’s been some bugs (I would be surprised, can you link these multiple examples?) but Google respects robot.txt files