3 ms·

Did you do some research of the legality of this?

• laurynas-s 8y ago
  I did some research, but did not found enough conclusive evidence whether it is legal or not. So if I receive a claim from Google legal team, I will take the site down.
  • 3dfan 8y ago
    Yes, crawling and publishing crawled data seems to be a very complicated, grey area.
  • pr33tish 8y ago
    Could you please point out the exact URL you are crawling? Love to match that with robots.txt file. I'm all for improving publicly available data.