3 ms·

I did some research, but did not found enough conclusive evidence whether it is legal or not. So if I receive a claim from Google legal team, I will take the s

I did some research, but did not found enough conclusive evidence whether it is legal or not. So if I receive a claim from Google legal team, I will take the site down.

3dfan 8y ago
Yes, crawling and publishing crawled data seems to be a very complicated, grey area.
pr33tish 8y ago
Could you please point out the exact URL you are crawling? Love to match that with robots.txt file. I'm all for improving publicly available data.