3 ms·
I believe you have to look at robots.txt file. If it lets you crawl, you should be fine.
by rgovind 13y ago
I believe you have to look at robots.txt file. If it lets you crawl, you should be fine.
- qq66 13y agoDoes the robots.txt have any legal significance? I would think that the Terms of Use can be more thorough and are more legally relevant.