4 ms·
Why can’t you just make your web crawler look like fb or googlebot (via user agent)? Do website owners actually check the ip?
by jtsiskin 6y ago
Why can’t you just make your web crawler look like fb or googlebot (via user agent)?
Do website owners actually check the ip?
- colinmhayes 6y agoSounds like this company is checking that the ip is from facebook. That would probably work on less secure sites though.