3 ms·
I've never actually reached out to a webmaster to ask permission, but I think that's a great idea. (They may even have some suggestions for a better way to achi
by cushychicken 5y ago
I've never actually reached out to a webmaster to ask permission, but I think that's a great idea. (They may even have some suggestions for a better way to achieve what I'm doing.)
How do you typically find contact info for someone like that?
I'm running very generous politeness intervals at the moment to try and ensure I'm not a nuisance - one query every two seconds.
- marginalia_nu 5y agoIf you can't find the address on the website, sometimes it's in robots.txt, you could also try support@domain, admin@domain or webmaster@domain. Contact forms can as well, usually they seem to get forwarded to the IT department if they look like they have IT stuff in them, and I've had reasonable success getting hold of the right people that way.
- cushychicken 5y agoI'll give that a shot! Do you know if this is generally an in-house position for companies that use third party platforms? I ask because Workday has been the absolute bane of my indexing existence, and I suspect they make it hard so they can own distribution of jobs to approved search engines. (Makes it easier to upcharge that way, I suppose.) If the administrator for the job site is the Workday customer (i.e. Qualcomm or NXP or whoever is using Workday to host their job ads), I'd suspect I'd have a chance at getting a better way to index their jobs. (My god, I'd love API access if that's a thing I can get. I'd be a fly on the wall in most cases - one index a day is plenty for my purposes!)
- Terry_Roll 5y agoIf their security is any good, their firewall may tarpit you anyway if they can see you are spidering links quicker than a human can read, your browser agent offers no clues and/or your ip address (range) offers no clues.