As others pointed out, no, you can not rely on HEAD requests. You may send conditional GET's though, as you pointed out: by using ETag/If-None-Match and/or Last

As others pointed out, no, you can not rely on HEAD requests. You may send conditional GET's though, as you pointed out: by using ETag/If-None-Match and/or Last-Modified/If-Modified-Since. Keeping a hash database in order to track changes is also reliable. You can also place a proxy between you and the rest of the world in order to deal with that, if the disk space isn't an issue. I wrote a couple of services that need to poll for remote resources. They look like your usage model. I'd advise against PHP though. We already tried that and failed. Several times actually. The timeouts kill the workload if the remote isn't responding. Using an async framework like node.js proved to be more appropriate.