couple of additional points: - encoding - your input will have different encodings and its quite hard to guess the correct one, however you should at least try

couple of additional points: - encoding - your input will have different encodings and its quite hard to guess the correct one, however you should at least try to convert everything into 1 encoding (f.e. utf8) - by setting CURLOPT_ENCODING to '' you don't have to worry about (un)gzipping as curl ll do this for you (or it should) - it might be a good idea to use url hash as url id (f.e. crc32) - you should check content-length and content-type to avoid downloading huge files btw your coding style is very disturbing. there shouldn't be spaces before or after ->