4 ms·

wget?

• acomjean 8y ago
  wget with wildcards is what we use to get biology data. Those are ftp sites, so some experimentation may be necessary to make it work with http...
  • luke0016 8y ago
    What? No, it works fine with http out of the box. From the man page: GNU Wget is a free utility for non-interactive download of files from the Web. It supports HTTP, HTTPS, and FTP protocols, as well as retrieval through HTTP proxies.
    • ejstronge 8y ago
      I think the GP's uncertainty relates to download URL wildcards with HTTP. Unlike FTP, I don't believe that HTTP servers have to support your request to list all files in a directory.
      • luke0016 8y ago
        Thankfully the particular website in question already spits out a file listing for you.