author here. AMA. since I wrote this, i started experimenting with grab-site: https://github.com/ludios/grab-site https://github.com/ludios/grab-site it's a

author here. AMA. since I wrote this, i started experimenting with grab-site: https://github.com/ludios/grab-site https://github.com/ludios/grab-site it's a wrapper around (and a fork of) wpull but the main advantage over it is it can do on-the-fly reconfiguration of delay, concurrency, ignore patterns and so on. it also provides a nice web interface. if you're only crawling one site every once in a while, wpull and crawl are fine, but for larger projects, grab-site is a must.