Good questions! For this particular project I needed access to more than just the page itself - I needed to interact with scripts and have access to the page r

Good questions! For this particular project I needed access to more than just the page itself - I needed to interact with scripts and have access to the page resources on a level which matched that of a browser. What you're seeing is only the tip of a much larger iceberg, for which efficiency wasn't the main objective. You're correct that you can write more efficient spiders if you don't want to do anything fancy (indeed, I've written simple scrapers and spiders in various languages), but this project leant itself better to a full browser environment.