Which way to go will depend on your use-case and what websites you want to scrape. For occasional web crawling, headless browser is great as it's easy to set u

Which way to go will depend on your use-case and what websites you want to scrape. For occasional web crawling, headless browser is great as it's easy to set up and you're almost guaranteed to get the data you need. For frequent or large-scale crawling, it's a different story. Surely you can implement something with just HTTP library, but you'll need to test it throughly and make some research before hand. That said, most scraped, content-heavy websites use either static HTML or SSR, in which case you can use HTTP no problem.