For personal projects I generally follow the steps this blog lays out; start with the light and low-level APIs and work my way up as needed. I do usually skip o

For personal projects I generally follow the steps this blog lays out; start with the light and low-level APIs and work my way up as needed. I do usually skip over sending raw sockets when I start, but I think knowing them is worthwhile for troubleshooting and optimizing. I often find myself jumping to different levels when navigating scraping--from http headers to javascript rendering. While you can touch most of those things with requests, I find it easier to reproduce exactly what I see my browser doing with lower level APIs. The backend might be tightly-coupled with the front end. So you might get stuck on a specific header, user-agent string, or something often related to sessions or login.