I do a bit of scraping for hobby projects, and much of that comes down to basically this (but I do it in node instead of python). Sometimes you need to use jsdo

I do a bit of scraping for hobby projects, and much of that comes down to basically this (but I do it in node instead of python). Sometimes you need to use jsdom or puppeteer, but the second step (after checking if there are official data dumps made available or some official API) is always checking the full data flow in devtools if there is some undocumented way to more quickly get the raw data I want.