4 ms·

People love BeautifulSoup, but I've had better luck with fully emulating a browser with Selenium. While more complicated it seems to run into less issues. Is th

• 131012 6y ago
  When a website uses javascript to dynamically render information, Selenium (or any other way to use a headless browser) is usually required.
• d0mine 6y ago
  They are orthogonal: 1. you could use selenium to get javascript-generated html 2. then use beautifulsoup to extract the desired info from the html.
• gabrielsroka 6y ago
  I looked into using Selenium, but it seems like you have to download an 80 MB executable. If you noticed above, I also posted JavaScript that you could run in the browser that you already have. It seemed like a better way to go. I admit Selenium does have some useful features, too.
• derivagral 6y ago
  Nowadays you could just use headless Chrome[1] or Firefox[2], though it looks like Firefox is lagging behind a bit and still wants to require Selenium. Personally, I loved using BS for hobbies until the SPA era started, and then I had to either use headless (selenium back then was great) and/or monitor the network for their API. [1] https://developers.google.com/web/updates/2017/04/headless-chrome https://developers.google.com/web/updates/2017/04/headless-c... [2] https://developer.mozilla.org/en-US/docs/Mozilla/Firefox/Headless_mode https://developer.mozilla.org/en-US/docs/Mozilla/Firefox/Hea...