3 ms·
I've had this problem recently with parsing a webpage using some webscraping library, and returning nothing since everyone seems to be running a SPA now. It's i
by rabuse 5y ago
I've had this problem recently with parsing a webpage using some webscraping library, and returning nothing since everyone seems to be running a SPA now. It's infuriating that you pretty much need to run some headless browser to just scrape a damn website now.
- beiller 5y agoDon't spas make scrapers easier cause you can just call the APIs directly?
- BenjiWiebe 5y agoYes, if your bot understands that particular website's API. If your bot only scrapes one website, that's great. If your bot needs to scrape 100's of SPAs all with their own homegrown APIs, have fun.
- beiller 5y agoI know that it is a different challenge reverse engineering a spa's APIs but you gotta admit, the existing system of XML / html parsing and websites making slight tweaks to their layout is the real nightmare.
- phist_mcgee 5y agoIt's been like this for years, google's own bots struggle to interpret a lot of sites without heavy manual intervention by the site developers.