2 ms·
As many pages are at least half-way SPAs, make sure to really understand the website's communication with their backend. Identify API calls and try to make API
by philippz 7y ago
As many pages are at least half-way SPAs, make sure to really understand the website's communication with their backend. Identify API calls and try to make API calls directly instead of downloading the full pages and extracting the required information from HTML afterwards. If you have certain data sets from specific API calls that almost never change, try to crawl them less regularly and instead cache the results.