2 ms·
You should consider focusing on intercepting network requests. Most if not all sites I scrape end up fetching data from some api. Like others have said, if you
by samsullivan 3y ago
You should consider focusing on intercepting network requests. Most if not all sites I scrape end up fetching data from some api. Like others have said, if you instead had the LLM create an ad hoc script for the scraping task and then use the feedback loop to continuously improve the outputs it would be really cool. I'd pay between $5 - $50 for each working output script.
- suchintan 3y agoWe're definitely planning this. Skyvern currently intercepts all network calls that gets made -- we save them all as a HAR file for debugging purposes... but never looks at them A lot lot lot of scraping use-cases become simpler if you can just inspect a search api or a details API and get the information you're looking for. I'll add this to our roadmap!