3 ms·
The idea is more interesting when you think about scrapers. Take any ecommerce website, there are several scrapers that download all pages every hours, it would
by polote 5y ago
The idea is more interesting when you think about scrapers. Take any ecommerce website, there are several scrapers that download all pages every hours, it would be more efficient if a provider had a live copy of the website and then serve the requests to the scrapers or could even send webhook.
A website could handle tons of scrappers without having high bandwidth, only the provider will need high bandwidth.
The issue is that scrapers often play with cookies and dynamic websites and such solution wouldn't work on these cases
- akouri 5y agoscraper*
- azalemeth 5y ago(in the nicest way and since your post is recent, please edit s/scrappers/scrapers! It very much changed how I read it first time around -- I thought you were referring to a type of failed startup!)
- polote 5y agoThanks, I need to improve my english haha
- atdrummond 5y agoIs scraper one “p” or two? My inclination would be scrapper is “scrap”-er rather than “scrape”-er.
- pjc50 5y agoScrapers. A scrAper is the thing you use to remove ice from windscreens or scrape data from websites. A scraPPer is someone who likes getting into fights or collecting scrap metal.