4 ms·

Wouldn’t it be possible to create a third-party site scraper that could generate custom RSS feeds independently of the company? About the only barrier would be

Wouldn’t it be possible to create a third-party site scraper that could generate custom RSS feeds independently of the company? About the only barrier would be paywalls, IMO.

lgl 6y ago
Yes, and even with paywalls, as long as the source has login details it can still be done (although this would probably be illegal). Anecdotal evidence: I work in a tour operator and we publish our products on tripadvisor and getyourguide which are two very popular places to advertise these products and that provide an extranet to suppliers where they can login to create and manage products, availability, etc. A few years ago before they had developed APIs or had only private APIs for the "big guys" and wouldn't really pay any attention to small shops' requests, I managed to use Puppeteer [0] [1] (a node.js library to manage headless Chrome) to dynamically update our products into their extranet. It's very powerful and with some tinkering you can automate pretty much any behavior, including authentication and xhr interactions inside their platform (although this will require a bit of studying their requests using the browsers developer tools). It's a great tool. [0] https://github.com/puppeteer/puppeteer https://github.com/puppeteer/puppeteer [1] https://pptr.dev https://pptr.dev
MichaelApproved 6y ago
They tweet all their articles, right? No need to scrape their site, if you're only looking for headlines. This tool will generate an RSS feed from their Twitter account http://fetchrss.com/twitter http://fetchrss.com/twitter You won't get the article content but it'll give you enough information to find out if you're interested in the article and then read it on their site. https://twitter.com/Reuters https://twitter.com/Reuters
taborj 6y ago
There are several options; fetchRSS, RSS-Bridge, etc. I personally have RSS-Bridge running on my local webserver, and use it for Twitter feeds, Instagram, Facebook, etc.

Wouldn’t it be possible to create a third-party site scraper that could generate custom RSS feeds independently of the company? About the only barrier would be paywalls, IMO.

lgl 6y ago

Yes, and even with paywalls, as long as the source has login details it can still be done (although this would probably be illegal). Anecdotal evidence: I work in a tour operator and we publish our products on tripadvisor and getyourguide which are two very popular places to advertise these products and that provide an extranet to suppliers where they can login to create and manage products, availability, etc. A few years ago before they had developed APIs or had only private APIs for the "big guys" and wouldn't really pay any attention to small shops' requests, I managed to use Puppeteer [0] [1] (a node.js library to manage headless Chrome) to dynamically update our products into their extranet. It's very powerful and with some tinkering you can automate pretty much any behavior, including authentication and xhr interactions inside their platform (although this will require a bit of studying their requests using the browsers developer tools). It's a great tool. [0] https://github.com/puppeteer/puppeteer https://github.com/puppeteer/puppeteer [1] https://pptr.dev https://pptr.dev

MichaelApproved 6y ago

They tweet all their articles, right? No need to scrape their site, if you're only looking for headlines. This tool will generate an RSS feed from their Twitter account http://fetchrss.com/twitter http://fetchrss.com/twitter You won't get the article content but it'll give you enough information to find out if you're interested in the article and then read it on their site. https://twitter.com/Reuters https://twitter.com/Reuters

taborj 6y ago

There are several options; fetchRSS, RSS-Bridge, etc. I personally have RSS-Bridge running on my local webserver, and use it for Twitter feeds, Instagram, Facebook, etc.