3 ms·
I'm worried that the websites I'm scraping from will be updated and will no longer work When I worked for Internet Brands, they bought a site called Tjoos (a c
by byoung2 15y ago
I'm worried that the websites I'm scraping from will be updated and will no longer work
When I worked for Internet Brands, they bought a site called Tjoos (a coupon/deal site) that amassed what was at the time the largest number of merchant sites using a proprietary screen scraping technology. It is a very fragile method, and a good portion of the codebase was dedicated to config classes that needed to be updated with each change to a given site. The founders had to keep hiring developers to keep up with the demands of constantly updating scraping definitions, so the site had a lot of revenue, but little profit when IB bought it. Surely by now Internet Brands has updated the site to use the same engine that powers UlitmateCoupons, BensBargains, DealLocker, etc.
- fatalerrorx3 15y agoThe newer coupon/deal aggregators are most likely using RSS feeds which is a more standardized format when compared to screen scraping, and thus less chance of breaking. There are still a surprisingly large amount of websites that are still stuck in the 90s in terms of data portability though