We're operating a quite large, custom built scraper - this, together with extensive monitoring, continuous engineering effort and topped with some manual checki

We're operating a quite large, custom built scraper - this, together with extensive monitoring, continuous engineering effort and topped with some manual checking of the most popular content & providers keeps our users happy. It's a pretty large topic actually and we're planning to blog about it - but for some first insights on our technology, see these two links: https://www.justwatch.com/blog/post/debugging-postgresql-performance-the-hard-way/ https://www.justwatch.com/blog/post/debugging-postgresql-per... https://prometheus.io/blog/2016/10/12/interview-with-justwatch/ https://prometheus.io/blog/2016/10/12/interview-with-justwat...