6 ms·
Thank you! The data is updated daily by automated tools that checks the services regularly.
by Saepirist 5y ago
Thank you! The data is updated daily by automated tools that checks the services regularly.
- ape4 5y agoThis seems like the hard part. Does it need accounts with all the services, use a VPN to appear to be in each country, use a fake/rotating User Agent, etc
- Saepirist 5y agoDepends on the service any of these might be needed.
- toomanydoubts 5y agoWhat an uninteresting answer. People are interested in the technichal details, come on.
- Saepirist 5y agoHaha, sorry. But actually it's pretty much it. You need VPNs to check catalogs in different countries, which is tricky for some services (like Netflix) as they try to block VPNs. So you gotta always have a large pool. Catalogs of services like Prime Video is just there without an account but for Netflix you need accounts to see the catalogs (and not to send too many requests otherwise they block you). Rest of it is just writing some regular crawlers.
- toomanydoubts 5y agoCool, thanks for expanding and congrats on the project!
- chrismeller 5y agoI too would love more technical details on the harvesting process. Something I’ve definitely thought about doing in the past but wasn’t patient enough to implement.