4 ms·
This seems like the hard part. Does it need accounts with all the services, use a VPN to appear to be in each country, use a fake/rotating User Agent, etc
by ape4 5y ago
This seems like the hard part. Does it need accounts with all the services, use a VPN to appear to be in each country, use a fake/rotating User Agent, etc
- Saepirist 5y agoDepends on the service any of these might be needed.
- toomanydoubts 5y agoWhat an uninteresting answer. People are interested in the technichal details, come on.
- Saepirist 5y agoHaha, sorry. But actually it's pretty much it. You need VPNs to check catalogs in different countries, which is tricky for some services (like Netflix) as they try to block VPNs. So you gotta always have a large pool. Catalogs of services like Prime Video is just there without an account but for Netflix you need accounts to see the catalogs (and not to send too many requests otherwise they block you). Rest of it is just writing some regular crawlers.
- toomanydoubts 5y agoCool, thanks for expanding and congrats on the project!
- chrismeller 5y agoI too would love more technical details on the harvesting process. Something I’ve definitely thought about doing in the past but wasn’t patient enough to implement.