4 ms·
PiCrawler: A distributed web crawler using PiCloud
- Fomite 13y agoI'm rather fond of PiCloud as a platform - it's good to see example projects out there using it.
- ikuyamada 13y agoI am also a fan of using PiCloud as a platform. Using PiCloud for implementing crawler requires some hacks, so I decided to create this as a separate package.
- brianobush 13y agoWhat are the advantages over Rackspace, Amazon, Digital Ocean? Seems they are targeting science and big data apps.
- spartango 13y agoPiCloud is a layer of abstraction above the IaaS providers you've listed. That is to say that it gives you software infrastructure to design workflows and distribute them across the machines provided by an IaaS provider. In some ways it's like Heroku, and in others it's another layer beyond Heroku (App Engine-like?). You architect your work around the PiCloud tools. I think PiCloud currently runs on AWS.
- Fomite 13y agoFor me? Well, I'm a computational epidemiologist, so targeting science is exactly it's advantage. PiCloud allows the use of AWS resources fairly seamlessly from inside scripts, iPython notebooks, etc. They're also very responsive to questions, and their package is far, far easier to use for the things I use it for than trying to build something myself via AWS, Rackspace, etc. Disclosure: They also gave me a $500 research grant.