3 ms·
The crawler uses PhantomJS, but you can also create jobs that use a plain HTML downloader or even cheerio. The platform is very flexible.
by jancurn 9y ago
The crawler uses PhantomJS, but you can also create jobs that use a plain HTML downloader or even cheerio. The platform is very flexible.
- dannyhw 9y agoVery cool. I'm looking at it more in depth and I'm pretty impressed. This is basically all things I've been doing for years, but encapsulated in a way that I'm pretty sure end users will be able to use. I've always built these beasts and tried to evangelize for a MEAN stack as the best option but people haven't been ready. I have never met a problem I felt promises were a good solution for, but you've done a really great job of using them to let people who can't think asynchronously get things done. Pseudo URLs are also something I've done before but are never the optimal solution. Again though, this is something that will allow a total beginner to catch up to me about 75%, and I've been developing for 25 years.