4 ms·
Nice timing, as just this week I had my first exposure to the Cloudscrape platform. Credit where credit is due, I found the platform refreshingly feature rich e
by zer0defex 11y ago
Nice timing, as just this week I had my first exposure to the Cloudscrape platform. Credit where credit is due, I found the platform refreshingly feature rich even at lower tier (i.e. free) account levels compared to other offerings from competitors.
Downsides were like so many other crawlers/spiders - too much in trying to meet the needs of all. Aiming for the 95% is easy, 95 to 99% though, not so much. Hence all the corpses littered along this path.
Recommendations:
Define solid, real-world examples of your platform in practice and really, no bullshit, what makes you different from every other platform that's 97% polished out there?
You didn't meet my needs because I needed much more in the way of parsing network requests from pages, mostly in handling if/then scenarios. A relatively simple scenario, to ME, and there-in lies the challenge of the crawler market, it's the last 5% that's custom to each scenario that so often delivers 98& of the value.
Suggestion: Identify and use the talent you obviously have to destroy some high value markets. Hint: Prioritize targets that still champion a desktop app at any point in the top 5 in terms of data collection.
Outside of that and far more niche (and subsequently far lower priority, allow for parsing / handling handling network requests and responses via custom app/API instead of just block all, some, or none)
- henrik1409 11y agoThank you for the recommandations! Would love to know more about your exact use cases - we're aiming for the 100% :) If you'd reach out on support@cloudscrape.com - we'd be happy to investigate if we can't make CloudScrape work for you as well.