6 ms·
I have looked at a number of web scraping products and my largest client uses one of the established products pretty extensively. Your front page is very focus
by mswen 12y ago
I have looked at a number of web scraping products and my largest client uses one of the established products pretty extensively. Your front page is very focused on using site searches as the starting point. Are there more full blown features available?
I note that the pricing page mentions an API but doesn't give much of a sense of what features are available programmatically.
- logn 12y agoYes, the focus (at this point) is on finding a search box on the target site and executing a search there. The first version of the API is still being developed but would be a pretty similar mapping of the current feature set. Notable power of the API: paging indefinitely through the search results, searching multiple sites at once, clustering of multiple instances on VPSs, queueing of searches to pop the results back later, POST and GET requests (so you could bookmark searches). Edit: for the API, also considering a feature to disable AJAX support for scraping per each request. Much time is spent in DOM parsing and timeouts for unknown async requests on target sites.