5 ms·
That is me :) 1. Who makes the crawling method? We give you the ability to write your own crawling logic. You don't have to though.. we have a default crawler
by jdrock 17y ago
That is me :)
1. Who makes the crawling method?
We give you the ability to write your own crawling logic. You don't have to though.. we have a default crawler that runs as well.
2. Crawling specific pages?
You can specify pages to crawl using regular expressions or your own custom code.
3. How often and what gets crawled?
Up to you :)
4. How does the data get transferred over?
It's better if you don't transfer all the data over. You can push in your own compute functions to process the data you crawl, and just return much smaller result data sets.
5. What's provided as crawl data?
You can specify the results you return in your code. It's up to you.
- Keyframe 17y agoapropos 4. - those processed data sets could get huge too, that is why I was asking. If crawling logic can be at clients control, but there is a provided crawler that runs well too, and those prices, it sounds like an excellent product you've got there :) Let's just go over this tineye example we have here - they need to crawl web and retrieve images - so we have "Only pay $2 per million pages crawled" where they would pay $2 per million pages crawled, but what would they pay for retrieving images then? I see so much potential in this service, now that is one hell of a product there at one hell of a price - congratulations on it, I hope you do well with it!
- jdrock 17y agoFor 4: We will be pricing data transfer at about $0.10 per GB. Any computation done on the images would be priced at $0.03 per CPU-hour.