3 ms·
What would be interesting would be to let users hook their own custom functions to your web crawler. In other words, whenever you crawl a link, you feed the lin
by olalonde 17y ago
What would be interesting would be to let users hook their own custom functions to your web crawler. In other words, whenever you crawl a link, you feed the link to the developer's script which does whatever it wants with it. I've had plenty of ideas that would involve a web crawler, but I just don't have the time to customize an open source crawler. There are too many complications like not falling into infinite loops (such as a calendar type pages).
- coderdude 17y agoI have something in mind along those lines, and I'm going to provide at least two or three ways to create custom datasets from my 'snapshot of the Web.' I thought about your suggestion of allowing customers to execute their own scripts over the snapshot but the security issues involved are my big concern. I would need a way to sandbox it. My plan right now is to offer the ability to run simple data tasks on the snapshot. For example, regex (filtered for things like SSN matchers, and so on), a language with a limited vocabulary for grabbing features with certain textual of visual characteristics, or even something as simple as xpath. I would run these tasks on the cluster and the result would be returned to the customer.
- jdrock 17y agoIf you're interested in doing this, just use 80legs, which lets you provide your own applications to customize web content processing (http://80legs.pbworks.com/80apps http://80legs.pbworks.com/80apps).