3 ms·
You know, I'd try this, but the hard part is that the processing is still on me (and I can't code, so I'm SOL). That, frankly, is the hard part, is it not?
by sachinag 17y ago
You know, I'd try this, but the hard part is that the processing is still on me (and I can't code, so I'm SOL). That, frankly, is the hard part, is it not?
- tdavis 17y agoYou'd be surprised how difficult it is to create an extremely fast, scalable architecture for crawling web sites. Took me a few months, anyway (and would probably take a few more to make it "fill in a web form and go" usable). After that, the processing is pretty straight forward. You've got a URL, headers, and body content. Just extract what you want and crunch.
- jennwilson 17y agoHi, I work for 80legs and wanted to address this--we will be building an "App Store" over the next couple of months that will allow you to use Apps written by other developers to do some pretty cool stuff. We already have several of these 80Apps (as we call them) available, many of which have been written by the semantic search engine company swingly.com. You can view descriptions for them on the "Create Job" page if you sign up to use our Portal.
- jdrock 17y agoWhile complex processing can be difficult, having a scalable infrastructure is also unbelievably hard. Sure, you can easily crawl from a single IP address, but if you want to do millions of pages in minutes, you would typically need several servers, IP addresses, and much more.