6 ms·
Crawling the web at $2 per million pages
- ujjwalg 17y agoThis seems to be a very neat idea. How about processing of the crawled data, if someone wants to process the data and show it to the end user in a comprehensive way? We are looking for a solution which is affordable and provide both.
- jhammerb 17y agoStuff the results of the crawl into Hadoop, write some MapReduce jobs to process the data, and then serve the structured data back up with HBase?
- sachinag 17y agoYou know, I'd try this, but the hard part is that the processing is still on me (and I can't code, so I'm SOL). That, frankly, is the hard part, is it not?
- tdavis 17y agoYou'd be surprised how difficult it is to create an extremely fast, scalable architecture for crawling web sites. Took me a few months, anyway (and would probably take a few more to make it "fill in a web form and go" usable). After that, the processing is pretty straight forward. You've got a URL, headers, and body content. Just extract what you want and crunch.
- jennwilson 17y agoHi, I work for 80legs and wanted to address this--we will be building an "App Store" over the next couple of months that will allow you to use Apps written by other developers to do some pretty cool stuff. We already have several of these 80Apps (as we call them) available, many of which have been written by the semantic search engine company swingly.com. You can view descriptions for them on the "Create Job" page if you sign up to use our Portal.
- jdrock 17y agoWhile complex processing can be difficult, having a scalable infrastructure is also unbelievably hard. Sure, you can easily crawl from a single IP address, but if you want to do millions of pages in minutes, you would typically need several servers, IP addresses, and much more.
- kbrower 17y agoI would love it if it processed, stored, and indexed the data. http://parselets.com/ http://parselets.com/ + mysql + sphinx
- lsb 17y agoTry Yahoo Pipes for things like that?
- qeorge 17y agoWow - parselets.com is amazing. Thanks for sharing. I think 80legs can do some level of parsing now, but Parsely seems like a great way to describe the analysis for targeted crawls on predictable data. If it supported the combination you've described it would really open up some cool possibilities.
- jdrock 17y agoWe are looking at storing data, doing pre-done crawls, offering data feeds, etc. It's something on the horizon for us. Hopefully guys that write parselets will be interested in becoming developers for the upcoming 80legs App Store. They'll be able to sell their parselets and earn 100% recurring revenue from 80legs users using their parselets.
- andreyf 17y agoI remember they had an explanation about this, but I don't remember what it was, and it seems to have disappeared: what are these 50,000 computers they're using?
- e1ven 17y agoPlura has a java applet which you can embed into a webpage that gets viewed for a long duration, such as a game or a streaming video. Affiliates embed an iframe which loads their applet as the user plays their game. The user's CPU goes up a bit, and they can help generate revenue for the game makers.
- lsb 17y agoThey're targeting desktop apps. The Java app downloads the pages, so it needs high permissions, so you get the default Java unfriendly popup asking you for confirmation.
- e1ven 17y agoFrom http://pluraprocessing.com/developer/index.html http://pluraprocessing.com/developer/index.html Plura supports desktop and web-based games. If the game is hosted on a website, like a Flash game, the developer only needs to include 1 line of iframe code. We will soon be releasing a Javascript API for dynamically controlling how this iframe is loaded, giving the developer control over starting, stopping, and controlling CPU usage in Plura. The iframe loads a Java web applet, which runs completely in memory. This applet is forcibly restricted from accessing the user's computer by the sandbox model provided by Sun. See also - http://pluraprocessing.com/developer/index.html http://pluraprocessing.com/developer/index.html So far as I understand, they have multiple models. One affiliate model is "Plura for Java Applets", where-as another is "Signed Java Applets" I imagine there may be fewer options for unsigned applets, leaving the developer with less potential revenue every month, where as desktop application developers and signed java applications are left with the providers who don't need signed applications. That said, I agree the Java dialog is ugly and scary ;( If I were an Affiliate, I'd want to avoid it. It breaks the user-experience of your site into some gaudy and jarring, not to mention unbranded and unrelated to the information the user is after.
- teeja 17y agoPricey! I'd pimp for Open Google instead.
- Flemlord 17y agoI wonder how they arrived at their $2 per million pricing model. Why not $10 per million? Either one sounds ridiculously inexpensive.
- charlesju 17y agoThey're probably using EC2s and then it's a function of how many pages can be scrapped per EC2 compute hour and then taking a premium on top of that. The reason why they didn't want it to be too expensive (ie. 5 times as expensive) is (1) competitors can easily equilibrate and steal market share if their idea works b/c of the economic inefficiencies in their pricing model and (2) this game plan is more a game of dependence rather than up front profits, so it makes sense to take very little profit up front to get user traction.
- jdrock 17y agoWe're not using EC2. AWS doesn't and can't scale to our size, actually :) Side note: AWS guys have asked if we use them. We said "No, we'd be losing money then and wouldn't be able to scale to our size with you."
- fsniper 17y agoThat reminds me about my fools day joke about MassiveClouding. I imagined about a company buying cpu/hours from regular desktop computer users and selling it to anyone interested in this kind of cpu power. Of course any computer software could be run on the MassiveClouding without any changes. My good imagination :) I don't know about the technology behind 80Legs but these 50000 computers might be a bot net.
- Evgeny 17y agoIsn't that like what the SETI project does? (Except that they don't buy the cpu power, they ask people to volunteer) http://en.wikipedia.org/wiki/SETI#SETI.40home http://en.wikipedia.org/wiki/SETI#SETI.40home "Any individual can become involved with SETI research by downloading the Berkeley Open Infrastructure for Network Computing (BOINC) software program, attaching to the SETI@home project, and allowing the program to run as a background process that uses idle computer power."
- fsniper 17y agoYes it's something like it. There is another base difference; SETI@home or other folding projects have a centralized infrastructure for the data to be collected and bound. And all the software to be run on the client side are written for a specific mission. So every software should be rewritten for these platforms. My MassiveClouding dream has a some way of dissecting any software into parts and run on the clients. That's why it's a dream.
- skolor 17y agoThere may be a market for this. I know several people who run various types of folding just to be able to brag about the performance they get, and love to show off their stats. Getting paid for it would just be icing on the cake. Of course, the pricing structure would be difficult, as would setting up the infrastructure for it. My (uniformed, by-the-seat-of-my-pants) best guess would be that subverting the current folding method of jobs would be the best way to do it. Accept jobs, send out to client computers, clients get paid based off of how many they complete. The problem arises from: sandboxing the jobs, to make sure they can't steal client info, and making it relatively easy to program jobs, while still allowing users to do various things to increase their performance (like using GPUs and the like).
- keefe 17y agoThey're inflating the cost difference. Cloud : $0.10 / CPU-hour for "large scale" crawling. There's no reason to use a small, unreserved EC2 instance for large scale anything? I have a small reserved instance I use for git and bugzilla and so forth, it's slow as hell. That's why there are bigger ones.
- jdrock 17y agoThe cost savings are still there with reserved instances if you do the math. Actually, the cost is not the biggest issue with the cloud. If you're talking about large-scale crawling, AWS will not adequately scale. You can't get enough nodes or enough bandwidth.
- keefe 17y agoMy point was just in questioning a $4/million figure using a 0.10/hr instance. Why don't you run benchmarks against a reserved high CPU instance for 100M pages crawled? I was never contending that you cannot offer cost savings over custom crawling in ec2, just the way the #s on the page were calculated. If you have enough traffic to keep your instances saturated of course it is more economical to buy dedicated servers, but nobody starts out that way. Why in the world do you think you can't get enough nodes or bandwidth to do web crawling off of EC2? I remember seeing Bezos talk about animoto scaling up to 3500 instances instantly for video transcoding, I find it extremely hard to believe such a parallelizable task as web crawling could not be done on EC2. It's one thing to say that you can do it cheaper, it's another thing entirely to say that it can't be done on EC2 at all. How did you come to this conclusion?
- jdrock 17y agoThe $4/million actually is not based on compute time, it's based on data transfer in/out of AWS. If I included compute time, it would be higher. And it's the bandwidth aspect that makes web crawling not feasible on AWS. Yes, you have a few thousand nodes, but they're all going through a handful of external IPs, which will cause serious performance issues.. the worst case is that you'll get blocked entirely from the sites you're trying to crawl. In other words, the bandwidth is not parallelizable on the cloud.
- anurag 17y agoI tried 80legs a couple of weeks ago when it was still in beta. It works well if you're willing to write Java code to process HTTP responses (see http://80legs.pbworks.com/80Apps http://80legs.pbworks.com/80Apps). You can issue new requests in your processing code and write processed data to files which you can download once the crawl finishes. Even without Java, you can perform basic keyword/regex matching on crawled data literally by filling out a web form. And they have an API of course. Very impressive. However, if you want to run crawling on your own infrastructure, I recommend Scrapy (http://scrapy.org/ http://scrapy.org/), a python crawling framework introduced on HN last year. Scrapy solves some of the more time-consuming problems involved in writing a crawler from scratch (multiple simultaneous requests, pipelined processing, raw caching, duplicate URL filtering) and comes with nifty development and administration tools. More importantly, it has an active and helpful set of core developers and good documentation. I am comfortable with both Python and Java but I chose Scrapy over 80legs because I can crawl for free on the machines I already have and I can afford to spend more time crawling from a single IP compared to 80legs which will let me crawl much faster but isn't free. Also, with Scrapy my bot can be 'naughty' - 80legs jobs obey robots.txt and limit the crawling rate per domain. When my crawling needs outgrow my infrastructure, I am going to look at 80legs again.
- jdrock 17y agoJust a quick heads up - we'll be looking into allowing JRuby and Jython fairly soon for when you create 80Apps. You'll also be able to mashup third-party apps and libraries into your own code. This is something we hope to have in 2-3 months.
- coderdude 17y agoHow come there isn't a REST API? Java, JRuby, Jython. Ewwww. This isn't how outfits are doing things anymore...
- jdrock 17y agoThe crawl runs on a super-heterogenous network of computers from around the world. The JVM sandbox is the only thing we know. Java is the only language that has something like this. Yes, Java isn't as sexy anymore. But running your code on 50,000 computers sounds sexier to me than a REST API.
- mwexler 17y agoOne limitation is the throttling for each domain. If I had a smaller set of pages that I wanted to read frequently, this solution would not work. I understand the need to avoid DOS attacks, but in some cases, it would nice to be able to read a million pages from 3 large domains instead of 1 from a million pages.
- bliving 17y agoThat and... "Your parseLinks() and processDocument() methods must complete within a total 10 seconds per document processed" ... as a limit on processing leaves room for competition. One advantage of the BYO-Cloud solution is that you can pay for more intensive processing of the crawl.
- tezza 17y ago..:: Loop Detection?? ::.. Are you supposed to implement your own cycle detection? If not, how deep is the cycle detection that 80legs offers? -- There are plenty of HoneyTrap[1] OSS projects which will quickly rack up lots of $$$ if the 80legs spider is backlisted. For those who don't know, these projects create deeply linked pages, and sometimes create infinite cycles. They are trying to hinder spammers, but may hinder 80legs too. -- [1] http://www.davidnaylor.co.uk/stopping-bad-robots-with-honeytraps.html http://www.davidnaylor.co.uk/stopping-bad-robots-with-honeyt...
- jdrock 17y agoBecause of the way 80legs handles crawls, users don't need to worry about loop detection. There are really two issues here: 1. For each user crawl, we only allow the same url once, so any simple loops that involve the same urls are eliminated by this process no matter how large the loop 2. For more sophisticated "spider traps" that work with different urls and domains, they can have a limited effect on your crawls. Because of our per-domain rate throttling, the worst these traps can do is add a few cents per day to someone's crawl.
- yarapavan 17y agoMIT Technology Review also has an article on 80legs. Accessible at http://www.technologyreview.com/web/23528/ http://www.technologyreview.com/web/23528/