4 ms·

I realize the author wasn't proposing that something like this could be a business, but humor me: Had this idea a few years back with a business model that pai

by henryl 18y ago

I realize the author wasn't proposing that something like this could be a business, but humor me: Had this idea a few years back with a business model that paid publishers for cpu cycles gathered from a javascript or flash widget. We hoped to then sell this service to data-intensive industries. Decided it wasn't feasible. We need to consider cpu cycles gained from this regime vs. bandwidth and cpu cycles lost from the hundreds of web, queue, data servers needed to run this model. IMO it is unlikely that this model will pay off once you consider things like network latency, and trade-offs like job size (higher job size is better) vs. job completion probability (lower job size is better). Even if the potential for viability were there, it isn't clear that there is a market for something like this. Large scale computing challenges obviously exist, and a lot of people are making money with solutions like cloud computing, but these problems typically involve proprietary data sets, using proprietary or industry standard (good 'ol apps like MySQL) software. Chopping up your sensitive data and sending it en masse to the public to be processed on javascript instead of C++ doesn't exactly fit client needs.

igrigorik 18y ago
Yep, great points. The HTTP workflow definitely adds a lot of overhead, but I think it is still usable for a whole class of aplications. For example, anytime you can saturate the CPU, or outgoing bandwidth (use clients machine as a spider), this works really well. As far as centralized servers & data storage.. I didn't cover this in the post, but I've been thinking a lot about using bittorrent to address this, and I think it's totally feasible. All you need is several thousand seed servers and you'll have a worldwide file system / job tracking queue. ;)
mchadwick 18y ago
I had actually tried to do this exact thing. Problem A is that you can't execute very much in a clients browser at any one time. Okay, so you make the jobs smaller and fetch more. Problem B is that it turns out that by the time you've pulled the data from disk, shipped it to the browser and back, put it back to disk, and done the same cycle for the reduce, it ends up being cheaper to stream 64M HDFS blocks around EC2. That doesn't even get in to verifying results from an untrusted client. Here's my half-implemented proof of concept from a while back that runs on AppEngine: http://github.com/markchadwick/emarer/tree/master http://github.com/markchadwick/emarer/tree/master Slightly different implementation, but the same idea (which I think is a very cool idea!).
- henryl 18y ago
  You can extend the amount of processing time available on the client by storing intermediate results in window.name (which gives you up to 2MB of semi persistent storage)

by henryl 18y ago

igrigorik 18y ago

Yep, great points. The HTTP workflow definitely adds a lot of overhead, but I think it is still usable for a whole class of aplications. For example, anytime you can saturate the CPU, or outgoing bandwidth (use clients machine as a spider), this works really well. As far as centralized servers & data storage.. I didn't cover this in the post, but I've been thinking a lot about using bittorrent to address this, and I think it's totally feasible. All you need is several thousand seed servers and you'll have a worldwide file system / job tracking queue. ;)

mchadwick 18y ago

I had actually tried to do this exact thing. Problem A is that you can't execute very much in a clients browser at any one time. Okay, so you make the jobs smaller and fetch more. Problem B is that it turns out that by the time you've pulled the data from disk, shipped it to the browser and back, put it back to disk, and done the same cycle for the reduce, it ends up being cheaper to stream 64M HDFS blocks around EC2. That doesn't even get in to verifying results from an untrusted client. Here's my half-implemented proof of concept from a while back that runs on AppEngine: http://github.com/markchadwick/emarer/tree/master http://github.com/markchadwick/emarer/tree/master Slightly different implementation, but the same idea (which I think is a very cool idea!).

henryl 18y ago
You can extend the amount of processing time available on the client by storing intermediate results in window.name (which gives you up to 2MB of semi persistent storage)