3 ms·
My attempt at a more friendly description of S4 (based on my very limited understanding of it - please correct me if I'm wrong): When processing large amounts
by olalonde 16y ago
My attempt at a more friendly description of S4 (based on my very limited understanding of it - please correct me if I'm wrong):
When processing large amounts of streaming data, you have to process the data as fast as it comes in or else you can't keep up. You probably wonder: "Why not simply delay the processing of new data while the old data is being processed?". The problem lies in the fact we're dealing with a stream which always brings in new data. Eventually, the virtual line up of delayed data will occupy all available memory.
A solution to this is to dispatch the data between multiple computers which can each independently process the data they receive and then send back the result of their processing to a central computer whose job is to put back the results together. Can't keep up with the stream? Simply add a new computer! That's more or less what S3 does.
- scott_s 16y agoGood, except stream processing does not imply that there needs to be a master node - it can truly be distributed. I don't know if S4 is set up like this, though.
- anandkesari 16y agoIt does. Processing is distributed across multiple nodes. See http://wiki.s4.io/Manual/S4Overview http://wiki.s4.io/Manual/S4Overview
- ronnoch 16y agoThank you for this. Yahoo! should copy and paste your comment onto the S4 homepage.
- bluesmoon 16y agoit's opensource, anyone could do that, not just Yahoo!
- eru 16y agoAre you confusing open source with a wiki editable by everyone? Open source just means that you can fork it and do whatever you want with your own version, e.g. fix it. The original owner's don't have to accept your fix.
- martinkallstrom 16y agoWhich, incidentally, often is true for wikis as well.