5 ms·

S4 is a general-purpose, distributed, scalable, partially fault-tolerant, pluggable platform that allows programmers to easily develop applications for processi

by sabat 16y ago

S4 is a general-purpose, distributed, scalable, partially fault-tolerant, pluggable platform that allows programmers to easily develop applications for processing continuous unbounded streams of data. I'm sure this is cool and useful technology. At this moment, from the marketing-speak, I have no idea what it does except that it has something to do with volumes of streaming data. Whose data? Is it a service? (Maybe not, since you can download it?) What could it do for me (in simple terms)? What's a basic use case? Why do people assume that we can mind-read?

olalonde 16y ago
Up voted the story just so more people read your comment. This kind of "marketing-speak" focused on technical details is well too widespread. The project may be technical by nature but there's got to be a higher level way of describing it.
- sabat 16y ago
  It's frustrating, isn't it? S4 sounds like it does something cool and useful, and I'd like to know what this is, just for reference at least. But I don't have time to go and figure it out on my own; I'm too busy trying to stay productive.
  - sanswork 16y ago
    If you're too busy to spend the time to read the docs to figure out if its useful for yourself. How will you ever find the time to learn the system and implement it into your work processes?
    - bluesnowmonkey 16y ago
      Do you read the docs of every new software product that gets released, in order to see if it's useful to you? No, there aren't enough hours in the day.
- th0ma5 16y ago
  could just be we're not the audience yet, they see this as something they want to break out within businesses, so they have to communicate it to business types that want to know practically and conceptually how this new thing is better than what they have, but don't care about the actual code.
- yangyang 16y ago
  Surely it's aimed at technical people, able to understand technical language?
- mzl 16y ago
  It's not marketing speak. It is a clear, simple, and short sentence that describes exactly what S4 is. It doesn't have irrelevant details ("we use technology XYZ for distributing our nodes"), nor very much marketing describing the value ("S4 solves all your problems!!!", although they do use the word "easily" once). If the terms used are not clear to someone reading it, they are probably not the right person to implement a good distributed system anyway.
- rjurney 16y ago
  Maybe you should cultivate some fucking respect. Yahoo, which gave us such critical technologies as Hadoop and Pig, just gave us what appears to be a robust distributed stream processing thingamabob. They open sourced it first, and will quickly bring the docs and presentation up to speed. Instead of mouthing off, just say, 'thanks!' or nothing at all.
noodle 16y ago
looks like it lets you take a large, continuously updating stream of data and distribute the processing of it across multiple nodes. kinda like mapreduce.
samratjp 16y ago
"At Yahoo! Labs we design algorithms that are primarily driven by large scale applications for data mining and machine learning in a production environment. We show that the S4 design is surprisingly flexible and lends itself to run in large clusters built with commodity hardware." Via http://labs.yahoo.com/event/99 http://labs.yahoo.com/event/99 I could've sworn there was a blurb there about where they are using and I recall using "real time map-reduce jobs" for things live bidding on ADs and such; another use case would be with stock market data. You're right that it's marketing speak, this project has gotten too much attention on HN in the last few days even when all the git repo had was initial commit. It's too bad they don't have proper explanation, maybe it's because they weren't probably expecting all this attention yet.
- LiveTheDream 16y ago
  Lets be clear. S4 is not real-time MapReduce, its a stream processing system.[1] [1] http://twitter.com/s4project/status/29611855285 http://twitter.com/s4project/status/29611855285
jamii 16y ago
As I understand it the processing model is similar to Yahoo pipes or good old unix pipes, except with easy support for parallel processing, distribution and fault tolerance.
- pyrhho 16y ago
  So... It's like Unix pipes with GNU Parallel?
igrekel 16y ago
Just looked a little at the documentation. It seems like its an engine for stream processing. Think of it of data mining up front. You figure out what information you want and collect it as the data comes in instead of storing all the data and mining for what you want from the accumulated pile.
allertonm 16y ago
One way of looking at it is that S4 is to map-reduce what Streambase or Coral8 are to SQL. In the enterprise software world, this is what's called CEP - Complex Event Processing: http://en.wikipedia.org/wiki/Complex_event_processing http://en.wikipedia.org/wiki/Complex_event_processing
- chrisaycock 16y ago
  Yeah, I had gotten the impression it was distributed CEP.
scott_s 16y ago
It's not marketing speak, it's research speak. I have worked on a similar project (and will work on it again in the future), and I know exactly what they mean by those things. General purpose: in the same way C is a "general purpose" language. It can handle arbitrary problems. Distributed: designed to be used across multiple compute nodes. Scalable: they've made the effort to ensure that performances increases as they increase the number of compute nodes. Partially fault-tolerant: node failure does not mean the results of the computation are lost. "Partially," I assume, implies they can't guarantee this completely. Continuous unbound streams of data: think sensors that are constantly sending more data. Or a stock market ticker. Or radio telescopes constantly monitoring the sky. Or a medical patient's various monitors. The reason these terms don't resonate with you is that these type of applications - this type of programming - is something you're not familiar with.
- devmonk 16y ago
  No, I think what sabat is saying is that the front page documentation is geared towards the wrong audience and needs to have less general information on usage. I think if the following actually had examples of use, I'd try it out: http://wiki.s4.io/Cookbook/Cookbook http://wiki.s4.io/Cookbook/Cookbook Currently it only has: UsingEclipse 1. Check out sources from git 2. Create Eclipse configurations: mvn eclipse:eclipse 3. Import project into Eclipse 4. set variable M2_REPO to the local Maven repository: e.g. ~/.m2/repository 5. Set up formatting: * Spaces for indentation * Tab width = 4 I completely understand that this isn't there yet, though. It is a new project.
  - bluesmoon 16y ago
    it's opensource, right? so anyone could go in and fix any problems they see, right?
  - anishnair 16y ago
    Please start at http://wiki.s4.io/Tutorials/GettingStarted http://wiki.s4.io/Tutorials/GettingStarted
LiveTheDream 16y ago
> Whose data? Anyone's data that you can stuff into the system: "The drivers to read from and write to the platform can be implemented in any language making it possible to integrate with legacy data sources and systems." > Is it a service? No, it's a platform. You could turn it into a Platform-as-a-Service, like Amazon does with various technologies. "S4 is a ... platform" > What could it do for me (in simple terms)? What's a basic use case? My first thought would be real-time trending calculations. You have a massive, never-ending stream of data...how do you extract real-time insights from that? > Why do people assume that we can mind-read? Perhaps because after being immersed in a project for a long time, it's easy to forget what is obvious to you, but non-obvious to others.
olalonde 16y ago
My attempt at a more friendly description of S4 (based on my very limited understanding of it - please correct me if I'm wrong): When processing large amounts of streaming data, you have to process the data as fast as it comes in or else you can't keep up. You probably wonder: "Why not simply delay the processing of new data while the old data is being processed?". The problem lies in the fact we're dealing with a stream which always brings in new data. Eventually, the virtual line up of delayed data will occupy all available memory. A solution to this is to dispatch the data between multiple computers which can each independently process the data they receive and then send back the result of their processing to a central computer whose job is to put back the results together. Can't keep up with the stream? Simply add a new computer! That's more or less what S3 does.
- scott_s 16y ago
  Good, except stream processing does not imply that there needs to be a master node - it can truly be distributed. I don't know if S4 is set up like this, though.
  - anandkesari 16y ago
    It does. Processing is distributed across multiple nodes. See http://wiki.s4.io/Manual/S4Overview http://wiki.s4.io/Manual/S4Overview
- ronnoch 16y ago
  Thank you for this. Yahoo! should copy and paste your comment onto the S4 homepage.
  - bluesmoon 16y ago
    it's opensource, anyone could do that, not just Yahoo!
    - eru 16y ago
      Are you confusing open source with a wiki editable by everyone? Open source just means that you can fork it and do whatever you want with your own version, e.g. fix it. The original owner's don't have to accept your fix.
earl 16y ago
If this were relevant to you, you'd know what most of that means, particularly stream processing. Why exactly is yahoo faulted for your lack of understanding of the technical vocabulary? I'm mostly annoyed that you think that using the usual vocabulary to discuss a problem means that people have to mind-read.

by sabat 16y ago

olalonde 16y ago

Up voted the story just so more people read your comment. This kind of "marketing-speak" focused on technical details is well too widespread. The project may be technical by nature but there's got to be a higher level way of describing it.

sabat 16y ago
It's frustrating, isn't it? S4 sounds like it does something cool and useful, and I'd like to know what this is, just for reference at least. But I don't have time to go and figure it out on my own; I'm too busy trying to stay productive.
- sanswork 16y ago
  If you're too busy to spend the time to read the docs to figure out if its useful for yourself. How will you ever find the time to learn the system and implement it into your work processes?
  - bluesnowmonkey 16y ago
    Do you read the docs of every new software product that gets released, in order to see if it's useful to you? No, there aren't enough hours in the day.
th0ma5 16y ago
could just be we're not the audience yet, they see this as something they want to break out within businesses, so they have to communicate it to business types that want to know practically and conceptually how this new thing is better than what they have, but don't care about the actual code.
yangyang 16y ago
Surely it's aimed at technical people, able to understand technical language?
mzl 16y ago
It's not marketing speak. It is a clear, simple, and short sentence that describes exactly what S4 is. It doesn't have irrelevant details ("we use technology XYZ for distributing our nodes"), nor very much marketing describing the value ("S4 solves all your problems!!!", although they do use the word "easily" once). If the terms used are not clear to someone reading it, they are probably not the right person to implement a good distributed system anyway.
rjurney 16y ago
Maybe you should cultivate some fucking respect. Yahoo, which gave us such critical technologies as Hadoop and Pig, just gave us what appears to be a robust distributed stream processing thingamabob. They open sourced it first, and will quickly bring the docs and presentation up to speed. Instead of mouthing off, just say, 'thanks!' or nothing at all.

noodle 16y ago

looks like it lets you take a large, continuously updating stream of data and distribute the processing of it across multiple nodes. kinda like mapreduce.

samratjp 16y ago

"At Yahoo! Labs we design algorithms that are primarily driven by large scale applications for data mining and machine learning in a production environment. We show that the S4 design is surprisingly flexible and lends itself to run in large clusters built with commodity hardware." Via http://labs.yahoo.com/event/99 http://labs.yahoo.com/event/99 I could've sworn there was a blurb there about where they are using and I recall using "real time map-reduce jobs" for things live bidding on ADs and such; another use case would be with stock market data. You're right that it's marketing speak, this project has gotten too much attention on HN in the last few days even when all the git repo had was initial commit. It's too bad they don't have proper explanation, maybe it's because they weren't probably expecting all this attention yet.

LiveTheDream 16y ago
Lets be clear. S4 is not real-time MapReduce, its a stream processing system.[1] [1] http://twitter.com/s4project/status/29611855285 http://twitter.com/s4project/status/29611855285

jamii 16y ago

As I understand it the processing model is similar to Yahoo pipes or good old unix pipes, except with easy support for parallel processing, distribution and fault tolerance.

pyrhho 16y ago
So... It's like Unix pipes with GNU Parallel?

igrekel 16y ago

Just looked a little at the documentation. It seems like its an engine for stream processing. Think of it of data mining up front. You figure out what information you want and collect it as the data comes in instead of storing all the data and mining for what you want from the accumulated pile.

allertonm 16y ago

One way of looking at it is that S4 is to map-reduce what Streambase or Coral8 are to SQL. In the enterprise software world, this is what's called CEP - Complex Event Processing: http://en.wikipedia.org/wiki/Complex_event_processing http://en.wikipedia.org/wiki/Complex_event_processing

chrisaycock 16y ago
Yeah, I had gotten the impression it was distributed CEP.

scott_s 16y ago

It's not marketing speak, it's research speak. I have worked on a similar project (and will work on it again in the future), and I know exactly what they mean by those things. General purpose: in the same way C is a "general purpose" language. It can handle arbitrary problems. Distributed: designed to be used across multiple compute nodes. Scalable: they've made the effort to ensure that performances increases as they increase the number of compute nodes. Partially fault-tolerant: node failure does not mean the results of the computation are lost. "Partially," I assume, implies they can't guarantee this completely. Continuous unbound streams of data: think sensors that are constantly sending more data. Or a stock market ticker. Or radio telescopes constantly monitoring the sky. Or a medical patient's various monitors. The reason these terms don't resonate with you is that these type of applications - this type of programming - is something you're not familiar with.

devmonk 16y ago
No, I think what sabat is saying is that the front page documentation is geared towards the wrong audience and needs to have less general information on usage. I think if the following actually had examples of use, I'd try it out: http://wiki.s4.io/Cookbook/Cookbook http://wiki.s4.io/Cookbook/Cookbook Currently it only has: UsingEclipse 1. Check out sources from git 2. Create Eclipse configurations: mvn eclipse:eclipse 3. Import project into Eclipse 4. set variable M2_REPO to the local Maven repository: e.g. ~/.m2/repository 5. Set up formatting: * Spaces for indentation * Tab width = 4 I completely understand that this isn't there yet, though. It is a new project.
- bluesmoon 16y ago
  it's opensource, right? so anyone could go in and fix any problems they see, right?
- anishnair 16y ago
  Please start at http://wiki.s4.io/Tutorials/GettingStarted http://wiki.s4.io/Tutorials/GettingStarted

LiveTheDream 16y ago

> Whose data? Anyone's data that you can stuff into the system: "The drivers to read from and write to the platform can be implemented in any language making it possible to integrate with legacy data sources and systems." > Is it a service? No, it's a platform. You could turn it into a Platform-as-a-Service, like Amazon does with various technologies. "S4 is a ... platform" > What could it do for me (in simple terms)? What's a basic use case? My first thought would be real-time trending calculations. You have a massive, never-ending stream of data...how do you extract real-time insights from that? > Why do people assume that we can mind-read? Perhaps because after being immersed in a project for a long time, it's easy to forget what is obvious to you, but non-obvious to others.

olalonde 16y ago

My attempt at a more friendly description of S4 (based on my very limited understanding of it - please correct me if I'm wrong): When processing large amounts of streaming data, you have to process the data as fast as it comes in or else you can't keep up. You probably wonder: "Why not simply delay the processing of new data while the old data is being processed?". The problem lies in the fact we're dealing with a stream which always brings in new data. Eventually, the virtual line up of delayed data will occupy all available memory. A solution to this is to dispatch the data between multiple computers which can each independently process the data they receive and then send back the result of their processing to a central computer whose job is to put back the results together. Can't keep up with the stream? Simply add a new computer! That's more or less what S3 does.

scott_s 16y ago
Good, except stream processing does not imply that there needs to be a master node - it can truly be distributed. I don't know if S4 is set up like this, though.
- anandkesari 16y ago
  It does. Processing is distributed across multiple nodes. See http://wiki.s4.io/Manual/S4Overview http://wiki.s4.io/Manual/S4Overview
ronnoch 16y ago
Thank you for this. Yahoo! should copy and paste your comment onto the S4 homepage.
- bluesmoon 16y ago
  it's opensource, anyone could do that, not just Yahoo!
  - eru 16y ago
    Are you confusing open source with a wiki editable by everyone? Open source just means that you can fork it and do whatever you want with your own version, e.g. fix it. The original owner's don't have to accept your fix.

earl 16y ago

If this were relevant to you, you'd know what most of that means, particularly stream processing. Why exactly is yahoo faulted for your lack of understanding of the technical vocabulary? I'm mostly annoyed that you think that using the usual vocabulary to discuss a problem means that people have to mind-read.