4 ms·

I.B.M. Unveils Real-Time Software to Find Trends in Vast Data Sets

10 points by brandonkm 17y ago

jnorthrop 17y ago
This sounds a bit too good to be true, and an awfully good way for IBM to sell it's consulting services. How can a system be capable of analyzing financial data, weather system, medical information, etc. without someone plugging in all of the data sources and explicitly telling the system how to correlate the data -- of course to do that you simply pay for a large helping of consulting... cha-ching! Maybe I'm missing something fundamental (the article was vague) but I just don't see anything really new here. I read it as marketing.
- wmf 17y ago
  To be clear, System S is a framework and programming language for building streaming apps; it doesn't actually know anything about finance, weather, etc. For the technical research underlying it, see http://domino.research.ibm.com/comm/research_projects.nsf/pages/esps.index.html http://domino.research.ibm.com/comm/research_projects.nsf/pa...
dkarl 17y ago
What does "real-time" mean? Oh yeah, it means "faster than our competitors." And "stream" means "the Cell marketing team encouraged us to use the word 'stream' a lot." Factor out those two buzzwords, and you have an article about IBM releasing some software that analyzes data really fast and helps you make decisions based on that data. If newspapers are in such dire financial straits they really need to start charging for stuff like this.
- wmf 17y ago
  I know I shouldn't, but... What does "real-time" mean? It means that data is processed at the same rate that it is produced. (Or as Wikipedia says, "originally it referred to a simulation that proceeded at a rate that matched that of the real process it was simulating.") A side effect of real-time is that RAM can be used for buffering rather than using disk for storage. And "stream" means "the Cell marketing team encouraged us to use the word 'stream' a lot." Wikipedia defines a stream as "a succession of data elements made available over time" and stream processing as "the quasi-continuous flow of data which is processed in a dataflow programming language as soon as the program state meets the starting condition of the stream". I don't think System S even runs on Cell. While "real-time" and "stream" are too frequently used as buzzwords these days, IBM is actually using them correctly in this case.
  - dkarl 17y ago
    "Real-time" is relative to the time scale on which you look at it. Nobody dealing with a daily flow of business data actually falls behind and ends up making decisions on older and older data because processing is backed up. "Real-time" just means "faster;" it's relative to human expectations. If seismic data gathered this morning were processed and available for inspection late this afternoon, that would be "real-time." If an MRI was performed this morning, and the image wasn't available until this afternoon, that would not be "real-time." It's a mooshy term -- it has meaning in a specific domain, relative to human expectations in that domain. In the article, outside any specific domain, it doesn't mean anything. "a succession of data elements made available over time" and stream processing as "the quasi-continuous flow of data which is processed in... Basically automated batch processing where the "chunkiness" lies beneath a certain threshold of perception. While "real-time" and "stream" are too frequently used as buzzwords these days, IBM is actually using them correctly in this case. How can you tell? If you have a better link, with real information, post it!
    - scott_s 17y ago
      http://portal.acm.org/citation.cfm?id=1376616.1376729 http://portal.acm.org/citation.cfm?id=1376616.1376729 http://portal.acm.org/citation.cfm?id=1289612.1289615 http://portal.acm.org/citation.cfm?id=1289612.1289615
      - dkarl 17y ago
        This looks interesting. Now if somebody could explain how I was supposed to realize there was something interesting behind that crappy New York Times article (short of already knowing?) My original post got modded into oblivion, but I stand by it. The article looks like press material provided by IBM and lightly worked over by the Times writer whose name appears on it.
        wmf 17y ago
        I know where you're coming from; I love writing cynical "buzzword alarm" comments. This time you got unlucky and criticized a legitimate project; I wouldn't worry too much about it.
- joshu 17y ago
  real-time is defined as "will process in a deterministic amount of time" in systems. a processor could make a hard commitment to have an answer in 3ms or whatever. this is hard to guarantee on linux or whatever, because you never know when you'll get swapped out or whatever.
smokinn 17y ago
People seem to be confused about what IBM's System S actually is. For a more technical look at their new offering I suggest these two posts: http://www.dbms2.com/2009/05/13/ibm-system-s-infosphere-streams-processing/ http://www.dbms2.com/2009/05/13/ibm-system-s-infosphere-stre... http://www.dbms2.com/2009/05/18/followup-on-ibm-system-sinfosphere-streams/ http://www.dbms2.com/2009/05/18/followup-on-ibm-system-sinfo... Quick summary: It's for doing complex event processing of large streams of data. Think financial data or any other such sort of never ending flow of massive amounts of data that you would need to process very quickly to identify patterns and for other analysis such as determining when you hit certain thresholds or trends. It's supposed to be very flexible so that it can act on a large amount of unstructured data as well as very scalable. It defines its own processing language too.
- tom_b 17y ago
  There seem to be echoes of what I've seen in kdb+ (http://kx.com/Products/kdb+.php http://kx.com/Products/kdb+.php) in this offering from IBM. I know that kdb+ is an entrenched part of the software developed internally at financial companies to deal with huge streaming datasets Any ideas how they compare? System S and kdb+ sound very similar, both offering a solution to development centered around streaming datasets with extremely low latencies, both with their own programming language.

10 points by brandonkm 17y ago

This sounds a bit too good to be true, and an awfully good way for IBM to sell it's consulting services. How can a system be capable of analyzing financial data, weather system, medical information, etc. without someone plugging in all of the data sources and explicitly telling the system how to correlate the data -- of course to do that you simply pay for a large helping of consulting... cha-ching! Maybe I'm missing something fundamental (the article was vague) but I just don't see anything really new here. I read it as marketing.

wmf 17y ago
To be clear, System S is a framework and programming language for building streaming apps; it doesn't actually know anything about finance, weather, etc. For the technical research underlying it, see http://domino.research.ibm.com/comm/research_projects.nsf/pages/esps.index.html http://domino.research.ibm.com/comm/research_projects.nsf/pa...

dkarl 17y ago

What does "real-time" mean? Oh yeah, it means "faster than our competitors." And "stream" means "the Cell marketing team encouraged us to use the word 'stream' a lot." Factor out those two buzzwords, and you have an article about IBM releasing some software that analyzes data really fast and helps you make decisions based on that data. If newspapers are in such dire financial straits they really need to start charging for stuff like this.

wmf 17y ago
I know I shouldn't, but... What does "real-time" mean? It means that data is processed at the same rate that it is produced. (Or as Wikipedia says, "originally it referred to a simulation that proceeded at a rate that matched that of the real process it was simulating.") A side effect of real-time is that RAM can be used for buffering rather than using disk for storage. And "stream" means "the Cell marketing team encouraged us to use the word 'stream' a lot." Wikipedia defines a stream as "a succession of data elements made available over time" and stream processing as "the quasi-continuous flow of data which is processed in a dataflow programming language as soon as the program state meets the starting condition of the stream". I don't think System S even runs on Cell. While "real-time" and "stream" are too frequently used as buzzwords these days, IBM is actually using them correctly in this case.
- dkarl 17y ago
  "Real-time" is relative to the time scale on which you look at it. Nobody dealing with a daily flow of business data actually falls behind and ends up making decisions on older and older data because processing is backed up. "Real-time" just means "faster;" it's relative to human expectations. If seismic data gathered this morning were processed and available for inspection late this afternoon, that would be "real-time." If an MRI was performed this morning, and the image wasn't available until this afternoon, that would not be "real-time." It's a mooshy term -- it has meaning in a specific domain, relative to human expectations in that domain. In the article, outside any specific domain, it doesn't mean anything. "a succession of data elements made available over time" and stream processing as "the quasi-continuous flow of data which is processed in... Basically automated batch processing where the "chunkiness" lies beneath a certain threshold of perception. While "real-time" and "stream" are too frequently used as buzzwords these days, IBM is actually using them correctly in this case. How can you tell? If you have a better link, with real information, post it!
  - scott_s 17y ago
    http://portal.acm.org/citation.cfm?id=1376616.1376729 http://portal.acm.org/citation.cfm?id=1376616.1376729 http://portal.acm.org/citation.cfm?id=1289612.1289615 http://portal.acm.org/citation.cfm?id=1289612.1289615
    - dkarl 17y ago
      This looks interesting. Now if somebody could explain how I was supposed to realize there was something interesting behind that crappy New York Times article (short of already knowing?) My original post got modded into oblivion, but I stand by it. The article looks like press material provided by IBM and lightly worked over by the Times writer whose name appears on it.
      - wmf 17y ago
        I know where you're coming from; I love writing cynical "buzzword alarm" comments. This time you got unlucky and criticized a legitimate project; I wouldn't worry too much about it.
joshu 17y ago
real-time is defined as "will process in a deterministic amount of time" in systems. a processor could make a hard commitment to have an answer in 3ms or whatever. this is hard to guarantee on linux or whatever, because you never know when you'll get swapped out or whatever.

smokinn 17y ago

People seem to be confused about what IBM's System S actually is. For a more technical look at their new offering I suggest these two posts: http://www.dbms2.com/2009/05/13/ibm-system-s-infosphere-streams-processing/ http://www.dbms2.com/2009/05/13/ibm-system-s-infosphere-stre... http://www.dbms2.com/2009/05/18/followup-on-ibm-system-sinfosphere-streams/ http://www.dbms2.com/2009/05/18/followup-on-ibm-system-sinfo... Quick summary: It's for doing complex event processing of large streams of data. Think financial data or any other such sort of never ending flow of massive amounts of data that you would need to process very quickly to identify patterns and for other analysis such as determining when you hit certain thresholds or trends. It's supposed to be very flexible so that it can act on a large amount of unstructured data as well as very scalable. It defines its own processing language too.

tom_b 17y ago
There seem to be echoes of what I've seen in kdb+ (http://kx.com/Products/kdb+.php http://kx.com/Products/kdb+.php) in this offering from IBM. I know that kdb+ is an entrenched part of the software developed internally at financial companies to deal with huge streaming datasets Any ideas how they compare? System S and kdb+ sound very similar, both offering a solution to development centered around streaming datasets with extremely low latencies, both with their own programming language.