4 ms·
I work on IBM InfoSphere Streams (https://www.ibmdw.net/streamsdev/ https://www.ibmdw.net/streamsdev/), and the language our system uses is SPL (Streams Process
by scott_s 13y ago
I work on IBM InfoSphere Streams (https://www.ibmdw.net/streamsdev/ https://www.ibmdw.net/streamsdev/), and the language our system uses is SPL (Streams Processing Language). I was curious to see what this problem would look like implemented in SPL, so came up this this:
composite Main {
graph
@parallel(width=2)
stream<rstring word> Words = Custom() {
logic onProcess: {
while (true) {
// a "word" looks like "0", "1", ..., "99"
submit({word=(rstring)((int32)(random() * 100))}, Words);
}
}
}
@parallel(width=3, partitionBy=[{port=Words, attributes=[word]}])
stream<rstring word, int32 rolling_count> Counts = Aggregate(Words) {
window Words: sliding, time(9), time(3), partitioned;
param partitionBy: word;
output Counts: word=Any(), rolling_count=Count();
}
stream<rstring word, int32 rolling_count> Sorted = Sort(Words) {
window Words: sliding, time(2), time(2);
param sortBy: rolling_count;
}
stream<list<rstring word, int32 rolling_count> hist> Rankings = Aggregate(Sorted) {
window Sorted: tumbling, punctuation;
output Rankings: hist=Collect();
}
}
I did not parallelize the reduce part. I'd need to do some more thinking on how to do that, as I split it into two operators. I think it shows the flavor, though.
Two caveats: One, I have not compiled and tested this, so errors may abound. I was just curious what the "flavor" of it would look like in our system. Two, standard disclaimer of I speak in an unofficial capacity, what I say does not represent IBM or Streams.