5 ms·
The Github repo has an example application: https://github.com/s4/examples/tree/master/twittertopiccount/src/main/java/io/s4/example/twittertopiccount/ https://
by chr15 16y ago
The Github repo has an example application: https://github.com/s4/examples/tree/master/twittertopiccount/src/main/java/io/s4/example/twittertopiccount/ https://github.com/s4/examples/tree/master/twittertopiccount...
It's a twitter topic counter: "This application detects popular hashtags on Twitter by listening to the Twitter gardenhose."
From http://labs.yahoo.com/event/99 http://labs.yahoo.com/event/99 :
"S4 is a general-purpose, distributed, scalable, partially fault-tolerant, pluggable platform that allows programmers to easily develop applications for processing continuous unbounded streams of data. Keyed data events are routed with affinity to Processing Elements (PEs), which consume the events and do one or both of the following: (1) emit one or more events which may be consumed by other PEs, (2) publish results. The architecture resembles the Actors model [1], providing semantics of encapsulation and location transparency, thus allowing applications to be massively concurrent while exposing a simple programming interface to application developers."
Actor model: http://en.wikipedia.org/wiki/Actor_model http://en.wikipedia.org/wiki/Actor_model
- jamii 16y agoThe model is very similar to the one argued for in this paper: http://www.cidrdb.org/cidr2007/papers/cidr07p15.pdf http://www.cidrdb.org/cidr2007/papers/cidr07p15.pdf That paper was a big inspiration when we were redesigning the betting exchange at smarkets. It's a very well reasoned exposition of why this is the only sensible architecture for large scale distributed systems.
- pants 16y agoDoes your version have "full" fault tolerance or only the "partial" fault tolerance noted on the s4 site.
- jamii 16y agoI guess it depends how you define 'full' fault tolerance. In the case of a machine failure it would lose a couple of seconds worth of transactions. In a data-processing scenario thats not an issue - just re-run the data that hasn't been processed yet. In an exchange the recovery delay is a bit more painful and there is always the potential to permanently lose transactions. Unfortunately there doesn't seem to be a way around that - we must have consistency and performance so machine failure is always going to cause some interruption to availability
- protomyth 16y agoI always thought it would very interesting to try an build a language that had an Actor focus but used simple objects for basic stuff (a pure model seems a little odd - I guess C++ for actors). It just seems like a natural way to organize a big system. We talk about certain objects as doing things to other objects and it would provide a simpler concurrency. Actors, linda, tuples..... hum....