5 ms·
Tigon: High-throughput, real-time stream processing framework
- czbond 12y agoSo this is their own custom solution rather than using Kafka and Storm?
- haranraj 12y agoYes, it is a transactional stream processing solution implemented on top of Hadoop YARN and HBase (using Tephra as the transaction engine). Why not Kafka+Storm : Tigon is primarily targeted at simplifying stream processing application development -- and in that regard provides exactly-once processing guarantee, dynamic scaling of processing nodes, transactionally access to user datastore etc.
- necubi 12y agoCan you talk more about how you guarantee exactly-once semantics? The documentation is pretty vague on the architectural details.
- deleted 12y ago[deleted]
- terenceyim 12y agoYou can get more technical details from the QCon SF 2014 talk: Slides: http://www.slideshare.net/TerenceYim1/tigon-qcon-sf20141103 http://www.slideshare.net/TerenceYim1/tigon-qcon-sf20141103 Video: http://www.infoq.com/presentations/acid-stream-processing http://www.infoq.com/presentations/acid-stream-processing
- terenceyim 12y agoUnderneath it uses Tephra (https://github.com/caskdata/tephra https://github.com/caskdata/tephra) as the distributed transaction manager for HBase together with a queue implementation on HBase to provide exactly-once processing.
- czbond 12y agoThanks for clearing up the use case. I can now understand how it is completely different in use than a Kafka/Storm implementation. The reference implementations seem intriguing with the example of a real time ad network marketplace.
- haranraj 12y agoSure! I would encourage you to also take a look at CDAP (http://cdap.io http://cdap.io) (http://github.com/caskdata/cdap http://github.com/caskdata/cdap), where Tigon is used as the realtime stream processing framework. CDAP's features allow developers to build even more powerful bigdata applications (https://github.com/caskdata/cdap-apps https://github.com/caskdata/cdap-apps). Here are some CDAP guides/tutorials that might be of interest: http://cask.co/guides http://cask.co/guides. Specifically this one: https://github.com/cdap-guides/cdap-flow-guide https://github.com/cdap-guides/cdap-flow-guide
- czbond 12y agoThanks a lot!! You helped immensely. Coincidentally, you posted this at a time that a large project came up requiring transactional and "Exactly Once" delivery requirements - where Kafka, Rabbit, SQS, etc won't work well. Thanks again!
- haranraj 12y agoGreat! If you have any questions about CDAP, please don't hesitate to contact us at : cdap-user@googlegroups.com. We are also on IRC Freenode : #cdap
- swah 12y agoIn the Big Data era, I can no longer feel excited about most projects, because my project ideas don't operate at that scale - at least initially. So I can't imagine anything cool to build with it - unlike Redis or Postgres, say.
- haranraj 12y agoYou can take a look at sample applications built using Tigon : https://github.com/caskdata/tigon-apps https://github.com/caskdata/tigon-apps Also, Tigon is the realtime processing framework used in CDAP (cdap.io). Sample applications built using CDAP (Cask Data Application Platform): https://github.com/caskdata/cdap-apps https://github.com/caskdata/cdap-apps
- orthecreedence 12y agoHate to nitpick, but in FF on windows, the font is almost unreadable (very choppy, and the top few pixels of each character is cut off). The GothamLight font is not doing you any favors!
- joltie 12y agoPoint taken and your are right. It sucks! we will fix it.