4 ms·

Yes, it is a transactional stream processing solution implemented on top of Hadoop YARN and HBase (using Tephra as the transaction engine). Why not Kafka+Storm

Yes, it is a transactional stream processing solution implemented on top of Hadoop YARN and HBase (using Tephra as the transaction engine). Why not Kafka+Storm : Tigon is primarily targeted at simplifying stream processing application development -- and in that regard provides exactly-once processing guarantee, dynamic scaling of processing nodes, transactionally access to user datastore etc.

necubi 12y ago
Can you talk more about how you guarantee exactly-once semantics? The documentation is pretty vague on the architectural details.
- deleted 12y ago
  [deleted]
- terenceyim 12y ago
  You can get more technical details from the QCon SF 2014 talk: Slides: http://www.slideshare.net/TerenceYim1/tigon-qcon-sf20141103 http://www.slideshare.net/TerenceYim1/tigon-qcon-sf20141103 Video: http://www.infoq.com/presentations/acid-stream-processing http://www.infoq.com/presentations/acid-stream-processing
- terenceyim 12y ago
  Underneath it uses Tephra (https://github.com/caskdata/tephra https://github.com/caskdata/tephra) as the distributed transaction manager for HBase together with a queue implementation on HBase to provide exactly-once processing.
czbond 12y ago
Thanks for clearing up the use case. I can now understand how it is completely different in use than a Kafka/Storm implementation. The reference implementations seem intriguing with the example of a real time ad network marketplace.
- haranraj 12y ago
  Sure! I would encourage you to also take a look at CDAP (http://cdap.io http://cdap.io) (http://github.com/caskdata/cdap http://github.com/caskdata/cdap), where Tigon is used as the realtime stream processing framework. CDAP's features allow developers to build even more powerful bigdata applications (https://github.com/caskdata/cdap-apps https://github.com/caskdata/cdap-apps). Here are some CDAP guides/tutorials that might be of interest: http://cask.co/guides http://cask.co/guides. Specifically this one: https://github.com/cdap-guides/cdap-flow-guide https://github.com/cdap-guides/cdap-flow-guide
  - czbond 12y ago
    Thanks a lot!! You helped immensely. Coincidentally, you posted this at a time that a large project came up requiring transactional and "Exactly Once" delivery requirements - where Kafka, Rabbit, SQS, etc won't work well. Thanks again!
    - haranraj 12y ago
      Great! If you have any questions about CDAP, please don't hesitate to contact us at : cdap-user@googlegroups.com. We are also on IRC Freenode : #cdap

necubi 12y ago

Can you talk more about how you guarantee exactly-once semantics? The documentation is pretty vague on the architectural details.

deleted 12y ago
[deleted]
terenceyim 12y ago
You can get more technical details from the QCon SF 2014 talk: Slides: http://www.slideshare.net/TerenceYim1/tigon-qcon-sf20141103 http://www.slideshare.net/TerenceYim1/tigon-qcon-sf20141103 Video: http://www.infoq.com/presentations/acid-stream-processing http://www.infoq.com/presentations/acid-stream-processing
terenceyim 12y ago
Underneath it uses Tephra (https://github.com/caskdata/tephra https://github.com/caskdata/tephra) as the distributed transaction manager for HBase together with a queue implementation on HBase to provide exactly-once processing.

czbond 12y ago

Thanks for clearing up the use case. I can now understand how it is completely different in use than a Kafka/Storm implementation. The reference implementations seem intriguing with the example of a real time ad network marketplace.

haranraj 12y ago
Sure! I would encourage you to also take a look at CDAP (http://cdap.io http://cdap.io) (http://github.com/caskdata/cdap http://github.com/caskdata/cdap), where Tigon is used as the realtime stream processing framework. CDAP's features allow developers to build even more powerful bigdata applications (https://github.com/caskdata/cdap-apps https://github.com/caskdata/cdap-apps). Here are some CDAP guides/tutorials that might be of interest: http://cask.co/guides http://cask.co/guides. Specifically this one: https://github.com/cdap-guides/cdap-flow-guide https://github.com/cdap-guides/cdap-flow-guide
- czbond 12y ago
  Thanks a lot!! You helped immensely. Coincidentally, you posted this at a time that a large project came up requiring transactional and "Exactly Once" delivery requirements - where Kafka, Rabbit, SQS, etc won't work well. Thanks again!
  - haranraj 12y ago
    Great! If you have any questions about CDAP, please don't hesitate to contact us at : cdap-user@googlegroups.com. We are also on IRC Freenode : #cdap