9 ms·
Hermes – A message broker built on top of Kafka
- exabrial 7y agoIronically: Hermes was the name of a JMS user interface/testing tool.
- fwip 7y agoI think it's more a case of the same inspiration (Hermes, the Greek messenger of the gods) than irony.
- KaiserPro 7y agoalso the god of thievery and tricksters.
- codeduck 7y agoIt's also the name of a hilariously bad courier in the UK - so bad in fact that most people call it Herpes instead.
- vowelless 7y agoIt’s also the name of a bureaucrat on a certain tv show.
- chimen 7y agocuriously, nobody mentions the clothing designer :)
- user5994461 7y agothought it was a handbag designer.
- RickJWagner 7y agoMy thoughts exactly. (I immediately remembered the old-time JMS UI. I think it was on SourceForge.)
- deleted 7y ago[deleted]
- DiseasedBadger 7y agoIncreasingly, all technology news sounds like: "X: a blazingly fast X built on top of {something I vaguely thought did X}"
- dang 7y ago"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- colechristensen 7y agoHermes seems to wrap Kafka with HTTP to make integrating it simpler (and I think less robust).
- ravenstine 7y agoDon't forget "modern".
- lllr_finger 7y agoCan someone help me understand the value proposition for Hermes? The only thing I can see is that it abstracts away producing to and consuming from Kafka. The use cases provided answer why you'd use a message broker system, but not why you'd want to do it over HTTP. Edit: I understand HTTP is easier than Kafka, but is this something developers really struggle with when adopting Kafka? My experience is that they struggle with the nuances, behavior, and maintenance of Kafka/ZooKeeper more than anything. I also didn't see how it dealt with concepts like exactly once delivery - any experiences in that area?
- napsterbr 7y agoAs someone evaluating Kafka for the first time, it would be useful to know what Hermes provides other than Kafka already does. After glancing at the homepage I see the REST api and the fact it is push based. Honestly I don't see how it would fit on my use case but interesting project nonetheless. > exactly once delivery Kafka is known to provide exactly once semantics - given your producers and consumers follow some rules, notably being idempotent. When ingesting from Kafka Stream API, it is actually exactly once delivery. There are a couple posts on confluent.io explaining how they achieve this (sorry, currently on mobile, can't copy-paste without having an outburst on how unusable touch devices are for me).
- lllr_finger 7y agoSure, I'm aware of how Kafka works - this is an abstraction over all that, so I'm curious if and how Hermes can provide the same behaviors. I didn't see any relevant documentation at a quick glance.
- infecto 7y agoI am not too familiar with Hermes but there is a lot of power exposing a http endpoint. I think you are thinking too inside of the box here. The benefit here is not adoption from developers or inability to understand how Kafka works with more native libs. The value here is 1) publishing messages from more unique sources. Perhaps allowing your clients to publish messages. 2) You can enforce additional guarantees. Does the message conform to the a defined schema?
- twa927 7y agoCan someone provide actual high-level use cases for using Kafka? Prefereably use-cases not handled by RabbitMQ. I've seen a few talks about Kafka but they focused on the internals. My guess is that Kafka is for large systems for which managing a multi-node RabbitMQ cluster is too much trouble.
- chrisjc 7y agoThis is worth a read. https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha...
- twa927 7y agoI skimmed over it but it's again mostly about internal design. The high level use case I see is "publish-subscribe" which is handled by RabbitMQ and a dozen of other solutions. One use case I see is that the events published into Kafka are persisted so e.g. some component can see a history of some events (so this is something not handled by RabbitMQ). Is it right?
- lllr_finger 7y agoEvent streams between decoupled systems is kind of the sweet spot for Kafka. It's extremely easy to scale horizontally, and handles distributed work and network partitions in an easy to reason about way. I've also seen Rabbit be the bottleneck before where I've never really seen Kafka be the bottleneck in an architecture - it's very analogous to a firehose. For organizations shuttling messages and events between teams, it makes a very convenient lingua franca.
- chrisjc 7y agoI often do the same thing... skim through articles and papers to get the gist. Trust me, this is one to ready all the way through.
- Plugawy 7y agoThat's correct - being able to "rewind" a history of a topic (queue) is a powerful concept. But, Kafka is a bit harder to operate than RabbitMQ in my experience. (Somewhat related, one of our subsystems originally was built around Kafka, but later was migrated to RabbitMQ and Postgres)
- hestefisk 7y agoVery nice with a simple wrapper. That said I’m wondering if 9 out of 10 use cases could do with something simpler, ie zeromq, which scales really well.
- theomega 7y agoThis sounds very interesting. Did anyone get from the homepage what kinds of guarantees this offers? What if the HTTP endpoint where Hermes should push the data to is down? Does it retry? If yes, for how long?
- theomega 7y agoAnswering my own question (RTFM): https://hermes-pubsub.readthedocs.io/en/latest/user/subscribing/ https://hermes-pubsub.readthedocs.io/en/latest/user/subscrib... You can configure how long it retries and with what strategy.
- SkyRocknRoll 7y agoIf any of you looking for message and streaming system under single system then pulsar.apache.org supports both and lot more reliable and scales better than kafka