4 ms·

Can someone help me understand the value proposition for Hermes? The only thing I can see is that it abstracts away producing to and consuming from Kafka. The u

Can someone help me understand the value proposition for Hermes? The only thing I can see is that it abstracts away producing to and consuming from Kafka. The use cases provided answer why you'd use a message broker system, but not why you'd want to do it over HTTP. Edit: I understand HTTP is easier than Kafka, but is this something developers really struggle with when adopting Kafka? My experience is that they struggle with the nuances, behavior, and maintenance of Kafka/ZooKeeper more than anything. I also didn't see how it dealt with concepts like exactly once delivery - any experiences in that area?

napsterbr 7y ago
As someone evaluating Kafka for the first time, it would be useful to know what Hermes provides other than Kafka already does. After glancing at the homepage I see the REST api and the fact it is push based. Honestly I don't see how it would fit on my use case but interesting project nonetheless. > exactly once delivery Kafka is known to provide exactly once semantics - given your producers and consumers follow some rules, notably being idempotent. When ingesting from Kafka Stream API, it is actually exactly once delivery. There are a couple posts on confluent.io explaining how they achieve this (sorry, currently on mobile, can't copy-paste without having an outburst on how unusable touch devices are for me).
- lllr_finger 7y ago
  Sure, I'm aware of how Kafka works - this is an abstraction over all that, so I'm curious if and how Hermes can provide the same behaviors. I didn't see any relevant documentation at a quick glance.
infecto 7y ago
I am not too familiar with Hermes but there is a lot of power exposing a http endpoint. I think you are thinking too inside of the box here. The benefit here is not adoption from developers or inability to understand how Kafka works with more native libs. The value here is 1) publishing messages from more unique sources. Perhaps allowing your clients to publish messages. 2) You can enforce additional guarantees. Does the message conform to the a defined schema?
- lllr_finger 7y ago
  Developers can do both pretty easily today without too much effort. The TCO of Kafka clusters is considerably higher than the costs in developing an API that marries a POST to a Producer. It feels like you're saving time on one of the easier parts of the solution. Totally open to hearing more out of the box ideas. I don't think Hermes is bad for existing, I'm just not envisioning why I'd recommend it to someone.
  - bpicolo 7y ago
    It's a scale thing when you want a ton of different endpoints. This is one of the options that Google Pub/Sub and SNS can do for you (and if you use e.g. App Engine, this is also queueing model). It can sometimes be an easier model for bootstrapping message passing.
thibauts 7y ago
Exactly what I thought. It can alleviate the need to use flaky Kafka clients in some languages, but kind of disappointing in that it doesn't soften the main pain point of Kafka: operational and cognitive load.
thanatos_dem 7y ago
Exactly once delivery is not a thing, and confluent needs to stop openly lying to people about it. At least once delivery and idempotence is not the same, and has existed forever. Calling it “exactly once” is marketing BS. It’s the same as Oracle claiming for years to support serializable transactions when they didn’t, except that one is technically possible, they just didn’t support it in actuality.
- ryanworl 7y ago
  They do not use the phrase exactly-once delivery as far as I’ve seen, they say “exactly-once semantics”. And this is referring to a specific set of Kafka features that you previously had to build yourself if you used Kafka. I have not seen any reference to them claiming their solution is somehow novel. In fact, it just wraps up patterns people were doing anyway. Oracle claiming to support serializable transactions is also false (as you say), but calling it a lie is not the whole story. “ANSI serializable” and actually serializable are not the same thing. Oracle is “ANSI SQL-92 serializable”, as in no named anomalies from the spec. There just happen to be more anomalies like write skew which are not in the spec.
  - tiew9Vii 7y ago
    No they do, I completely agree with the original poster on this. Confluent over egg the marketing, I think they may of changed to the phrase “exactly once schematics” as Apache Pulsar labelled it that from the start and people calling out exactly once.
    - thanatos_dem 7y ago
      Even from their announcement blog post, they're intentionally mixing words - https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/ https://www.confluent.io/blog/exactly-once-semantics-are-pos... They may never describe Kafka itself without the word "semantics", but here are some other snippets: - "I know what some of you are thinking. Exactly once delivery is impossible" - "While some have outright said that exactly once delivery is probably impossible!" They mix their phrasing depending on what they're talking about, and whether they are referring to Kafka directly or indirectly.
guhcampos 7y ago
I see a bit of their reasoning. Taking from the article: "When you have an environment with 20+ services, code sharing, maintenance and following updates become problematic. At Allegro we had the chance to find it out. It’s better to take out dependencies from business services as much as possible." By adopting something like Hermes you are not really "taking out" a dependency, but abstracting it - as you said. Yet, by talking HTTP to the message broker you are abstracting away Kafka from your developers and your code. One less lib to depend on for each language you have in your architecture. One less version to control among your services, etc.
- gcb0 7y ago
  if you already have http clients on all you components. and it's not like bindings for kafka is that much uncommon than a decent http client.
  - joesb 7y ago
    You can use HTTP client to contact any HTTP compatible service. You can only use Kafka client to connect to Kafka. That sounds like a tautology but it's practical. There is way higher chance that you can make any of your future service response speak HTTP, than all of them speaking Kafka. If you add HTTP client libs today, there's higher chance of reuse than adding Kafka libs.
xeronz 7y ago
I only skimmed the write up on it, but knowing kafka fairly well, the following could potentially work better than what is included by kafka: - push model (dependent on use case) - filtering - throttling/rate negotiation - exactly once (kafka out of the box does not dedupe on the broker)
Groxx 7y ago
re HTTP specifically: one major benefit I've seen to even "just" HTTP wrappers for systems is that the HTTP ecosystem is extremely mature, even on relatively exotic languages / platforms / coding patterns / design constraints / etc. You want load balancing, context propagation, multiplexing, proxying, authentication, request tracing, [anything from a truly gigantic list, both in-code and around-your-system]? HTTP has it. Probably several. And they probably already work with everything you already have, and happily run unattended for years. Kafka... might? Kafka for language X.... might? But probably not. You want to extend Kafka to add X between Y and Z? Does the protocol even allow it? HTTP does, choose your flavor. Odds are even decent that a fair number of your engineers have already heard of or used it. --- There are benefits to specialized protocols, absolutely. But there are also benefits to letting everything just use the same robust HTTP client as everything else.
- deleted 7y ago
  [deleted]
rswail 7y ago
Having just completed a project using Kafka as an event pipeline and a data store, one of the issues we found was that consumer polling takes a large chunk of resources. Having a push model for consumption would certainly remove some of the complexity we had to deal with for scaling out consumption.
ethnoe 7y ago
Hi there, I have been the technical team lead of Hermes team for ~four years, before Łukasz (the author of blogpost) took over. Thanks for taking time to read, think and write about our product :) Our value proposition is built on four main aspects: * ease of integration * easier Kafka management * centralised management and validation * increased stability / reliability Mind that some of the points don't make much sense unless you have a lot of services managed by a lot of independent teams. Thus Łukasz remark about "20+ microservices" in the original post. We run 700 microservices on prod managed by something close to 70 teams. Ease of integration has been nicely summed up by others in this thread. HTTP tends to be the simplest way to integrate anything nowadays, at least in our case. While this comes at a cost, being able to get projects started up very quickly, without getting into details of proper handling of Kafka producer/consumer clients provided great value for us. Also while history might not be so important considering using Hermes in 2019 because Kafka matured, gained traction and recognisability among Software Engineers, it wasn't so easy to handle Kafka in 0.7/0.8 days when we started. Of course switching to HTTP comes at a cost. I think the biggest one is using pure HTTP in push model. This makes it impossible to take advantage of Kafka data model, which guarantees event ordering at partition level. Zalando took a different approach with Nakadi (https://github.com/zalando/nakadi https://github.com/zalando/nakadi). I would say that at some point Hermes should consider following this path for more advanced users. Easier Kafka management. Since we abstract away Kafka and hide it behind HTTP/REST API, we can easily introduce many changes to Kafka clusters. One of them was splitting Kafka cluster into two operate ones (one per our DC) without clients noticing. They were still publishing to same old Hermes instances, discovered via Consul. While doing it with clients might seem like a trivial thing to do when you have just a few services that use Kafka, with a few hundreds of clients it generates a lot of unnecessary work for developers. Now whenever we need to do some maintenance with Kafka clusters (rebalance partitions, change cluster/hosts etc) we just route the traffic at Hermes level and no interaction with clients/developers is necessary. Centralised management and validation. We started with publishing JSON. Along the way, as more and more people started consuming data offline (from Hadoop), it turned out that moving to some structured/schema based format is necessary, thus Avro. Hermes helped us a lot with this. It enables us to fail fast when someone starts publishing malformed requests for whatever reason, instead of relying on consumer (online and offline) to be hit and have to communicate with producer. Secondly support for Avro in JVM (our main microservice platform) is not that great and we put a lot effort into making it better (including publishing https://github.com/allegro/json-avro-converter https://github.com/allegro/json-avro-converter). By having Hermes to do on-the-fly conversion for both publishers and subscribers we made it possible to only define schema and deal as little with Avro as possible in simple cases when it might not be beneficial for the team. We also have Hermes integrated with our Service Catalog, so we can easily track ownership of topics and subscriptions. People publishing have easy access to information about who not only subscribes to online data, but also who accesses data offline (via Hadoop) using our offline clients feature. This way Hermes provides central place to manage our data streams. Increased stability/reliability. This last one might be controversial, but in practice it did save us a few times. Mind, that I mean increased (more nines), not totally bulletproof. Kafka is a great, resilient piece of software. it is also complex and incidents happen. It might not even be that cluster is down - but increasing response times from few ms to 1second can be just as deadly. Hermes Frontend on the other hand is really simple. By putting it in front of Kafka together with built-in buffering support, we added a layer which increased our reliability. Now even if Kafka cluster has huge problems, we can accept incoming events for 2-3 hours, having time to either resolve the issue or reroute traffic to other cluster. This means that microservices don’t have to deal with data buffering on their own. Of course Hermes is still pretty much stateless by itself, so when traffic to Kafka flows normally, we can restart, spin up and spin down instances at will. Entering danger zone: if both Kafka goes down and Hermes hosts blow up - the data is lost. This is a trade off and we are happy to say that for years running Hermes + Kafka on production, it never failed and saved us a few times. I hope that I managed to clarify why we are using Hermes as main message bus powering our microservice architecture. We open sourced it, as we wanted to do our work in the open, sharing it with anyone who finds it useful and beneficial :)

napsterbr 7y ago

As someone evaluating Kafka for the first time, it would be useful to know what Hermes provides other than Kafka already does. After glancing at the homepage I see the REST api and the fact it is push based. Honestly I don't see how it would fit on my use case but interesting project nonetheless. > exactly once delivery Kafka is known to provide exactly once semantics - given your producers and consumers follow some rules, notably being idempotent. When ingesting from Kafka Stream API, it is actually exactly once delivery. There are a couple posts on confluent.io explaining how they achieve this (sorry, currently on mobile, can't copy-paste without having an outburst on how unusable touch devices are for me).

lllr_finger 7y ago
Sure, I'm aware of how Kafka works - this is an abstraction over all that, so I'm curious if and how Hermes can provide the same behaviors. I didn't see any relevant documentation at a quick glance.

infecto 7y ago

I am not too familiar with Hermes but there is a lot of power exposing a http endpoint. I think you are thinking too inside of the box here. The benefit here is not adoption from developers or inability to understand how Kafka works with more native libs. The value here is 1) publishing messages from more unique sources. Perhaps allowing your clients to publish messages. 2) You can enforce additional guarantees. Does the message conform to the a defined schema?

lllr_finger 7y ago
Developers can do both pretty easily today without too much effort. The TCO of Kafka clusters is considerably higher than the costs in developing an API that marries a POST to a Producer. It feels like you're saving time on one of the easier parts of the solution. Totally open to hearing more out of the box ideas. I don't think Hermes is bad for existing, I'm just not envisioning why I'd recommend it to someone.
- bpicolo 7y ago
  It's a scale thing when you want a ton of different endpoints. This is one of the options that Google Pub/Sub and SNS can do for you (and if you use e.g. App Engine, this is also queueing model). It can sometimes be an easier model for bootstrapping message passing.

thibauts 7y ago

Exactly what I thought. It can alleviate the need to use flaky Kafka clients in some languages, but kind of disappointing in that it doesn't soften the main pain point of Kafka: operational and cognitive load.

thanatos_dem 7y ago

Exactly once delivery is not a thing, and confluent needs to stop openly lying to people about it. At least once delivery and idempotence is not the same, and has existed forever. Calling it “exactly once” is marketing BS. It’s the same as Oracle claiming for years to support serializable transactions when they didn’t, except that one is technically possible, they just didn’t support it in actuality.

ryanworl 7y ago
They do not use the phrase exactly-once delivery as far as I’ve seen, they say “exactly-once semantics”. And this is referring to a specific set of Kafka features that you previously had to build yourself if you used Kafka. I have not seen any reference to them claiming their solution is somehow novel. In fact, it just wraps up patterns people were doing anyway. Oracle claiming to support serializable transactions is also false (as you say), but calling it a lie is not the whole story. “ANSI serializable” and actually serializable are not the same thing. Oracle is “ANSI SQL-92 serializable”, as in no named anomalies from the spec. There just happen to be more anomalies like write skew which are not in the spec.
- tiew9Vii 7y ago
  No they do, I completely agree with the original poster on this. Confluent over egg the marketing, I think they may of changed to the phrase “exactly once schematics” as Apache Pulsar labelled it that from the start and people calling out exactly once.
  - thanatos_dem 7y ago
    Even from their announcement blog post, they're intentionally mixing words - https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/ https://www.confluent.io/blog/exactly-once-semantics-are-pos... They may never describe Kafka itself without the word "semantics", but here are some other snippets: - "I know what some of you are thinking. Exactly once delivery is impossible" - "While some have outright said that exactly once delivery is probably impossible!" They mix their phrasing depending on what they're talking about, and whether they are referring to Kafka directly or indirectly.

guhcampos 7y ago

I see a bit of their reasoning. Taking from the article: "When you have an environment with 20+ services, code sharing, maintenance and following updates become problematic. At Allegro we had the chance to find it out. It’s better to take out dependencies from business services as much as possible." By adopting something like Hermes you are not really "taking out" a dependency, but abstracting it - as you said. Yet, by talking HTTP to the message broker you are abstracting away Kafka from your developers and your code. One less lib to depend on for each language you have in your architecture. One less version to control among your services, etc.

gcb0 7y ago
if you already have http clients on all you components. and it's not like bindings for kafka is that much uncommon than a decent http client.
- joesb 7y ago
  You can use HTTP client to contact any HTTP compatible service. You can only use Kafka client to connect to Kafka. That sounds like a tautology but it's practical. There is way higher chance that you can make any of your future service response speak HTTP, than all of them speaking Kafka. If you add HTTP client libs today, there's higher chance of reuse than adding Kafka libs.

xeronz 7y ago

I only skimmed the write up on it, but knowing kafka fairly well, the following could potentially work better than what is included by kafka: - push model (dependent on use case) - filtering - throttling/rate negotiation - exactly once (kafka out of the box does not dedupe on the broker)

Groxx 7y ago

re HTTP specifically: one major benefit I've seen to even "just" HTTP wrappers for systems is that the HTTP ecosystem is extremely mature, even on relatively exotic languages / platforms / coding patterns / design constraints / etc. You want load balancing, context propagation, multiplexing, proxying, authentication, request tracing, [anything from a truly gigantic list, both in-code and around-your-system]? HTTP has it. Probably several. And they probably already work with everything you already have, and happily run unattended for years. Kafka... might? Kafka for language X.... might? But probably not. You want to extend Kafka to add X between Y and Z? Does the protocol even allow it? HTTP does, choose your flavor. Odds are even decent that a fair number of your engineers have already heard of or used it. --- There are benefits to specialized protocols, absolutely. But there are also benefits to letting everything just use the same robust HTTP client as everything else.

deleted 7y ago
[deleted]

rswail 7y ago

Having just completed a project using Kafka as an event pipeline and a data store, one of the issues we found was that consumer polling takes a large chunk of resources. Having a push model for consumption would certainly remove some of the complexity we had to deal with for scaling out consumption.

ethnoe 7y ago

Hi there, I have been the technical team lead of Hermes team for ~four years, before Łukasz (the author of blogpost) took over. Thanks for taking time to read, think and write about our product :) Our value proposition is built on four main aspects: * ease of integration * easier Kafka management * centralised management and validation * increased stability / reliability Mind that some of the points don't make much sense unless you have a lot of services managed by a lot of independent teams. Thus Łukasz remark about "20+ microservices" in the original post. We run 700 microservices on prod managed by something close to 70 teams. Ease of integration has been nicely summed up by others in this thread. HTTP tends to be the simplest way to integrate anything nowadays, at least in our case. While this comes at a cost, being able to get projects started up very quickly, without getting into details of proper handling of Kafka producer/consumer clients provided great value for us. Also while history might not be so important considering using Hermes in 2019 because Kafka matured, gained traction and recognisability among Software Engineers, it wasn't so easy to handle Kafka in 0.7/0.8 days when we started. Of course switching to HTTP comes at a cost. I think the biggest one is using pure HTTP in push model. This makes it impossible to take advantage of Kafka data model, which guarantees event ordering at partition level. Zalando took a different approach with Nakadi (https://github.com/zalando/nakadi https://github.com/zalando/nakadi). I would say that at some point Hermes should consider following this path for more advanced users. Easier Kafka management. Since we abstract away Kafka and hide it behind HTTP/REST API, we can easily introduce many changes to Kafka clusters. One of them was splitting Kafka cluster into two operate ones (one per our DC) without clients noticing. They were still publishing to same old Hermes instances, discovered via Consul. While doing it with clients might seem like a trivial thing to do when you have just a few services that use Kafka, with a few hundreds of clients it generates a lot of unnecessary work for developers. Now whenever we need to do some maintenance with Kafka clusters (rebalance partitions, change cluster/hosts etc) we just route the traffic at Hermes level and no interaction with clients/developers is necessary. Centralised management and validation. We started with publishing JSON. Along the way, as more and more people started consuming data offline (from Hadoop), it turned out that moving to some structured/schema based format is necessary, thus Avro. Hermes helped us a lot with this. It enables us to fail fast when someone starts publishing malformed requests for whatever reason, instead of relying on consumer (online and offline) to be hit and have to communicate with producer. Secondly support for Avro in JVM (our main microservice platform) is not that great and we put a lot effort into making it better (including publishing https://github.com/allegro/json-avro-converter https://github.com/allegro/json-avro-converter). By having Hermes to do on-the-fly conversion for both publishers and subscribers we made it possible to only define schema and deal as little with Avro as possible in simple cases when it might not be beneficial for the team. We also have Hermes integrated with our Service Catalog, so we can easily track ownership of topics and subscriptions. People publishing have easy access to information about who not only subscribes to online data, but also who accesses data offline (via Hadoop) using our offline clients feature. This way Hermes provides central place to manage our data streams. Increased stability/reliability. This last one might be controversial, but in practice it did save us a few times. Mind, that I mean increased (more nines), not totally bulletproof. Kafka is a great, resilient piece of software. it is also complex and incidents happen. It might not even be that cluster is down - but increasing response times from few ms to 1second can be just as deadly. Hermes Frontend on the other hand is really simple. By putting it in front of Kafka together with built-in buffering support, we added a layer which increased our reliability. Now even if Kafka cluster has huge problems, we can accept incoming events for 2-3 hours, having time to either resolve the issue or reroute traffic to other cluster. This means that microservices don’t have to deal with data buffering on their own. Of course Hermes is still pretty much stateless by itself, so when traffic to Kafka flows normally, we can restart, spin up and spin down instances at will. Entering danger zone: if both Kafka goes down and Hermes hosts blow up - the data is lost. This is a trade off and we are happy to say that for years running Hermes + Kafka on production, it never failed and saved us a few times. I hope that I managed to clarify why we are using Hermes as main message bus powering our microservice architecture. We open sourced it, as we wanted to do our work in the open, sharing it with anyone who finds it useful and beneficial :)