8 ms·

DNS for Service Discovery in HAProxy

87 points by phil21 9y ago

otterley 9y ago
Looks like both HAProxy and Nginx are going down the "pay for more advanced features" route: https://www.haproxy.com/products/haproxy-enterprise-edition/ https://www.haproxy.com/products/haproxy-enterprise-edition/ https://www.nginx.com/products/nginx/ https://www.nginx.com/products/nginx/
- arianvanp 9y ago
  Note, this feature will be in the free version when 1.8 is out. Early access is for enterprise customers.
  - abiox 9y ago
    is there official confirmation of this? it would seem like a change of pace given how the community vs enterprise feature lists play out: http://www.haproxy.com/products/haproxy-community-vs-enterprise-edition.html http://www.haproxy.com/products/haproxy-community-vs-enterpr...
  - bedis9 9y ago
    no, this feature is already available in -dev community branch. So it's already available for free. Note this feature has not yet been backported into HAProxy Enterprise...
- wtarreau 9y ago
  What ? Are you kidding ? We're extremely open, we currently have 7 devs working full time for most of the stuff you'll have in 1.8, we provide all the sources with our packages (and some of our customers do their own builds), and all the stuff we add in the enterprise version is in fact a fine selection of stable enough features from the dev branch for people who need them right now without waiting for the next version to be released. When a customer asks for some specific feature we explain them that it first has to be in mainline so we can backport it and it gets a wider exposure to spot bugs. How can you be that insulting ? If it's a matter of unclear explanation on the web site, it's possible and then please report this to contact@ or something like this (I don't know the address). If it's just because you want to criticize without knowing, well, in this case I hope I'll never count you among our users!
  - joeskyyy 9y ago
    Easy there, tiger. Not doing well for the brand of your team.
    - wtarreau 9y ago
      ?
      - jamesjamesm 9y ago
        You're not... he has a point.
    - throw2016 9y ago
      Correcting an assumption can hardly be termed as 'lashing out', maybe a 'robust' response. Rather have founders like wtarreau speak their mind than adopt some fake passive aggressive tone or have a 'dev relations manager' spinning, at least here.
      - otterley 9y ago
        What assumption was being corrected here? Please cite with specificity.
        girvo 9y ago
        To me, it seems that one could read an (unsaid) implication in your original comment that is “and those advanced features will only ever be behind a paywall” — as I think in nginx’s case that’s actually true for some features (but I might be remembering incorrectly!), and by listing and comparing HAproxy and nginx in that manner I can see that implication. But that’s being kinda nit picky, though if I was an HAproxy dev and all features were developed in the open and released into the mainline for free in a new stable version in the future, I would probably read it that way and be a little miffed, I suppose
        9y ago
- bedis9 9y ago
  Well, nignx inc does first develop their proprietary software... Then, they "open" some features. With nginx plus, you're locked to nginx, like when you were using a F5 load-balancer! HAProxy Technologies pushes first its developments in the -dev community branch. Check the commits for the feature given above (SRV records, non exhaustive list of patches): http://git.haproxy.org/?p=haproxy.git&a=search&h=HEAD&st=commit&s=SRV+record http://git.haproxy.org/?p=haproxy.git&a=search&h=HEAD&st=com... HAProxy technologies simply makes the effort of maintaining a branch of HAProxy in which some -dev features are backported. That way, users of HAProxy Enterprise have the most stable and feature reach version of HAProxy. And as Willy stated, every user of HAProxy Enterprise also have access to the source packages. So from a business point of view, the main difference between HAProxy and nginx is that HAProxy is the respects both its community and customers.
arianvanp 9y ago
I hope the non-blocking DNS Client will be released separately as well. This seems very useful on its own
- otterley 9y ago
  Have you tried c-ares? It's venerable and useful.
  - clan 9y ago
    For the lazy (like me!): https://c-ares.haxx.se https://c-ares.haxx.se
- teddyh 9y ago
  Why not use GetDNS¹? 1. https://getdnsapi.net/ https://getdnsapi.net/
- bedis9 9y ago
  Unfortunately, this is not possible! The DNS in HAProxy relies on the internal task scheduler which is itself event-driven.
dcow 9y ago
I've been exploring DNS PTR/SRV records for seamless service discovery regardless of whether you have access to an authoritative DNS server configured for a given local domain or not (so things simply devolve to whether or not use use multicast or unicast to resolve SRV records). Basically DNS-SD. The idea of using SRV records in general seems foreign even in the ops realm. It's awesome to see HAProxy beginning to familiarize people with the concept of SRV records.
- jlgaddis 9y ago
  > The idea of using SRV records in general seems foreign even in the ops realm. I'm not suggesting you use it but (as for how it works in general) look into how Microsoft's Active Directory works. They've had the "service discovery" problem figured out for going on two decades, including preferences for "closer" servers, weighting, and so on. I'm no MS fan by any means but AD is one MS product that really is awesome.
  - snuxoll 9y ago
    And how does MS do it? SRV records, that’s right.
    - jlgaddis 9y ago
      Yeah, exactly. That's why I suggested he look into how it works (since "the idea ... seems foreign" to him). I didn't mean to suggest that they had some revolutionary new unique way of handling service discovery or even that they were the first. I think maybe you misinterpreted or misunderstood my intentions.
      - snuxoll 9y ago
        It’s all good, I got what you meant :) LDAP and Kerberos of all implementation are probably the most common use case for SRV records, most people probably don’t even know it. Real shame, in many ways I wish A/AAAA records weren’t even a thing for the vast majority of services (HTTP, I’m looking at you!)
linkmotif 9y ago
Maybe there will there be a Kubernetes a Ingress Controller?? :)
- aiharos 9y ago
  Kubernetes Ingress Controllers for HAProxy already exist. A nice one is at https://github.com/jcmoraisjr/haproxy-ingress https://github.com/jcmoraisjr/haproxy-ingress Other examples of usage are spread around in the examples section of the ingress repo, like here https://github.com/kubernetes/ingress/tree/master/examples/deployment/haproxy https://github.com/kubernetes/ingress/tree/master/examples/d... The DNS-SD code will ship with HAProxy 1.8, and we will gladly help out by adding it to the above ingress controller after the release. A WIP of that code is up on github in case you’re interested. In fact, HAProxy is full of surprises and little known features (like the dynamic scaling via haproxy runtime api we already contributed to the controller) and we're happy to share them with everyone!
Whitespace 9y ago
Anyone here ever used SRV records for local development? I really like the idea of running a local DNS server with port information built in instead of a full-blown service discovery/routing/service mesh setup.
- Cieplak 9y ago
  Yes. Current setup runs a local DNS server which serves SRV records containing the local IP addresses and ports of services running inside FreeBSD jails. For instance, the payments service in one jail can communicate with the risk service running in another jail on the same host by sending an HTTP message to http://risk.local http://risk.local Currently using dnsmasq but would like to use erl-dns once I finish wrapping it with an HTTP admin API. https://github.com/aetrion/erl-dns https://github.com/aetrion/erl-dns
- doublerebel 9y ago
  Yes, try Consul. It's extremely easy to configure and run.
  - bedis9 9y ago
    I do agree on this statement!!! I personally validated HAProxy SRV records with both Kubernetes and Consul and I was positively supersized of how simple is consul. (Well, it does not cover all the stuff kubernetes does, you may need nomad for this purpose).
doublerebel 9y ago
Interesting that HAProxy believes it is incumbent enough to charge for directly copying a feature Consul has had for years. When I added Consul to my stack 3 years ago HAProxy was already looking behind the times... I don't think they can catch up for any new installation.
- wtarreau 9y ago
  To be honest it's not a matter of duplicating, catching up or anything. It's just that when a significant number of our users ask for exactly the same stuff with similar and sometimes different justifications, it becomes quite hard to avoid adding it. And if they want it there, they very likely have their own very valid reasons. Baptiste (the DNS subsystem maintainer) has a significant amount of use cases in mind and could likely much better explain than me.
  - bedis9 9y ago
    We first developped DNS in HAProxy for our AWS users who wanted HAProxy to follow-up a node when it is restarted (and it's IP is changed)... That was the very first request from both community and customers. After we released this feature, the community came back with some requirements regarding the ability to use DNS resolution to resolve all the servers (or a set of servers at list) of a backend. So we had to improve a lot the first dev we did to make this possible. Support for SRV record is just the last stone we put on top of many other devs :) We're quite proud of the result because it makes HAProxy able to scale up / down at run time without being reloaded and compatible with any service registry able to export a list of nodes delivering a same service through DNS. HAProxy can use multiple name for the resolution, pointing to different set of DNS servers, enforcing custom "hold" timers (to bypass server's TTL or negative TTLs in case of NX, etc...) and mix all of this with "old style hardcoded" servers in the backend... HAProxy is flexible :)
- vidarh 9y ago
  I don't understand what you mean here. Consul is not a load balancer. It's functionality is almost entirely orthogonal to Haproxy. If anything, a lot of people pair the two, but have so far often had to resort to cumbersome solutions for rewriting the Haproxy config to keep the set of backends in Haproxy updated. If anything, this will make Haproxy and Consul work much better together, given that Consul serves up SRV records for the services you register with it.
bitly 9y ago
Hi there. This is really awesome news and i am delighted to hear about this. Any change to get some directions how to test this functionality? So far I have currently 1.8-dev2 up and running but now I find no real references about how to try it .. and if available in this repo. (maybe i have to dig into the code ?) Thanks and all the best.
- bitly 9y ago
  http://git.haproxy.org/?p=haproxy.git;a=commitdiff;h=e2d03d2a43473a637491c0a8c9e10d5caed8e63b;hp=e70bc05b3a62ca84b5ce4440d340440f00411886 http://git.haproxy.org/?p=haproxy.git;a=commitdiff;h=e2d03d2... Thanks and sorry for the noise

87 points by phil21 9y ago

Looks like both HAProxy and Nginx are going down the "pay for more advanced features" route: https://www.haproxy.com/products/haproxy-enterprise-edition/ https://www.haproxy.com/products/haproxy-enterprise-edition/ https://www.nginx.com/products/nginx/ https://www.nginx.com/products/nginx/

arianvanp 9y ago
Note, this feature will be in the free version when 1.8 is out. Early access is for enterprise customers.
- abiox 9y ago
  is there official confirmation of this? it would seem like a change of pace given how the community vs enterprise feature lists play out: http://www.haproxy.com/products/haproxy-community-vs-enterprise-edition.html http://www.haproxy.com/products/haproxy-community-vs-enterpr...
- bedis9 9y ago
  no, this feature is already available in -dev community branch. So it's already available for free. Note this feature has not yet been backported into HAProxy Enterprise...
wtarreau 9y ago
What ? Are you kidding ? We're extremely open, we currently have 7 devs working full time for most of the stuff you'll have in 1.8, we provide all the sources with our packages (and some of our customers do their own builds), and all the stuff we add in the enterprise version is in fact a fine selection of stable enough features from the dev branch for people who need them right now without waiting for the next version to be released. When a customer asks for some specific feature we explain them that it first has to be in mainline so we can backport it and it gets a wider exposure to spot bugs. How can you be that insulting ? If it's a matter of unclear explanation on the web site, it's possible and then please report this to contact@ or something like this (I don't know the address). If it's just because you want to criticize without knowing, well, in this case I hope I'll never count you among our users!
- joeskyyy 9y ago
  Easy there, tiger. Not doing well for the brand of your team.
  - wtarreau 9y ago
    ?
    - jamesjamesm 9y ago
      You're not... he has a point.
  - throw2016 9y ago
    Correcting an assumption can hardly be termed as 'lashing out', maybe a 'robust' response. Rather have founders like wtarreau speak their mind than adopt some fake passive aggressive tone or have a 'dev relations manager' spinning, at least here.
    - otterley 9y ago
      What assumption was being corrected here? Please cite with specificity.
      - girvo 9y ago
        To me, it seems that one could read an (unsaid) implication in your original comment that is “and those advanced features will only ever be behind a paywall” — as I think in nginx’s case that’s actually true for some features (but I might be remembering incorrectly!), and by listing and comparing HAproxy and nginx in that manner I can see that implication. But that’s being kinda nit picky, though if I was an HAproxy dev and all features were developed in the open and released into the mainline for free in a new stable version in the future, I would probably read it that way and be a little miffed, I suppose
        9y ago
bedis9 9y ago
Well, nignx inc does first develop their proprietary software... Then, they "open" some features. With nginx plus, you're locked to nginx, like when you were using a F5 load-balancer! HAProxy Technologies pushes first its developments in the -dev community branch. Check the commits for the feature given above (SRV records, non exhaustive list of patches): http://git.haproxy.org/?p=haproxy.git&a=search&h=HEAD&st=commit&s=SRV+record http://git.haproxy.org/?p=haproxy.git&a=search&h=HEAD&st=com... HAProxy technologies simply makes the effort of maintaining a branch of HAProxy in which some -dev features are backported. That way, users of HAProxy Enterprise have the most stable and feature reach version of HAProxy. And as Willy stated, every user of HAProxy Enterprise also have access to the source packages. So from a business point of view, the main difference between HAProxy and nginx is that HAProxy is the respects both its community and customers.

arianvanp 9y ago

I hope the non-blocking DNS Client will be released separately as well. This seems very useful on its own

otterley 9y ago
Have you tried c-ares? It's venerable and useful.
- clan 9y ago
  For the lazy (like me!): https://c-ares.haxx.se https://c-ares.haxx.se
teddyh 9y ago
Why not use GetDNS¹? 1. https://getdnsapi.net/ https://getdnsapi.net/
bedis9 9y ago
Unfortunately, this is not possible! The DNS in HAProxy relies on the internal task scheduler which is itself event-driven.

dcow 9y ago

I've been exploring DNS PTR/SRV records for seamless service discovery regardless of whether you have access to an authoritative DNS server configured for a given local domain or not (so things simply devolve to whether or not use use multicast or unicast to resolve SRV records). Basically DNS-SD. The idea of using SRV records in general seems foreign even in the ops realm. It's awesome to see HAProxy beginning to familiarize people with the concept of SRV records.

jlgaddis 9y ago
> The idea of using SRV records in general seems foreign even in the ops realm. I'm not suggesting you use it but (as for how it works in general) look into how Microsoft's Active Directory works. They've had the "service discovery" problem figured out for going on two decades, including preferences for "closer" servers, weighting, and so on. I'm no MS fan by any means but AD is one MS product that really is awesome.
- snuxoll 9y ago
  And how does MS do it? SRV records, that’s right.
  - jlgaddis 9y ago
    Yeah, exactly. That's why I suggested he look into how it works (since "the idea ... seems foreign" to him). I didn't mean to suggest that they had some revolutionary new unique way of handling service discovery or even that they were the first. I think maybe you misinterpreted or misunderstood my intentions.
    - snuxoll 9y ago
      It’s all good, I got what you meant :) LDAP and Kerberos of all implementation are probably the most common use case for SRV records, most people probably don’t even know it. Real shame, in many ways I wish A/AAAA records weren’t even a thing for the vast majority of services (HTTP, I’m looking at you!)

linkmotif 9y ago

Maybe there will there be a Kubernetes a Ingress Controller?? :)

aiharos 9y ago
Kubernetes Ingress Controllers for HAProxy already exist. A nice one is at https://github.com/jcmoraisjr/haproxy-ingress https://github.com/jcmoraisjr/haproxy-ingress Other examples of usage are spread around in the examples section of the ingress repo, like here https://github.com/kubernetes/ingress/tree/master/examples/deployment/haproxy https://github.com/kubernetes/ingress/tree/master/examples/d... The DNS-SD code will ship with HAProxy 1.8, and we will gladly help out by adding it to the above ingress controller after the release. A WIP of that code is up on github in case you’re interested. In fact, HAProxy is full of surprises and little known features (like the dynamic scaling via haproxy runtime api we already contributed to the controller) and we're happy to share them with everyone!

Whitespace 9y ago

Anyone here ever used SRV records for local development? I really like the idea of running a local DNS server with port information built in instead of a full-blown service discovery/routing/service mesh setup.

Cieplak 9y ago
Yes. Current setup runs a local DNS server which serves SRV records containing the local IP addresses and ports of services running inside FreeBSD jails. For instance, the payments service in one jail can communicate with the risk service running in another jail on the same host by sending an HTTP message to http://risk.local http://risk.local Currently using dnsmasq but would like to use erl-dns once I finish wrapping it with an HTTP admin API. https://github.com/aetrion/erl-dns https://github.com/aetrion/erl-dns
doublerebel 9y ago
Yes, try Consul. It's extremely easy to configure and run.
- bedis9 9y ago
  I do agree on this statement!!! I personally validated HAProxy SRV records with both Kubernetes and Consul and I was positively supersized of how simple is consul. (Well, it does not cover all the stuff kubernetes does, you may need nomad for this purpose).

doublerebel 9y ago

Interesting that HAProxy believes it is incumbent enough to charge for directly copying a feature Consul has had for years. When I added Consul to my stack 3 years ago HAProxy was already looking behind the times... I don't think they can catch up for any new installation.

wtarreau 9y ago
To be honest it's not a matter of duplicating, catching up or anything. It's just that when a significant number of our users ask for exactly the same stuff with similar and sometimes different justifications, it becomes quite hard to avoid adding it. And if they want it there, they very likely have their own very valid reasons. Baptiste (the DNS subsystem maintainer) has a significant amount of use cases in mind and could likely much better explain than me.
- bedis9 9y ago
  We first developped DNS in HAProxy for our AWS users who wanted HAProxy to follow-up a node when it is restarted (and it's IP is changed)... That was the very first request from both community and customers. After we released this feature, the community came back with some requirements regarding the ability to use DNS resolution to resolve all the servers (or a set of servers at list) of a backend. So we had to improve a lot the first dev we did to make this possible. Support for SRV record is just the last stone we put on top of many other devs :) We're quite proud of the result because it makes HAProxy able to scale up / down at run time without being reloaded and compatible with any service registry able to export a list of nodes delivering a same service through DNS. HAProxy can use multiple name for the resolution, pointing to different set of DNS servers, enforcing custom "hold" timers (to bypass server's TTL or negative TTLs in case of NX, etc...) and mix all of this with "old style hardcoded" servers in the backend... HAProxy is flexible :)
vidarh 9y ago
I don't understand what you mean here. Consul is not a load balancer. It's functionality is almost entirely orthogonal to Haproxy. If anything, a lot of people pair the two, but have so far often had to resort to cumbersome solutions for rewriting the Haproxy config to keep the set of backends in Haproxy updated. If anything, this will make Haproxy and Consul work much better together, given that Consul serves up SRV records for the services you register with it.

bitly 9y ago

Hi there. This is really awesome news and i am delighted to hear about this. Any change to get some directions how to test this functionality? So far I have currently 1.8-dev2 up and running but now I find no real references about how to try it .. and if available in this repo. (maybe i have to dig into the code ?) Thanks and all the best.

bitly 9y ago
http://git.haproxy.org/?p=haproxy.git;a=commitdiff;h=e2d03d2a43473a637491c0a8c9e10d5caed8e63b;hp=e70bc05b3a62ca84b5ce4440d340440f00411886 http://git.haproxy.org/?p=haproxy.git;a=commitdiff;h=e2d03d2... Thanks and sorry for the noise