10 ms·

If you're using Consul for web services, I really recommend the Traefik web server: https://traefik.io https://traefik.io Traefik replaces Nginx: it's the reve

If you're using Consul for web services, I really recommend the Traefik web server: https://traefik.io https://traefik.io Traefik replaces Nginx: it's the reverse proxy that maps the incoming requests to your various services, which are advertising on some arbitrary localhost port. The amazing thing is that Traefik integrates with Consul: you only need to point it to your Consul endpoint, and it can automatically publish your services! You can also do other dynamic configuration of Traefik, e.g. by publishing a service via a REST API, via Kubernetes, etc. I've struggled for years to get Nginx configured correctly, and it's been frustrating to have no alternative. In Nginx, dynamic binding is a premium feature. In the free version, you have to rewrite the config and restart the service. That's not fun if you expect services to come and go as part of your natural life-cycle. Traefik's still pretty young. Notably, the docs are shit. But the application is well-designed, and it compiles as a single self-contained binary (as it's written in Go).

matthewmacleod 8y ago
In the free version, you have to rewrite the config and restart the service. That's not fun if you expect services to come and go as part of your natural life-cycle If that's happening super-frequently I could see the problem, but in practice I'd expect you to use `consul-template` and issue a `reload` to nginx to make it reload configurations with no downtime. This is the solution I've used and it works pretty well.
- tacticus 8y ago
  Old workers stay around until all the connections for them or their grace period has expired. So i hope you're not too close to the resource limit to handle many generations of workers sticking around. also if you have many services behind a single port with path based routing and do a deploy the connection will stick around on the old workers until everything dies. It's super fun with haproxy :(
  - oblio 8y ago
    Kind of bad, but what's the alternative? Cut everyone off on reload?
    - tacticus 8y ago
      Have a better proxy that doesn't need to create a new generation of workers for config changes. Like traefik or envoy or nginx when you pay for it or use the lua shit to do backend discovery (haproxy also has lua shit iirc)
      - pas 8y ago
        Could you elaborate on what's wrong with using lua with ningx, especially as used by kong?
        timdumol 8y ago
        I think GP means Lua "shit" in a nonderogative manner (synonymous to Lua stuff).
    - pas 8y ago
      kong. uses lua. has a wonderful API.
- poooogles 8y ago
  >This is the solution I've used and it works pretty well. Just make sure consul-template never writes bad config as that will stop everything reloading. That's bit me a couple of times.
  - dewey 8y ago
    Can't you just run a nginx -t to run the config test and see if it passes before you reload?
    - poooogles 8y ago
      nginx does that already. The problem is it'll stop you being able to reload configuration on other bits of config. Once there's a config error nothing can reload. Limited blast radius don't really apply when it comes to nginx config errors (which given its origins is understandable). One error in one bit of config and it all stops.
- swills 8y ago
  You can also use the consul k-v store as your source of traefik config and then update the config in consul and it updates in traefik instantly without any restart.
cremp 8y ago
We tried Traefik, but ultimately went with Fabio, which also integrates with Consul nicely; plus it does tcp based stuff for unencrypted gRPC services we shuffle around (which at the time, traefik didn't have.)
- garettmd 8y ago
  How production-ready is Fabio? It looks interesting but I can't tell how mature of a product it is
  - cremp 8y ago
    It's okay for our use case. Honestly, we were just wanting to get a load balancer up and working with consul, and we had grpc services and internal http apps to balance. I'm not going to lie; it works and does failover correctly, quickly, and plays nice with consul. Not to mention, the name brings back memories... If I had my way though, we would use HAProxy, but that's just because there is more options, ways to properly route, and more battle-tested.
hellcow 8y ago
Traefik isn't worth the trouble. We used it in production over the course of 2 years, and it was consistently our only source of downtime due to rediculous bugs: not closing file descriptors, breaking changes, and silent failures with no log output, panic or exit even with debug logs enabled. As you mentioned, the docs are terrible. What makes that worse are the undocumented breaking changes between each release. They don't even pretend to follow semver, so v1.5 broke v1.4, and v1.6 broke v1.5. Each update you pray that it doesn't take your whole setup down. If anything goes wrong, since nothing is documented and there's often no logs explaining what went wrong, you might be down for an extended period while you make 100 best-guess changes to the config that worked in staging, but for whatever reason isn't working in production. May the odds be ever in your favor. Last I checked, Traefik was 988,000 (!!!) lines of code. That's 20x the size of my very complex web application. I replaced it with 500 lines of go providing all the essential features for me. Higher reliability, way fewer bugs, no breaking changes.
- seeekr 8y ago
  Re LOC: Found the 988k LOC number hard to believe, so I checked. If you count the vendored dependencies, then it's indeed a lot, about 1M LOC. Traefik itself though is comparatively a lot less at about 60k lines of Go code with another 8k or so of scripts, config etc.
- syllogism 8y ago
  Thanks for the warnings! We don't have it bearing much load yet, so we haven't encountered those things.
- 1_player 8y ago
  Any suggestion? I'm stuck on this problem right now at work, don't like Traefik at all for the same reason as you, liked Docker Flow Proxy but it's not configurable enough and, similarly to Traefik, the documentation is quite bad if you need to do something a little more complicated (like, IP access control!). Fabio might be a good alternative, but we don't need Consul right now, so I don't want to have to manage yet another puzzle piece. Should I go with nginx?
vidarh 8y ago
With nginx, you can do dynamic binding in the free version in at least two ways: 1. If you "just" want to map a hostname, to a private IP, you can assign the hostname to a variable, and use the variable in your backend config. This works because Nginx resolves static addresses at start, but resolves names pulled from variables at runtime. 2. If you need to map ports as well, you can use a Lua or Mruby script. E.g. I have a blog post on doing it with mruby here [1], and it's run production sites for a couple of years. This options lets you integrate against pretty much whatever you want. It's extra hassle, though, of course. [1] http://hokstadconsulting.com/nginx/mruby-virtualhosts http://hokstadconsulting.com/nginx/mruby-virtualhosts
- jorge-fundido 8y ago
  For those interested in a lua example, from this article [1], I found & cherry-picked bits from here [2]. The key ingredients (to aid searching) are init_worker_by_lua_block & balancer_by_lua_block. [1] https://medium.com/@sigil66/dynamic-nginx-upstreams-from-consul-via-lua-nginx-module-2bebc935989b https://medium.com/@sigil66/dynamic-nginx-upstreams-from-con... [2] https://github.com/sigil66/nginx-consul-cookbook https://github.com/sigil66/nginx-consul-cookbook
sagichmal 8y ago
I strongly advise everyone to avoid Traefik. In addition to all the reasons enumerated by hellcow, the code quality is simply abysmal.
- emilevauge 8y ago
  Could you elaborate? I would never say that the code is perfect, like any other "big" open source project, but we follow a pretty strict review process on each PR (3 LGTM from maintainers). I'm curious to know why you are so negative.

matthewmacleod 8y ago

In the free version, you have to rewrite the config and restart the service. That's not fun if you expect services to come and go as part of your natural life-cycle If that's happening super-frequently I could see the problem, but in practice I'd expect you to use `consul-template` and issue a `reload` to nginx to make it reload configurations with no downtime. This is the solution I've used and it works pretty well.

tacticus 8y ago
Old workers stay around until all the connections for them or their grace period has expired. So i hope you're not too close to the resource limit to handle many generations of workers sticking around. also if you have many services behind a single port with path based routing and do a deploy the connection will stick around on the old workers until everything dies. It's super fun with haproxy :(
- oblio 8y ago
  Kind of bad, but what's the alternative? Cut everyone off on reload?
  - tacticus 8y ago
    Have a better proxy that doesn't need to create a new generation of workers for config changes. Like traefik or envoy or nginx when you pay for it or use the lua shit to do backend discovery (haproxy also has lua shit iirc)
    - pas 8y ago
      Could you elaborate on what's wrong with using lua with ningx, especially as used by kong?
      - timdumol 8y ago
        I think GP means Lua "shit" in a nonderogative manner (synonymous to Lua stuff).
  - pas 8y ago
    kong. uses lua. has a wonderful API.
poooogles 8y ago
>This is the solution I've used and it works pretty well. Just make sure consul-template never writes bad config as that will stop everything reloading. That's bit me a couple of times.
- dewey 8y ago
  Can't you just run a nginx -t to run the config test and see if it passes before you reload?
  - poooogles 8y ago
    nginx does that already. The problem is it'll stop you being able to reload configuration on other bits of config. Once there's a config error nothing can reload. Limited blast radius don't really apply when it comes to nginx config errors (which given its origins is understandable). One error in one bit of config and it all stops.
swills 8y ago
You can also use the consul k-v store as your source of traefik config and then update the config in consul and it updates in traefik instantly without any restart.

cremp 8y ago

We tried Traefik, but ultimately went with Fabio, which also integrates with Consul nicely; plus it does tcp based stuff for unencrypted gRPC services we shuffle around (which at the time, traefik didn't have.)

garettmd 8y ago
How production-ready is Fabio? It looks interesting but I can't tell how mature of a product it is
- cremp 8y ago
  It's okay for our use case. Honestly, we were just wanting to get a load balancer up and working with consul, and we had grpc services and internal http apps to balance. I'm not going to lie; it works and does failover correctly, quickly, and plays nice with consul. Not to mention, the name brings back memories... If I had my way though, we would use HAProxy, but that's just because there is more options, ways to properly route, and more battle-tested.

hellcow 8y ago

Traefik isn't worth the trouble. We used it in production over the course of 2 years, and it was consistently our only source of downtime due to rediculous bugs: not closing file descriptors, breaking changes, and silent failures with no log output, panic or exit even with debug logs enabled. As you mentioned, the docs are terrible. What makes that worse are the undocumented breaking changes between each release. They don't even pretend to follow semver, so v1.5 broke v1.4, and v1.6 broke v1.5. Each update you pray that it doesn't take your whole setup down. If anything goes wrong, since nothing is documented and there's often no logs explaining what went wrong, you might be down for an extended period while you make 100 best-guess changes to the config that worked in staging, but for whatever reason isn't working in production. May the odds be ever in your favor. Last I checked, Traefik was 988,000 (!!!) lines of code. That's 20x the size of my very complex web application. I replaced it with 500 lines of go providing all the essential features for me. Higher reliability, way fewer bugs, no breaking changes.

seeekr 8y ago
Re LOC: Found the 988k LOC number hard to believe, so I checked. If you count the vendored dependencies, then it's indeed a lot, about 1M LOC. Traefik itself though is comparatively a lot less at about 60k lines of Go code with another 8k or so of scripts, config etc.
syllogism 8y ago
Thanks for the warnings! We don't have it bearing much load yet, so we haven't encountered those things.
1_player 8y ago
Any suggestion? I'm stuck on this problem right now at work, don't like Traefik at all for the same reason as you, liked Docker Flow Proxy but it's not configurable enough and, similarly to Traefik, the documentation is quite bad if you need to do something a little more complicated (like, IP access control!). Fabio might be a good alternative, but we don't need Consul right now, so I don't want to have to manage yet another puzzle piece. Should I go with nginx?

vidarh 8y ago

With nginx, you can do dynamic binding in the free version in at least two ways: 1. If you "just" want to map a hostname, to a private IP, you can assign the hostname to a variable, and use the variable in your backend config. This works because Nginx resolves static addresses at start, but resolves names pulled from variables at runtime. 2. If you need to map ports as well, you can use a Lua or Mruby script. E.g. I have a blog post on doing it with mruby here [1], and it's run production sites for a couple of years. This options lets you integrate against pretty much whatever you want. It's extra hassle, though, of course. [1] http://hokstadconsulting.com/nginx/mruby-virtualhosts http://hokstadconsulting.com/nginx/mruby-virtualhosts

jorge-fundido 8y ago
For those interested in a lua example, from this article [1], I found & cherry-picked bits from here [2]. The key ingredients (to aid searching) are init_worker_by_lua_block & balancer_by_lua_block. [1] https://medium.com/@sigil66/dynamic-nginx-upstreams-from-consul-via-lua-nginx-module-2bebc935989b https://medium.com/@sigil66/dynamic-nginx-upstreams-from-con... [2] https://github.com/sigil66/nginx-consul-cookbook https://github.com/sigil66/nginx-consul-cookbook

sagichmal 8y ago

I strongly advise everyone to avoid Traefik. In addition to all the reasons enumerated by hellcow, the code quality is simply abysmal.

emilevauge 8y ago
Could you elaborate? I would never say that the code is perfect, like any other "big" open source project, but we follow a pretty strict review process on each PR (3 LGTM from maintainers). I'm curious to know why you are so negative.