4 ms·

Stupid question. What was wrong with hot reloading the configuration file? It doesn’t drop requests, doesn’t seem to impact CPU and RAM by much, and it’s straig

Stupid question. What was wrong with hot reloading the configuration file? It doesn’t drop requests, doesn’t seem to impact CPU and RAM by much, and it’s straightforward to script around.

rogerdonut 7y ago
Why not both? The HAProxy Data Plane API does use hitless reloads -- however, it simplifies the management of the configuration through a modern REST API.
zaroth 7y ago
Can you continue to do this if you want to? But think about a system which is dynamically adding domains with TLS and SNI, or constantly adjusting backend workers. Or trying to automate a clean rolling upgrade. You want to quickly shed load, or add new TLS certificates, etc. If you have a process which is accepting these change config requests and controlling a file, it potentially has to worry about atomicity if several different things are happening at the same time — could one change blow away another if you script a “read config, update config, link new config, trigger reload, poll till complete” function without locks, one change could blow away another? Presumably the API would protect against this and allow a simpler function to push a given state change? But I agree that it’s likely over designed and not strictly necessary assuming the hot reload doesn’t have unanticipated side effects. E.g. What happens to stick tables, session state, etc.? It’s not like the REST API eliminates all these concerns too. For example, if you make a change via the API, presumably it is persisted across reloads, or there is at least a way to make it be?
- user5994461 7y ago
  The configuration file allows to reconfigure at once in an atomic manner. The configuration is also verified before reconfiguring and can be aborted if it's not runnable. HTTP offers none of this. No atomicity and no consistency of the full configuration. Want to edit a service or maybe alter some hosts? Better hope it's already setup as expected before editing and you're doing all the right calls in the right order and they all succeed. You will have to write the hell of a state machine to ensure to get to the expected state. The only sane use case for live-changes is enabling/disabling a server, which is necessary to perform rolling maintenance or blue-green deployment.
  - nickramirez 7y ago
    The HAProxy Data Plane API performs a validation check: it sends the -c flag to the HAProxy process before reloading to make sure that it is a valid configuration. If the configuration is invalid, the changes will not take effect and you will see errors in your console or log. With transactions, it's not quite stateless (as HTTP is). Transactions provide a way to make multiple changes and apply them with a single commit. The main benefit of an API is towards program-ability (is that a word?), in which configuring HAProxy can be controlled by "control plane" software, slick looking UIs, tools like Ansible, etc. in an automated way. You can also generate client-side code from the API, such as for Go/Python/[insert language] coding. It goes beyond the use case of "I'm a human who wants to control HAProxy manually or with templates".
spenczar5 7y ago
One thing could be that a configuration file requires filesystem access for the operator or tool doing the configuration. An HTTP API only requires network access. That can make it easier to modify if the process is (say) running inside a container, or on a fleet with locked-down SSH access.
toast0 7y ago
HAProxy doesn't let you hot reload the config file. It lets you load it in a new process and takeover the listen sockets. That new process doesn't get the state from the old process. That probably isn't a big deal for many uses, but for others, it's a bigger problem. If you're balancing traffic by session count, the new process doesn't know about the sessions in the old process (not sure if you can workaround this, it's not a problem I have, just something I'm aware of) If you use source port ranges, the new process doesn't know what the old process is still using, so you get extra logs about ports in use, and extra CPU spent on syscalls to find free ports. For me, on a busy system, this makes it even busier for quite a few minutes before it slows down to normal again. I haven't looked at this API yet though to know if it will help for me.

Stupid question. What was wrong with hot reloading the configuration file? It doesn’t drop requests, doesn’t seem to impact CPU and RAM by much, and it’s straightforward to script around.

rogerdonut 7y ago

Why not both? The HAProxy Data Plane API does use hitless reloads -- however, it simplifies the management of the configuration through a modern REST API.

zaroth 7y ago

Can you continue to do this if you want to? But think about a system which is dynamically adding domains with TLS and SNI, or constantly adjusting backend workers. Or trying to automate a clean rolling upgrade. You want to quickly shed load, or add new TLS certificates, etc. If you have a process which is accepting these change config requests and controlling a file, it potentially has to worry about atomicity if several different things are happening at the same time — could one change blow away another if you script a “read config, update config, link new config, trigger reload, poll till complete” function without locks, one change could blow away another? Presumably the API would protect against this and allow a simpler function to push a given state change? But I agree that it’s likely over designed and not strictly necessary assuming the hot reload doesn’t have unanticipated side effects. E.g. What happens to stick tables, session state, etc.? It’s not like the REST API eliminates all these concerns too. For example, if you make a change via the API, presumably it is persisted across reloads, or there is at least a way to make it be?

user5994461 7y ago
The configuration file allows to reconfigure at once in an atomic manner. The configuration is also verified before reconfiguring and can be aborted if it's not runnable. HTTP offers none of this. No atomicity and no consistency of the full configuration. Want to edit a service or maybe alter some hosts? Better hope it's already setup as expected before editing and you're doing all the right calls in the right order and they all succeed. You will have to write the hell of a state machine to ensure to get to the expected state. The only sane use case for live-changes is enabling/disabling a server, which is necessary to perform rolling maintenance or blue-green deployment.
- nickramirez 7y ago
  The HAProxy Data Plane API performs a validation check: it sends the -c flag to the HAProxy process before reloading to make sure that it is a valid configuration. If the configuration is invalid, the changes will not take effect and you will see errors in your console or log. With transactions, it's not quite stateless (as HTTP is). Transactions provide a way to make multiple changes and apply them with a single commit. The main benefit of an API is towards program-ability (is that a word?), in which configuring HAProxy can be controlled by "control plane" software, slick looking UIs, tools like Ansible, etc. in an automated way. You can also generate client-side code from the API, such as for Go/Python/[insert language] coding. It goes beyond the use case of "I'm a human who wants to control HAProxy manually or with templates".

spenczar5 7y ago

One thing could be that a configuration file requires filesystem access for the operator or tool doing the configuration. An HTTP API only requires network access. That can make it easier to modify if the process is (say) running inside a container, or on a fleet with locked-down SSH access.

toast0 7y ago

HAProxy doesn't let you hot reload the config file. It lets you load it in a new process and takeover the listen sockets. That new process doesn't get the state from the old process. That probably isn't a big deal for many uses, but for others, it's a bigger problem. If you're balancing traffic by session count, the new process doesn't know about the sessions in the old process (not sure if you can workaround this, it's not a problem I have, just something I'm aware of) If you use source port ranges, the new process doesn't know what the old process is still using, so you get extra logs about ports in use, and extra CPU spent on syscalls to find free ports. For me, on a busy system, this makes it even busier for quite a few minutes before it slows down to normal again. I haven't looked at this API yet though to know if it will help for me.