4 ms·
This is pretty common knowledge if you have any experience working with kubernetes. Some less common knowledge: Do your pre-startup config with a shell script
by uberduper 4y ago
This is pretty common knowledge if you have any experience working with kubernetes.
Some less common knowledge:
Do your pre-startup config with a shell script and then `exec` your process.
If you need some sort of process manager, use supervisord and configure stdout/stderr capture and forwarding. Supervisord will do anything you need from init in a container.
If you're running a jvm in the container, set the dns cache ttl in the java security config.
Sometimes it's very useful to include a preStop hook that will end your process and then sleep for N seconds so that your process unbinds its listening socket and then for a period of time the pod will respond to incoming SYNs with a RST. So your upstream app doesn't sit there waiting on a timeout because for various reasons it still made new connection attempts despite the pod being removed from the service endpoints or w/e.
Fix your resolver configs. ndots. timeout. retry. Remove all the hosts search domains by setting `--resolv-conf=/dev/null` in your kubelet options. Then set dnsConfig.options in your pod spec. While you're at it, set `--allowed-unsafe-sysctls 'net.ipv4.tcp.keepalive*'` so you can configure your pods with sane values. Update pod spec with securityContext.sysctls to set the per pod values. Update the kubelet before applying that pod spec because lol if you don't... (your pods do not inherit these values from the host! So if you think you've set them, you're probably wrong.)
- azinman2 4y agoWhy aren’t these all just defaults?
- robertlagrant 4y agoThe DNS I suppose obeys the principle of least surprise. If I want to call the Twilio API I may be surprised at it not resolving on DNS by default.
- GauntletWizard 4y agoWriting good Lameducking is the single best trick for preserving SLO with Kubernetes. It's not that hard, but it is tricky, and testing it is the hardest part.
- chrsig 4y agoCould you define lameducking? I'm unfamiliar with the phrase in a tech context
- GauntletWizard 4y ago"Lame Duck" time[1] in politics is the time between losing a mandate (being voted out) and losing that position. Lame Duck time in a software process is the time between being told to shut down and shutting down. In many cases, you want to keep this time as small as possible. One valid solution is just to kill any requests that were ongoing, with the knowledge that a variety of error conditions would do the same. Alternatively, you can stop accepting new requests while giving time for the previous ones to complete; This is what we call lame-ducking. How it's accomplished differs. [1] https://en.wikipedia.org/wiki/Lame_duck_(politics) https://en.wikipedia.org/wiki/Lame_duck_(politics)