4 ms·
I would argue that features such as having explicit dependencies are absolutely required for 99.9% uptime. Building systems resilient to failures also means bei
by bad_user 11y ago
I would argue that features such as having explicit dependencies are absolutely required for 99.9% uptime. Building systems resilient to failures also means being able to predictably spin up new instances that you can trust.
I'm also a developer and for example one problem I had with my servers on AWS was that my process sometimes happened to start before my volume meant for logs was mounted, which meant that all logs were going to my primary volume, with the server crashing at some point due to out of space errors. Another time I was using a local database meant for local caching and again, sometimes it would fully initialize only after my main process was started.
And because I was using ELB with auto-scaling, it meant that these incorrectly started servers would be connected to the load-balancer, because the main process was suddenly responding to /ping, even though it wasn't ready yet. And then it would receive thousands of requests per second per server, at a moment in time in which the cluster badly needed fresh instances, which led to an unstable cluster. But only sometimes ;-)