3 ms·
Ive seen people having to wake up way more often when running overarchitected systems that is beyond their ops capability vs having simple solution that require
by qaq 8y ago
Ive seen people having to wake up way more often when running overarchitected systems that is beyond their ops capability vs having simple solution that requires manual intervention a few times a year. So the proper solution would be what say for PG: HAProxy in HA configuration + Patroni (so ETCD). So from running 2 postgres instances you went to running 3 postgres instances a distributed database Patroni HAproxy in HA configuration so you need floating IP some heartbeat service etc.
- scarface74 8y agoOr I can just let AWS do the work and not worry about it running HAProxy, Patroni, etc.
- qaq 8y agoSure but at a very different price point.
- scarface74 8y agoAnd too often, developers (smaller companies) and ops (larger companies) time is considered free because they are salaried and are expected to work more than 40-45 hours per week. Yes, earlier in my career I would have accepted that, but I’ve successfully pushed back a few times and said I won’t babysit a process/server etc but I will design a method where it is auto healing, autoscaling, etc.
- matwood 8y agoAs someone who has been both the server baby sitter and implementor of auto-healing/scaling, it still puts a smile on my face when a server fails a health check, AWS kills it, starts another one, and moves the work over. All with zero downtime. I just see the emails in the morning that it happened. The other huge benefit of building this process is we can use dirt cheap spot instances with near abandon. For me, any infrastructure that I can't treat as cattle is on my hit list.
- qaq 8y agoWhen AWS is failing over RDS PG there is def nonetrivial downtime
- matwood 8y agoAnd? You think running your own database is going to have better uptime and scalability than something like Aurora? If something like DB HA is important, I think it's much easier in AWS for a company with a smaller engineering department.
- qaq 8y agoI am just pointing out that no downtime applies to failing over very specific services. In general AWS had pretty horrible track record on outages. A single top tier DC has way better uptime vs AWS Region.
- scarface74 8y agoAn AWS region or an AWS availability zone? You should always make everything multi-AZ whenever feasible.
- qaq 8y agoRegion. The multi Region thing is partial BS too if US East has a total failure there is not enough spare capacity to absorb all of US East compute in other Regions.
- falcolas 8y agoSadly, you'll still encounter pages in the middle of the night (well, if you implement the monitoring) when AWS services fail and you need to write up a support ticket. Granted, they're usually already aware of it, but it can still take tens of minutes to hours to resolve - time during which you are down.
- scarface74 8y agoIn the case of the parent poster - using AWS to manage load balancers and database failovers, how often would a failure be noticed by the end user? Yes, I know what happens when client apps/servers cache the DNS entry too long and don’t notice when a failover takes place - been there done that. And the quickest way to recover from a failure of servers that don’t store state is just to let autoscaling kill it and bring up another instance based on an AMI/startup scripts. I’m ruthless about not having servers as pets for anything that we have to manage ourselves. I tell our Devops guy that it is completely useless to keep an inventory of server names, IP addresses, etc. for anything that I’m responsible for.