3 ms·

The application server was actually a shared box with a bunch of my dotnet core applications. I used a 3 node cluster database because I've had single node fail

The application server was actually a shared box with a bunch of my dotnet core applications. I used a 3 node cluster database because I've had single node failures on mongo on $5 droplets, so I did 2 $10 droplets for the slaves and a $20 for the master, it only failed over once. Then I had the primary forwarding application that was a $10 droplet, with a secondary $5 droplet that was just standing by until a failed heartbeat (every 10 seconds), then my floating IP would switch to the backup. It was overly complicated for robustness. I tested multiple database crashes, servers going down in one zone, and it would always fail over just fine, because application code would connect to a slave if it could not connect to the master, so forwards would work, unless there were multiple outages in multiple data centers. Now switching the floating IP to the backup forwarder did take about 5 seconds, and I could have made that better by putting a proxy in front, but I'm just moving the point of failure up a level, and it was never needed anyway.

chrismorgan 6y ago
Ah, MongoDB. I was guessing something more like PostgreSQL, which is decidedly more robust. I haven’t used MongoDB at all for something like six years now, but the impression I get from the odd thing I hear about it is that although it’s better than it used to be it’s still decidedly less reliable than the likes of PostgreSQL. It sounds like you’ve tested the simple failure modes, machines dropping off altogether, which is good; but I’d still be sceptical about the system’s robustness, because those aren’t the interesting failure modes, and the interesting failure modes are the ones that give the most trouble. Systems are easy when a node is there or not there, but when a node becomes unreliable (e.g. overloaded or network troubles of a variety of kinds), that’s when the real trouble happens, and when it’s most common that measures that were supposed to make things available actually make them unavailable. (This stuff is hard.) That’s why avoiding a distributed architecture until you actually need it can be well worthwhile.
- jermaustin1 6y ago
  If I were to do this all over again, I don't think I would do it this way at all. I would use something like Caddy which manages it's own config and has an API for live updates. SQLite for the user management and administration. I would possibly have 2 Caddy nodes though, with a LB between them (instead of my potentially flawed heartbeat floating IP mechanism). The application would do an atomic update to both Caddy servers, if either failed, it would roll it back. But you are still looking at around $30-40/mo for hosting, not saving much money, but making it much simpler and saving headaches.

chrismorgan 6y ago

Ah, MongoDB. I was guessing something more like PostgreSQL, which is decidedly more robust. I haven’t used MongoDB at all for something like six years now, but the impression I get from the odd thing I hear about it is that although it’s better than it used to be it’s still decidedly less reliable than the likes of PostgreSQL. It sounds like you’ve tested the simple failure modes, machines dropping off altogether, which is good; but I’d still be sceptical about the system’s robustness, because those aren’t the interesting failure modes, and the interesting failure modes are the ones that give the most trouble. Systems are easy when a node is there or not there, but when a node becomes unreliable (e.g. overloaded or network troubles of a variety of kinds), that’s when the real trouble happens, and when it’s most common that measures that were supposed to make things available actually make them unavailable. (This stuff is hard.) That’s why avoiding a distributed architecture until you actually need it can be well worthwhile.

jermaustin1 6y ago
If I were to do this all over again, I don't think I would do it this way at all. I would use something like Caddy which manages it's own config and has an API for live updates. SQLite for the user management and administration. I would possibly have 2 Caddy nodes though, with a LB between them (instead of my potentially flawed heartbeat floating IP mechanism). The application would do an atomic update to both Caddy servers, if either failed, it would roll it back. But you are still looking at around $30-40/mo for hosting, not saving much money, but making it much simpler and saving headaches.