4 ms·
Agreed. Having done Akka.NET Remote/Cluster setups in prod that survived multiple 'new to the org' categories of DC Failures at their level of scale/capacity [
by to11mtm 2y ago
Agreed.
Having done Akka.NET Remote/Cluster setups in prod that survived multiple 'new to the org' categories of DC Failures at their level of scale/capacity [0] there's a lot to account for if you want to keep everything happy and visible [1][2][3]
[0] - Cut fiber between DCs, Rack failures due to IO-ish type issues, bad switches... at least 2 out of 3.
[1] - The upshot was we were able to survive all of the scenarios in at worst a degraded state, Once or twice we needed a restart.
[2] - We also had enough metrics going on that we could detect DC/server outages about as quickly as whoever actually was monitoring the failing subsystem.
[3] - But here's the funny rub. An APM tool was the Achilles heel for both our Akka Links, as well as our SQLServer connections. Once they installed an 'agent' we more frequently had to do a 'full cycle' to clean things up after an outage, or even an MSSQL Server reboot. After I left the shop I got confirmation that yes, the APM module was the problem.
- toast0 2y ago> We also had enough metrics going on that we could detect DC/server outages about as quickly as whoever actually was monitoring the failing subsystem. Yeah, my Erlang clustering experience was that we (the customer) were the monitoring system for the DC/managed hosting provider. Although, by the time we left there, they would have outage notifications before we put in tickets.