4 ms·

While I found this article very interesting, I feel like something is missing here. So linking this issues to a Cisco bug is very interesting, that dropping co

While I found this article very interesting, I feel like something is missing here. So linking this issues to a Cisco bug is very interesting, that dropping connections would cause the application to lock up / crash, while all the connections to the database were dead. My question is why would the application lock up and the servers would crash? I don't see it very often, but when striving for high availability and strong resiliency (which isn't reasonable for everyone), issues need to be looked at in great detail. So I would be trying to look at the second side of the story, which is why was there crashes encountered under these circumstances, and are there other plausible triggers that could cause a similar set of circumstances. Disabling timestamps does avoid the Cisco bug, but a similar set of triggers could be encountered anytime the VPN connection dropped, or if the firewall failed over without the state tables in sync, or any number of other network conditions. And don't take me wrong, I don't know if the OP did this, but based on the article, I would lean towards disabling timestamps as a workaround, and this might still be an indicator that something in the app isn't behaving correctly when the database is unavailable.

askldjd 9y ago
You are dead on. We do have a bug where we are not recovering the Oracle connectivity correctly. It is on our radar to address the issue. https://github.com/department-of-veterans-affairs/caseflow-monitor/issues/15 https://github.com/department-of-veterans-affairs/caseflow-m... However, There is actually another 50% of the story that I never posted. VACOLS is a really old Oracle DB (from the 80s) that is out of our control. Somehow, it has a "feature" where you can only make one TCP connection to it every 2-3 second. So if we lose connection to the database, it will take many seconds to recover. At that point, our ELB health-check would've fired and restarted our EC2 instances. This is why recoverability of the database connection is not an immediate priority. Here's how we preallocate the VACOLS connection pool to workaround this throttling feature. https://github.com/department-of-veterans-affairs/caseflow/blob/master/config/initializers/warmup_vacols.rb https://github.com/department-of-veterans-affairs/caseflow/b... The infrastructure we operate in are very challenging (and interesting) because of legacy systems. That's why common sense engineering often may not apply in USDS.
- kevin_nisbet 9y ago
  I also bet, those challenging legacy systems in many case are way better built than what "modern" systems would provide. Sure there will be whacky things to work around, but I've seen my share of whacky engineering in brand new systems too. Common sense engineering seems to be few and far between these day's. Kudos on having something interesting to work on.
rasz 9y ago
>dropping connections would cause the application to lock up / crash plenty of programs/libraries do that, for example ffmpeg, and by extension MPlayer. It so happens Google/YT has anti resource starvation measures in place on their content servers in case someone opens TCP session and keeps it paused with ZeroWindow packets for over 3 minutes. Guess how MPlayer handles/signals pausing a stream :(

askldjd 9y ago

You are dead on. We do have a bug where we are not recovering the Oracle connectivity correctly. It is on our radar to address the issue. https://github.com/department-of-veterans-affairs/caseflow-monitor/issues/15 https://github.com/department-of-veterans-affairs/caseflow-m... However, There is actually another 50% of the story that I never posted. VACOLS is a really old Oracle DB (from the 80s) that is out of our control. Somehow, it has a "feature" where you can only make one TCP connection to it every 2-3 second. So if we lose connection to the database, it will take many seconds to recover. At that point, our ELB health-check would've fired and restarted our EC2 instances. This is why recoverability of the database connection is not an immediate priority. Here's how we preallocate the VACOLS connection pool to workaround this throttling feature. https://github.com/department-of-veterans-affairs/caseflow/blob/master/config/initializers/warmup_vacols.rb https://github.com/department-of-veterans-affairs/caseflow/b... The infrastructure we operate in are very challenging (and interesting) because of legacy systems. That's why common sense engineering often may not apply in USDS.

kevin_nisbet 9y ago
I also bet, those challenging legacy systems in many case are way better built than what "modern" systems would provide. Sure there will be whacky things to work around, but I've seen my share of whacky engineering in brand new systems too. Common sense engineering seems to be few and far between these day's. Kudos on having something interesting to work on.

rasz 9y ago

>dropping connections would cause the application to lock up / crash plenty of programs/libraries do that, for example ffmpeg, and by extension MPlayer. It so happens Google/YT has anti resource starvation measures in place on their content servers in case someone opens TCP session and keeps it paused with ZeroWindow packets for over 3 minutes. Guess how MPlayer handles/signals pausing a stream :(