4 ms·
So... how was that battery the cause of an https problem?
by mikeash 10y ago
So... how was that battery the cause of an https problem?
- duskwuff 10y agoNot the OP, but a guess: A lot of RAID cards will switch from writeback mode to writethrough when the battery is low, because they can't guarantee that the (battery-backed) cache will persist across a power failure. Writethrough mode makes writes significantly slower, which could cause problems higher up the stack.
- bogomipz 10y agoYep they will also go write through during a battery "learn cycle". The DELL PERC controllers are notorious for this. And there is poor visibility into this. You basically have to know about the awful MegaCli utility to gain any insight. What is the learn cycle? "The purpose of the learn cycle is to determine the condition of the battery. The learn cycle charges, fully discharges, and then recharges the battery in order to determine the condition and health of the battery. The battery full charge capacity degrades over time and a battery is deemed completely degraded when it can no longer hold a charge for 24 hours and must be replaced." Source: https://discuss.pivotal.io/hc/en-us/articles/204955498-FAQ-Greenplum-DCA-PERC-Battery https://discuss.pivotal.io/hc/en-us/articles/204955498-FAQ-G...
- Dylan16807 10y agoWhat kind of trash-tier reliability-increasing hardware stops working for a day every 90 days? I'm really surprised they didn't put two or three batteries into it.
- true_tuna 10y agoAll of them. You learn to disable learn cycle, most people learn that the hard way. I switched to flash backed cache for this reason (along with being sick of replacing batteries)
- true_tuna 10y agoThis is correct. I managed a large database fleet and can confirm the performance degradation.
- kpatrick 10y agoGuessing the battery died and the clock reset throwing off a https parameter. Happens to my daughter's tablet when she forgets to charge it and then she can't connect to https sites without resetting the clock.
- parennoob 10y agoWhy on earth is this answer downvoted? I'm pretty sure clock resets can cause HTTPS problems (even though they are unlikely to be in the OP's case, it being a server). And if you disagree, please explain...so that we can all learn and understand.
- viraptor 10y agoRaid battery has nothing to do with the clock. Even the motherboard battery affects the clock only when the machine is turned off. Normal systems connected to the internet will sync to NTP both at boot time and continuously afterwards anyway. Also the server time doesn't matter for TLS. It's the client that has to verify the certificate validity. (https://tools.ietf.org/html/rfc5246.html#section-7.4.1.2 https://tools.ietf.org/html/rfc5246.html#section-7.4.1.2 "Clocks are not required to be set correctly by the basic TLS protocol") So no, raid battery should have nothing to do with the system clock.
- parennoob 10y agoOkay, that's a good explanation of why this is wrong. Cool!
- shermanyo 10y agoThis has popped up a couple of times and I never got around to confirming either way. I always went to the trouble of syncing to rule it out before moving on. (sometimes annoying on an isolated test network) Thanks for taking the time to link the relevant section of the RFC. I'll remember this next time (or at least know where to check quickly ;) )
- a_t48 10y ago
- Diederich 10y agoA failed battery on a nicer RAID controller will usually cause its overall performance to drop quite a bit. When a program writes bytes to the file system, those bytes aren't guaranteed to arrive on the underlying, permanent storage. They're cached for some variable amount of time in the memory of the host operating system. In UNIX, calling fsync will force the OS to flush any outstanding writes to the underlying storage, which should mean they're safe. So far so good. Nicer RAID controllers have a lot of high speed memory memory, and they are basically their own operating system. When they receive a write from the host OS, and their battery is good, they lie to the host OS and say that the write went through all the way to disk/SSD, even though it probably didn't. This has the effect of making the fsync call return a lot more quickly. The bytes are still safe in this case because if there's a power loss of the whole server, there's enough battery power left to allow the controller to write through any data it lied about writing. When the battery goes bad, by default the RAID controller will cease its lying about when the bytes it receives are written all the way through, which has the effect of slowing things down. I can't say how lower IO performance caused the https problem in question without more details, but there are a lot of possible correlations. NB: For brevity, I simplified a lot of how this stuff works, so some of what I said above isn't 100% accurate as stated.