3 ms·

Great post! I thought about all of the companies in my list as having the exact same general problem, and I see now that's not true. Could you elaborate on wha

Great post! I thought about all of the companies in my list as having the exact same general problem, and I see now that's not true. Could you elaborate on what the problems are? They are probably interesting.

lsc 16y ago
I don't have any special knowledge of google or facebook. (I worked at Yahoo for a while as a sysadmin, but that was some time ago) I have maintained web applications from the hardware up, and I have run co-location and VPS businesses. But hosting hardware (or virtual servers) that other people run stuff on is a fundamentally different thing than running an application soup to nuts. Like I was saying, if you own the app, there are all sorts of nice failover things you can do that makes the failure of one physical bit of hardware (or even a whole data center) not as big of a deal. This means you can put less effort into keeping your servers and datacenters up, and just keep more spare capacity around instead. If you are renting hardware (or virtual hardware) for someone else to run an application on, transparently dealing with hardware failure becomes /much/ more difficult. I mean, especially with virtual hardware it's not entirely impossible to move people off a failed server, but all the methods I know of for doing so are extremely expensive, either in terms of infrastructure, licences, or complexity. (and usually all three) I mean, at yahoo, at one point I fucked up and took out a data center. Now, I was on pager, sleep deprived and generally not performing well, but with a few commands, I was able to redirect all traffic across the country to another data center. Sure, my mistake cost the company quite a lot in terms of additional latency, but it was better than being down, right? It would be nearly impossible to do the same with a co-location or VPS company.
lsc 16y ago
oh yeah. I completely didn't write about the problems with oil immersion. Well, I'm far from an expert on the matter, but the idea does come up from time to time, usually with some home built thing or another. Now, 'water cooling' comes up more often than full-on oil immersion, but usually you talk about using non-conductive coolant, so 'water cooling' in the worst case is oil immersion. I mean, you will occasionally have leaks. The primary perceived problem with cooling with a liquid is that leaks suddenly become a really big deal. That's what'd scare me as an operator. what happens if there's a leak? I mean, air cooling systems usually have enough redundancy that even big holes in air pipes aren't going to overheat anything, at least at the nicer data centers, and the chiller fluid is far enough away from the servers that in the case of a leak the worst case is that the data center smells like mold for a while. (yeah, I've worked in some smelly data centers. no fun. But it wasn't a threat to our servers. the A/C was redundant, and there were drains.) There are other problems, I think, with how the hydrodynamics and thermodynamics of oil differ from the hydrodynamics and thermodynamics of air, but really I know so little about that subject that I can't speak intelligibly about the problem. Also, those racks look big, like they hold a bunch of smaller servers. what happens when I've got a hardware problem with one server and I've got to pull it out? do the remaining servers run okay? how do I work on the dead hardware? can I set it up on a test bench? will the dust in the room get in and clog up the works? will I need to test it in some smaller tank of coolant? Then, after that, you have the same problem as any new and unique system. Standard parts. We weren't born yesterday, we've all seen (and most of us have been burnt by) blades, and we all know that if you put yourself in a situation where you can only buy parts from one vendor, you get fucked. So once you solve the problems of oil immersion, you then need to standardize the set up. But yeah, like I said, I'm no expert at oil immersion, so some of these may be wrong. However, these are the sorts of things operators worry about when the subject comes up. Also, the idea of using standard coolant in and coolant out plugs on every server, you know, water cooling style, so every server would need water blocks on every hot part rather than just submerging the damn thing is far more common.

lsc 16y ago

I don't have any special knowledge of google or facebook. (I worked at Yahoo for a while as a sysadmin, but that was some time ago) I have maintained web applications from the hardware up, and I have run co-location and VPS businesses. But hosting hardware (or virtual servers) that other people run stuff on is a fundamentally different thing than running an application soup to nuts. Like I was saying, if you own the app, there are all sorts of nice failover things you can do that makes the failure of one physical bit of hardware (or even a whole data center) not as big of a deal. This means you can put less effort into keeping your servers and datacenters up, and just keep more spare capacity around instead. If you are renting hardware (or virtual hardware) for someone else to run an application on, transparently dealing with hardware failure becomes /much/ more difficult. I mean, especially with virtual hardware it's not entirely impossible to move people off a failed server, but all the methods I know of for doing so are extremely expensive, either in terms of infrastructure, licences, or complexity. (and usually all three) I mean, at yahoo, at one point I fucked up and took out a data center. Now, I was on pager, sleep deprived and generally not performing well, but with a few commands, I was able to redirect all traffic across the country to another data center. Sure, my mistake cost the company quite a lot in terms of additional latency, but it was better than being down, right? It would be nearly impossible to do the same with a co-location or VPS company.

lsc 16y ago

oh yeah. I completely didn't write about the problems with oil immersion. Well, I'm far from an expert on the matter, but the idea does come up from time to time, usually with some home built thing or another. Now, 'water cooling' comes up more often than full-on oil immersion, but usually you talk about using non-conductive coolant, so 'water cooling' in the worst case is oil immersion. I mean, you will occasionally have leaks. The primary perceived problem with cooling with a liquid is that leaks suddenly become a really big deal. That's what'd scare me as an operator. what happens if there's a leak? I mean, air cooling systems usually have enough redundancy that even big holes in air pipes aren't going to overheat anything, at least at the nicer data centers, and the chiller fluid is far enough away from the servers that in the case of a leak the worst case is that the data center smells like mold for a while. (yeah, I've worked in some smelly data centers. no fun. But it wasn't a threat to our servers. the A/C was redundant, and there were drains.) There are other problems, I think, with how the hydrodynamics and thermodynamics of oil differ from the hydrodynamics and thermodynamics of air, but really I know so little about that subject that I can't speak intelligibly about the problem. Also, those racks look big, like they hold a bunch of smaller servers. what happens when I've got a hardware problem with one server and I've got to pull it out? do the remaining servers run okay? how do I work on the dead hardware? can I set it up on a test bench? will the dust in the room get in and clog up the works? will I need to test it in some smaller tank of coolant? Then, after that, you have the same problem as any new and unique system. Standard parts. We weren't born yesterday, we've all seen (and most of us have been burnt by) blades, and we all know that if you put yourself in a situation where you can only buy parts from one vendor, you get fucked. So once you solve the problems of oil immersion, you then need to standardize the set up. But yeah, like I said, I'm no expert at oil immersion, so some of these may be wrong. However, these are the sorts of things operators worry about when the subject comes up. Also, the idea of using standard coolant in and coolant out plugs on every server, you know, water cooling style, so every server would need water blocks on every hot part rather than just submerging the damn thing is far more common.