7 ms·

Let me echo this as someone who once was responsible for HPC computing in a research intensive public university. Most career academics have NO IDEA how much e

Let me echo this as someone who once was responsible for HPC computing in a research intensive public university. Most career academics have NO IDEA how much enterprise computing infrastructure costs. If a 1 terabyte USB hard drive is $40 at Costco we (university IT) must be getting a much better deal than that. Take this argument and apply it to any aspect of HPC computing and that's what you're fighting against. The closet with racks of gear and no cooling is another fond memory. Don't forget the AC terminal strips that power the whole thing, sourced from the local dollar store.

0xbadcafebee 4y ago
I remember the first time a server caught fire in the closet we kept the rack in. Backups were kept on a server right below the one on fire. But, y'know, we saved money.
- eastbound 4y ago
  Don’t worry, we do incremental backups during weekdays and a full backup on Sunday. We use 2 tapes only, so one is always outside of the building. But you know, we saved money.
  - deleted 4y ago
    [deleted]
  - deleted 4y ago
    [deleted]
  - bilbo0s 4y ago
    A bit off topic, but I gotta say you guys are a riot! If there was a comedy tour for IT/Programmer types, I'd pay to see you guys in it. Best thing about your stuff is that it's literally all funny precisely because it's all true.
    - deleted 4y ago
      [deleted]
    - geoduck14 4y ago
      I don't know why you are being down voted. I'd watch a Netflix special on this stuff
  - Proven 4y ago
  - treeman79 4y ago
    We had a million dollars worth of hardware installed in a closet. It had a portable AC hooked up that needed it’s water bin changed every so often. Well I was in the middle of that. When the Director decided to show off the new security doors. So he closed the room up. Then found out that new security doors didn’t work. I find out as I’m coming back to turn AC back on. Room will get hot really fast. We get office Security to unlock door. He says he doesn’t have authority. His supervisor will be by later in the day. Completely deadpan, and in front of several VPs of a forth one 50. I turn to guy to my right who lived nearby. “Go home and get your chainsaw” We were quickly let in. Also got fast approval to install proper cooling.
    - deleted 4y ago
      [deleted]
- pbronez 4y ago
  This is my fear about my homelab lol Fire extinguisher nearby, smart temp sensors, but still...
  - W-Stool 4y ago
    What are you using for a homelab priced temperature sensor?
    - xani_ 4y ago
      Homelab-priced sensor is the temp sensor in your server, it's free! Actual servers have a bunch, usually have one at intake, "random old PC" servers can use motherboard temp as rough proxy for environment temp. Hell, even in DC you can look at temperatures and see in front of which server technican was standing just by those sensors. Second cheapest would be USB-to-1wire module + some DS18B20 1-sire sensors. Easy hobby job to make. They also come with unique ID which means if you put it in TSDB by that ID it doesn't matter where you plug those sensors.
  - rovr138 4y ago
    oh, nice idea with temp sensor. I have extinguishers all over the house, but hadn't considered a temperature sensor set to send alerts. Do you have any recommendations?
  - xani_ 4y ago
    No more or less chance that really anything else connected to power to catch on fire. Just make sure RCD works. Hell, some weeks ago my oven decided the lower heater's line is now connected to ground and blows fuses... Our 8 racks in DC only had single event of something blowing (power supply) and aside of smell and fuse blowing nothing really happened Servers are essentially metal boxes with a bit of glass-reinforced epoxy and some plastic inside so there is limited amount of stuff that can burn. UPS is probably bigger problem For example the OVH datacenter fire was more of "stuff around servers was flammable" (they had wooden ceilings for some reason...) rather than "just" servers.
    - 4y ago
- walnutclosefarm 4y ago
  I never saw an actual fire, but I we did see smoke, in a closet, on a floor that was 90% patient care, but happened to have a research area as well because the research needed access to expensive radiography equipment. The close was literally stuffed with what amounted to gaming machines, purchased with grant money through an importer, directly from China. The guy who set them up was smart enough to put them all behind a little firewall, so the enterprise network couldn't see them. It was a (literally) hot mess, both infrastructure and security-wise.
bluedino 4y ago
It's kind of funny around this time of year when some researchers have $10,000 in their budget they need to spend, and they want to 'gift' us with some GPU's.
- davidmr 4y ago
  That was definitely one of the weirdest things of working in academia IT: “hey. Can you buy me a workstation that’s as close to $6,328.45 as it is possible to get, and can you do it by 4pm?”
  - kjs3 4y ago
    Same thing happens in the government sector here (US). If you don't spend all of the budget you requested last year, you might not get it next year. There is an entire ecosystem of bottom-feeder GSA companies that apparently exist to spend year-end money that would otherwise go to 'waste'.
systemvoltage 4y ago
I am dealing with the exact opposite problem: "Oh you mean, we should leave the EC2 instance running 24/7??? No way, that would be too expensive"... to which I need to respond "No, it would be like $15/month. Trivial, stop worrying about costs in EC2 and S3, we're like 7 people here with 3 GB of data." I deal with Scientists that think AWS is some sort of a massively expensive enterprise thing. I can be, but not for the use case they're going to be embarking on. Our budget is $7M spanning 4 years.
- capableweb 4y ago
  > think AWS is some sort of a massively expensive enterprise thing Compared to using dedicated instances with way cheaper bandwidth, storage and compute power, it might as well be. Cloud makes sense when you have to scale up/down very quickly, or you'd be losing money fast. But most don't suffer from this problem.
  - pwinnski 4y ago
    At $15/month, it would take longer than a typical school year to beat.
- gonzo41 4y ago
  Don't say the budget outloud near AWS. They'll find a way to help you spend it.
  - systemvoltage 4y ago
    Hahaha, may be I need to just go into the AWS ether and start yakking big words like "Elastic Kubernetes Service" to confuse the scientists and get my aws fix. These people are too stingy. I want some shit running in AWS, what good is this admin IAM role.
  - absolutelynobo 4y ago
    GP has already lost that battle.
  - 4y ago
funstuff007 4y ago
I cannot wait for Ampere servers to become widespread. Most of the power and cooling issues will go away.
- getcrunk 4y ago
  sarcasm?
absolutelynobo 4y ago
> Don't forget the AC terminal strips that power the whole thing, sourced from the local dollar store. Love how you’ve fuzzed the root cause to make it seem like the “dollar store strip” is the problem and not that it was plugged into an overloaded outlet or run inside a closet at significantly elevated temperatures, leading to plastic melting and wires shorting. Always helps to keep the “magic” a secret so the rubes have to keep us wizards employed, right?
morbia 4y ago
Your comment about cooling reminded me of a fun anecdote from my time in academia. My PhD involved quite a lot of high performance computing simulations. We kept running into problems where on warm days, all our jobs would get killed pretty consistently. Our IT guy noticed a pattern where the temps would be perfectly normal, and then suddenly out of nowhere go through the roof, triggering a thermal shutdown of the racks. In the end our IT guy camped out in the server room on a hot day to watch what was happening. The Astrophysics' cabinet were directly infront of ours, and they had jerry rigged their cabinet door so that when it got too hot, it would swing open and their hot air would be blown all over the neighbouring cabinets...

0xbadcafebee 4y ago

I remember the first time a server caught fire in the closet we kept the rack in. Backups were kept on a server right below the one on fire. But, y'know, we saved money.

eastbound 4y ago
Don’t worry, we do incremental backups during weekdays and a full backup on Sunday. We use 2 tapes only, so one is always outside of the building. But you know, we saved money.
- deleted 4y ago
  [deleted]
- deleted 4y ago
  [deleted]
- bilbo0s 4y ago
  A bit off topic, but I gotta say you guys are a riot! If there was a comedy tour for IT/Programmer types, I'd pay to see you guys in it. Best thing about your stuff is that it's literally all funny precisely because it's all true.
  - deleted 4y ago
    [deleted]
  - geoduck14 4y ago
    I don't know why you are being down voted. I'd watch a Netflix special on this stuff
- Proven 4y ago
- treeman79 4y ago
  We had a million dollars worth of hardware installed in a closet. It had a portable AC hooked up that needed it’s water bin changed every so often. Well I was in the middle of that. When the Director decided to show off the new security doors. So he closed the room up. Then found out that new security doors didn’t work. I find out as I’m coming back to turn AC back on. Room will get hot really fast. We get office Security to unlock door. He says he doesn’t have authority. His supervisor will be by later in the day. Completely deadpan, and in front of several VPs of a forth one 50. I turn to guy to my right who lived nearby. “Go home and get your chainsaw” We were quickly let in. Also got fast approval to install proper cooling.
  - deleted 4y ago
    [deleted]
pbronez 4y ago
This is my fear about my homelab lol Fire extinguisher nearby, smart temp sensors, but still...
- W-Stool 4y ago
  What are you using for a homelab priced temperature sensor?
  - xani_ 4y ago
    Homelab-priced sensor is the temp sensor in your server, it's free! Actual servers have a bunch, usually have one at intake, "random old PC" servers can use motherboard temp as rough proxy for environment temp. Hell, even in DC you can look at temperatures and see in front of which server technican was standing just by those sensors. Second cheapest would be USB-to-1wire module + some DS18B20 1-sire sensors. Easy hobby job to make. They also come with unique ID which means if you put it in TSDB by that ID it doesn't matter where you plug those sensors.
- rovr138 4y ago
  oh, nice idea with temp sensor. I have extinguishers all over the house, but hadn't considered a temperature sensor set to send alerts. Do you have any recommendations?
- xani_ 4y ago
  No more or less chance that really anything else connected to power to catch on fire. Just make sure RCD works. Hell, some weeks ago my oven decided the lower heater's line is now connected to ground and blows fuses... Our 8 racks in DC only had single event of something blowing (power supply) and aside of smell and fuse blowing nothing really happened Servers are essentially metal boxes with a bit of glass-reinforced epoxy and some plastic inside so there is limited amount of stuff that can burn. UPS is probably bigger problem For example the OVH datacenter fire was more of "stuff around servers was flammable" (they had wooden ceilings for some reason...) rather than "just" servers.
  - 4y ago
walnutclosefarm 4y ago
I never saw an actual fire, but I we did see smoke, in a closet, on a floor that was 90% patient care, but happened to have a research area as well because the research needed access to expensive radiography equipment. The close was literally stuffed with what amounted to gaming machines, purchased with grant money through an importer, directly from China. The guy who set them up was smart enough to put them all behind a little firewall, so the enterprise network couldn't see them. It was a (literally) hot mess, both infrastructure and security-wise.

bluedino 4y ago

It's kind of funny around this time of year when some researchers have $10,000 in their budget they need to spend, and they want to 'gift' us with some GPU's.

davidmr 4y ago
That was definitely one of the weirdest things of working in academia IT: “hey. Can you buy me a workstation that’s as close to $6,328.45 as it is possible to get, and can you do it by 4pm?”
- kjs3 4y ago
  Same thing happens in the government sector here (US). If you don't spend all of the budget you requested last year, you might not get it next year. There is an entire ecosystem of bottom-feeder GSA companies that apparently exist to spend year-end money that would otherwise go to 'waste'.

systemvoltage 4y ago

I am dealing with the exact opposite problem: "Oh you mean, we should leave the EC2 instance running 24/7??? No way, that would be too expensive"... to which I need to respond "No, it would be like $15/month. Trivial, stop worrying about costs in EC2 and S3, we're like 7 people here with 3 GB of data." I deal with Scientists that think AWS is some sort of a massively expensive enterprise thing. I can be, but not for the use case they're going to be embarking on. Our budget is $7M spanning 4 years.

capableweb 4y ago
> think AWS is some sort of a massively expensive enterprise thing Compared to using dedicated instances with way cheaper bandwidth, storage and compute power, it might as well be. Cloud makes sense when you have to scale up/down very quickly, or you'd be losing money fast. But most don't suffer from this problem.
- pwinnski 4y ago
  At $15/month, it would take longer than a typical school year to beat.
gonzo41 4y ago
Don't say the budget outloud near AWS. They'll find a way to help you spend it.
- systemvoltage 4y ago
  Hahaha, may be I need to just go into the AWS ether and start yakking big words like "Elastic Kubernetes Service" to confuse the scientists and get my aws fix. These people are too stingy. I want some shit running in AWS, what good is this admin IAM role.
- absolutelynobo 4y ago
  GP has already lost that battle.
- 4y ago

funstuff007 4y ago

I cannot wait for Ampere servers to become widespread. Most of the power and cooling issues will go away.

getcrunk 4y ago
sarcasm?

absolutelynobo 4y ago

> Don't forget the AC terminal strips that power the whole thing, sourced from the local dollar store. Love how you’ve fuzzed the root cause to make it seem like the “dollar store strip” is the problem and not that it was plugged into an overloaded outlet or run inside a closet at significantly elevated temperatures, leading to plastic melting and wires shorting. Always helps to keep the “magic” a secret so the rubes have to keep us wizards employed, right?

morbia 4y ago

Your comment about cooling reminded me of a fun anecdote from my time in academia. My PhD involved quite a lot of high performance computing simulations. We kept running into problems where on warm days, all our jobs would get killed pretty consistently. Our IT guy noticed a pattern where the temps would be perfectly normal, and then suddenly out of nowhere go through the roof, triggering a thermal shutdown of the racks. In the end our IT guy camped out in the server room on a hot day to watch what was happening. The Astrophysics' cabinet were directly infront of ours, and they had jerry rigged their cabinet door so that when it got too hot, it would swing open and their hot air would be blown all over the neighbouring cabinets...