9 ms·

Having had the responsibility of providing HPC for a literal buildings full of scientists, I can say that it may be true that you can get computation cheaper wi

• mattpallissard 4y ago
  I also managed HPC data centers and I agree with you. I feel like the term data center is a key word there. There is a point of scaling where it's just cheaper to manage a data center with a dedicated in house team. I think that holds true in other industries as well. As far as HPC goes specifically, we could get some of the financial numbers to make sense in the cloud (cpu intensive jobs), but couldn't make it work for others (data intensive jobs shipping PB's around on the reg) That and HPC has a lot of grant funding. It can be quite advantageous for an org to use an on prem data center almost like a slush fund. Can keep key projects running that would otherwise die when they are having a rough funding year.
• W-Stool 4y ago
  Let me echo this as someone who once was responsible for HPC computing in a research intensive public university. Most career academics have NO IDEA how much enterprise computing infrastructure costs. If a 1 terabyte USB hard drive is $40 at Costco we (university IT) must be getting a much better deal than that. Take this argument and apply it to any aspect of HPC computing and that's what you're fighting against. The closet with racks of gear and no cooling is another fond memory. Don't forget the AC terminal strips that power the whole thing, sourced from the local dollar store.
  • 0xbadcafebee 4y ago
    I remember the first time a server caught fire in the closet we kept the rack in. Backups were kept on a server right below the one on fire. But, y'know, we saved money.
    • eastbound 4y ago
      Don’t worry, we do incremental backups during weekdays and a full backup on Sunday. We use 2 tapes only, so one is always outside of the building. But you know, we saved money.
      • deleted 4y ago
        [deleted]
      • deleted 4y ago
        [deleted]
      • bilbo0s 4y ago
        A bit off topic, but I gotta say you guys are a riot! If there was a comedy tour for IT/Programmer types, I'd pay to see you guys in it. Best thing about your stuff is that it's literally all funny precisely because it's all true.
        • deleted 4y ago
          [deleted]
        • geoduck14 4y ago
          I don't know why you are being down voted. I'd watch a Netflix special on this stuff
      • Proven 4y ago
      • treeman79 4y ago
        We had a million dollars worth of hardware installed in a closet. It had a portable AC hooked up that needed it’s water bin changed every so often. Well I was in the middle of that. When the Director decided to show off the new security doors. So he closed the room up. Then found out that new security doors didn’t work. I find out as I’m coming back to turn AC back on. Room will get hot really fast. We get office Security to unlock door. He says he doesn’t have authority. His supervisor will be by later in the day. Completely deadpan, and in front of several VPs of a forth one 50. I turn to guy to my right who lived nearby. “Go home and get your chainsaw” We were quickly let in. Also got fast approval to install proper cooling.
• ptero 4y ago
  I think it really depends on the task. Where HIPAA violation is a real threat, the equation changes. And just for CYA purposes those projects can get pushed to a cloud. Which does not necessarily involve any attempts to make them any more secure, but this is a different topic. That said, many scientists are operating on premise hardware like this: some servers in a shared rack and an el-cheapo storage solutions with an ssh access for people working in the lab. And it works just fine for them. Cloud services focus for running business computing in a cloud, emphasizing recurring revenue. Most research labs are much more comfortable with spending the hardware portion of a grant upfront and not worrying about some student who, instead of working on some fluid dynamics problem found a script to re-train a stable diffusion and left it running over winter break. My 2c.
  • crazygringo 4y ago
    > And it works just fine for them. Until it doesn't because there's a fire or huge power surge or whatever. That's the point -- there's a lot of risk they're not taking into account, and by focusing on the "it works just fine for them", you're cherry picking the ones that didn't suffer disaster.
    • withinboredom 4y ago
      Ummm. I’ve def been unable to do anything for entire days because our AWS region went down and we had to rebuild the database from scratch. AWS goes down, you twiddle your thumbs and the people you report to are going to be asking why, for how long, etc. and you can’t give them an answer until AWS comes back to see how fubar things are. When your own hardware rack goes down. You know the problem, how much it costs to fix it, and when it will come back up; usually within a few hours (or minutes) of it going down. Do things catch fire, yes. But I think you’re over-estimating how often. In my entire life, I’ve had a single SATA connector catch fire and it just melted plastic before going out.
      • crazygringo 4y ago
        I'm not talking about temporary outages, I'm talking about data loss. With AWS it's extremely easy to keep an up-to-date database backup in a different region. And it's great that you haven't personally encountered disaster, but of course once again that's cherry-picking. And it's not just a component overheating, it's the whole closet on fire, it's a broken ceiling sprinkler system going off, it's a hurricane, it's whatever.
• duxup 4y ago
  When I worked as a network engineer I spent months working with some great scientists / their team who built a crazy microscope (I assumed it was looking at atoms or something...) the size of a small building. Their budget for the network gear was a couple hundred bucks and some old garbage consumer grade network gear. For something that spit out 10s of GB a second (at least) across a ton of network connections (they didn't seem to know what would even happen when they ran it), and was so bursty all but the highest end of gear could handle it. Can confirm sometimes scientists aren't really up on the overall costs. Then they dump it "this isn't working" on their university IT team to absorb the costs / manpower costs.
• forgomonika 4y ago
  This nails so much of the discussion that should be had. When using any cloud service provider, you aren't just paying for the machines/hardware you use - you are paying for people to take care of a bunch of headaches of having to maintain this hardware. It's incredibly easy to overlook this aspect of costs and really easy to oversimplify what's involved if you don't know how these things actually work.
• COGlory 4y ago
  >the security infrastructure, provide redundancy and still get to shared compute when they have an overflow need The article points out that this is mostly not necessary for scientific computing.
  • jrumbut 4y ago
    Which I thought was the best point of the article, that a lot of IT best practice comes from the web app world. Web apps quickly become finely tuned factory machines, executing a million times a day and being duplicated thousands of times. Scientific computing projects are often more like workshops. Getting charged by the second while you're sitting at a console trying to figure out what this giant blob you were sent even is is unpleasant. The solution you create is most likely to be run exactly once. If it is a big hit, it may be run a dozen times. Trying to run scientific workloads on the cloud is like trying to put a human shoe on a horse. It might be possible but it's clearly not designed for that purpose.
• ipaddr 4y ago
  You are paying 10x more because no one gets fired for using IBM. AWS has many benefits most which you don't need. Pair up with another school in a different region and backup data. Computers are not scary they rarely catch fire.
• mangecoeur 4y ago
  I've been running a group server (basically a shared workstation) for 5 years and it's been great. Way cheaper than cloud, no worrying about national rules on where data can be stored, no waiting in a SLURM batch queue, Jupyter notebooks on tap for everyone. A single $~6k outlay (we don't need GPUs which helps). Classic big workstations are way more capable than people think - but at the same time it's hard to justify buying one machine per user unless your department is swimming in money. Also, academic budgets tend to come in fixed chunks, and university IT departments may not have your particular group as a priority - so often it's just better to invest once in a standalone server tower that you can set up to do exactly what you need it to than try to get IT to support your needs or the accounting department to pay recurring AWS bills.
  • killingtime74 4y ago
    Aren't you talking about 1 server when this is talking about HPC?
    • deleted 4y ago
      [deleted]
    • mangecoeur 4y ago
      Well the title is scientific computing, which includes HPC but not only. Anyway the fact is that a lot of "HPC" in university clusters is smaller jobs that are too much for an average PC to handle, but still fit into a single typical HPC node. These are usually the jobs that people think to farm out to AWS, but that you will generally find are cheaper, faster, and more reliable if you just run them on your own hardware.
• secabeen 4y ago
  They leave out major costs because they don't pay those costs. Power, Cooling, Real Estate are all significant drivers of AWS costs. Researchers don't pay those costs directly. The university does, sure, but to the researcher, that means those costs are pre-paid. Going to AWS means you're essentially paying for those costs twice. plus all the profit margin and availability that AWS provides that you also don't need.
  • walnutclosefarm 4y ago
    Definitely some truth in this, and the way research funding is managed, especially with Federal grants, makes it incredibly difficult to sort out and identify the right solution.
• whatever1 4y ago
  Nah for us it was the department IT guy who set up once everything (a full cluster of 50 r720s) and works like a dream. Properly provisioned linux machines need no maintenance. You drive them until there is a hardware failure.
  • fuzzfactor 4y ago
    Sounds like you've got the kind of outstanding IT guy that was motivated to make the electronics run like the scientists wanted more so than anything. At the other end of the spectrum there's labs [0] where the scientists need to carefully study and develop increasing skills at operating the electronics the way IT wants it done, and even worse of a distraction when there's a moving target keeping up with the IT electronics approach changing faster than the progress most labs make in their own scientific field. What labs need more of is your kind of IT operator who can bring that option (your extreme end of the spectrum in favor of lab workers) within reach for when it is the most appropriate choice. When labs fail to retain such adequate talent, they can rule out that option going forward, and that's one less tool in the toolbox. [0] Including many which have good records of breakthrough progress before becoming computerized to begin with.
• jacobr1 4y ago
  Also it assumes full utilization of hardware. If you have variable load (such as only needing to run compute after an experiment). The overhead costs of maintaining a cluster you don't need all time are probably much lower than resources you can schedule on-demand.
• pbronez 4y ago
  The article estimated: Running a modern AMD-based server that has 48 cores, at least 192 GB of RAM, and no included disk space costs: ~$2670.36/mo for a c5a.24xlarge AWS on-demand instance ~$1014.7/mo for a c5a.24xlarge AWS reserved instance on a three-year term, paid upfront ~$558.65/mo on OVH Cloud[1] ~$512.92/mo on Hetzner[2] ~$200/mo on your own infrastructure as a large institution[3] Footnote [3] explains this cost estimate as: "Assumes an AMD EPYC 7552 run at 100% load in Boston with high electricity prices of $0.23/kWh, for $33.24/mo in raw power. Hardware is amortized over five years, for an average monthly price of $67.08/mo. We assume that your large institution already has 24/7 security and public internet bandwidth, but multiply base hardware and power costs by 2x to account for other hardware, cooling, physical space, and a half-a-$120k-sysadmin amortized across 100 servers."
• insane_dreamer 4y ago
  Plus the supposed savings of in-house hardware only materialize if you have sufficiently managed and queued load to keep your servers running at 100% 24/7. The advantage of AWS/other is to be able to acquire the necessary amount of compute power for the duration that you need it. For a large university it probably makes sense to have and manage their own compute infrastructure (cheap post-doc labor, ftw!) but for smaller outfits, AWS can make a lot of sense for scientific computing (said as someone who uses AWS for scientific computing), especially if you have fluctuating loads. What works best IMO (and what we do) is have a minimum-to-moderate amount of compute resources in house that can satisfy the processing jobs most commonly run (and where you haven't had to overinvest in hardware), and then switch to AWS/other for heavier loads that run for a finite period. Another problem with in-house hardware is that you spent all that money on Nvidia V100's a few years ago and now there's the A100 that blows it away, but you can't just switch and take advantage of it without another huge capital investment.
• onetimeusename 4y ago
  Is a postdoc hacking a cluster something you have seen before? I am genuinely curious because I worked on a cluster owned by my university as an undergrad and everyone was kind of assumed to be trusted. If you had shell access on the main node you could run any job you wanted on the cluster. You could enhance security I just wonder about this threat model, that's an interesting one. I am sure it happens to be clear.
  • walnutclosefarm 4y ago
    Yes. Probably not surprise that the postdoc was a PRC national. Very competent in their field of study, but also in this country with instructions from an APT group.
    • onetimeusename 4y ago
      I had a feeling you would say that. I don't think this was part of our threat model until pretty recently. That was why I asked, because these stories aren't really internalized collectively yet I think and it's valuable to reconsider who we can trust and what threat actors might value.
    • kregasaurusrex 4y ago
      Sorry you were on the receiving end of that and had to learn the hard way. We had the dean of the college (large public research university) I was working at the time receive a gift of a bunch of new MBPs as a token of goodwill from a foreign country, and heard in the very next sentence that they would be going straight into a shredder. At the time I thought these brand-new laptops could easily be wiped & repurposed; but now realize it was a potential attack surface that can easily leak info to APT/espionage groups the second it connects to a network. Less than a year later a story made national headlines that a professor with direct access to classified material had mysteriously disappeared & never disclosed his close ties with their home country. So not only should you be worried about back doors and side doors; but also watch what's going through your front door as well!
• prpl 4y ago
  The things that tend to be "cheap" on campuses: Power (especially if there is some kind of significant scientific facility on premise), space (especially in reused buildings), manpower (undergrads, grad students, post docs, professional post graduates), running old/reused hardware, etc... You can get away with those at large research universities. Some of that you can get away with at national lab sorts of places (not going to find as much free/cheap labor, surplus hardware). If you start going down in scale/prestige, etc... none of that holds true. Running a bunch of hardware from the surplus store in a closet somewhere with Lasko fans taped to the door is cheap. To some extent, the university system encourages such subsidies. In any case, once you get to actually building a datacenter, if you have to factor in power, if you have a 4 hardware refresh cycle, professional staffing, etc... unless you are in one of those low CoL college towns - cloud is probably more more than 1.5 to 3x more expensive for compute (spot, etc...). Storage on prem is much cheaper - erasure coded storage systems are cheap to buy and run, and everybody wants their own high performance file system. One continuing cloud obstacle though - researchers don't want to spend their time figuring out how to get their code friendly to preemptible VMs - which is the cost effective way to run on cloud. Another real issue with sticking to on-prem HPC is talent acquisition and staff development. When you don't care about those things so much, it's easy to say it's cheap to run on-prem, but often the pay is crap for the required expertise, and ignoring cloud doesn't help your staff either.
• funstuff007 4y ago
  I'd much rather store HIPAA data on a server in my office or closet than worry I got all the IAM settings right. And if I fire someone, security makes sure they can't get in the building. You cannot say the same about the cloud. Yes, I know you can do cloud security right, but on prem security is just harder to mess up.
  • walnutclosefarm 4y ago
    More than half of security penetrations in our institution (A medical center - research - med school complex) over the 8 years I worked there, ending in 2020, came through the research arm, even though Research accounted for no more than 10% of enabled servers in the infrastructure. And we're talking APT penetrations. They weren't looking for HIPAA data (although I used that example in my original post), they were looking for a path to permanent presence in our network in order to mine research. So, why did they come through Research? Because a PI buys some equipment - servers or other network enabled stuff - puts a grad student or post doc in charge of it, and enjoys his or her cheap compute. But that student or post doc is not an infrastructure expert, and most definitely doesn't understand enterprise security. Next thing you know, we've got an APT owned server on the inside of the network. (And none of that is counting the ones where the post doc is a foreign national who actually intends to use their position to compromise their employer. Had that happen too.) There are a some computational scientists who actually do understand this stuff, but they're rare. Being on the cloud does not inherently fix this oroblem, but to fix it you have to be on institutionally, professionally managed infrastructure, and once you are, the cost differential between owned infrastructure and well negotiated, managed cloud infrastructure becomes much more nuanced.
    • funstuff007 4y ago
      I appreciate your perspective, but it seems the security team should be watching for reverse proxies, tunnels, and other firewall anomalies for on-prem hardware just as a normal course of biz. And if a PI installs a self-managed server, that really should not gum up the works. All that being said, I have never worked at a place (or in a dept) whose threat profile made APT a real thing.
• cdsx 4y ago
  Perspective from a computational biologist: Campus hosted HPC means the direct cost pressure is seen as IT staff and hardware related costs. Researchers are encouraged to use the available capacity. This is good. Externally-hosted HPC means every single compute job is seen as something that directly costs money. This negatively affects the quality of scientific output (research playfulness / creativity / focus on the research / etc).
  • walnutclosefarm 4y ago
    Yes. The costing models that are used (and often required by granting agencies) make apples to apples cost comparisons almost impossible, and impose undoubted and significant false costs on research budgets. No question this is true.
  • osigurdson 4y ago
    >> seen as something that directly costs money. Seen by whom? Dummies it seems. Maybe the dummies are the problem, not the computing or accounting models.