4 ms·

I think it really depends on the task. Where HIPAA violation is a real threat, the equation changes. And just for CYA purposes those projects can get pushed to

by ptero 4y ago

I think it really depends on the task. Where HIPAA violation is a real threat, the equation changes. And just for CYA purposes those projects can get pushed to a cloud. Which does not necessarily involve any attempts to make them any more secure, but this is a different topic. That said, many scientists are operating on premise hardware like this: some servers in a shared rack and an el-cheapo storage solutions with an ssh access for people working in the lab. And it works just fine for them. Cloud services focus for running business computing in a cloud, emphasizing recurring revenue. Most research labs are much more comfortable with spending the hardware portion of a grant upfront and not worrying about some student who, instead of working on some fluid dynamics problem found a script to re-train a stable diffusion and left it running over winter break. My 2c.

crazygringo 4y ago
> And it works just fine for them. Until it doesn't because there's a fire or huge power surge or whatever. That's the point -- there's a lot of risk they're not taking into account, and by focusing on the "it works just fine for them", you're cherry picking the ones that didn't suffer disaster.
- withinboredom 4y ago
  Ummm. I’ve def been unable to do anything for entire days because our AWS region went down and we had to rebuild the database from scratch. AWS goes down, you twiddle your thumbs and the people you report to are going to be asking why, for how long, etc. and you can’t give them an answer until AWS comes back to see how fubar things are. When your own hardware rack goes down. You know the problem, how much it costs to fix it, and when it will come back up; usually within a few hours (or minutes) of it going down. Do things catch fire, yes. But I think you’re over-estimating how often. In my entire life, I’ve had a single SATA connector catch fire and it just melted plastic before going out.
  - crazygringo 4y ago
    I'm not talking about temporary outages, I'm talking about data loss. With AWS it's extremely easy to keep an up-to-date database backup in a different region. And it's great that you haven't personally encountered disaster, but of course once again that's cherry-picking. And it's not just a component overheating, it's the whole closet on fire, it's a broken ceiling sprinkler system going off, it's a hurricane, it's whatever.
    - withinboredom 4y ago
      So was I also talking about data loss. Not everything can be replicated, but backups can and were made. For the rest, there’s insurance. Most calculations done in a research setting are dependent upon that research surviving. If there’s a fire and the whole building goes down, those calculations are probably worthless now too. Hell, most companies probably can’t survive their own building/factory burning down.
    - FpUser 4y ago
      >"With AWS it's extremely easy to keep an up-to-date database backup in a different region" It is just as extremely easy on Hetzner or on premises
      - monkmartinez 4y ago
        I would say even easier on prem as you don't need to wade 15 layers deep to do anything. Since I have moved to hosting my own stuff at my house, I have learned that connecting a monitor and keyboard to a 'sever' is awesome for productivity. I know where everything is, its fast as hell, and everything is locked down. Monitoring temps, adjusting and configuring hardware is just better in every imaginable way. Need more RAM, Storage, Compute? Slap those puppies in there and send it. For home gamers like myself, it's has become a no brainer with advances in tunneling, docker, and cheap prices on Ebay.
- kijin 4y ago
  Even that depends on what you're doing. Most scientists aren't running apps that require several 9's of availability, connect to an irreplaceable customer database, etc. An outage, or even permanent loss of hardware, might not be a big problem if you're running easily repeatable computations on data of which you have multiple copies. At worst, you might have to copy some data from an external hard drive and redo a few weeks' worth of computations.
- billythemaniam 4y ago
  The counterpoint to that point is that a significant percentage of scientific computing doesn't care about any of that. They are unlikely to have enough hardware to cause a fire and they don't care about outages or even data loss in many cases. As others have said, it depends on the specifics of the research. In the cases where that stuff matters, the cloud would be better option.
  - Fomite 4y ago
    This. If my lab-level server failed tomorrow, I'd be annoyed, order another one, and start the simulations again.
- ptero 4y ago
  > there's a lot of risk they're not taking into account I see it the other way: experimental scientists operate with unreliable systems all the time: fickle systems, soldered one-time setups, shared lab space, etc. Computing is just one more thing that is not 100% reliable (but way more reliable than some other equipment), and usb data sticks serve as a good enough data backup.
  - mangecoeur 4y ago
    Or your university might have it's own backup system. We have a massive central tape-based archive that you can run nightly backups to.
- horsawlarway 4y ago
  I'd counter by saying I think you're over-estimating how valuable mitigating that risk is to this crowd. I'd further say that you're probably over-estimating how valuable mitigating that risk is to anyone, although there are a few limited set of customers that genuinely do care. There are few places I can think of that would benefit more by avoiding cloud costs than scientific computing... They often have limited budgets that are driven by grants, not derived by providing online services (computer going down does not impact bottom line). They have real computation needs that mean hardware is unlikely to sit idle. There is no compelling reason to "scale" in the way that a company might need to in order to handle additional unexpected load from customers or hit marketing campaigns. Basically... the only meaningful offering from the cloud is likely preventing data loss, and this can be done fairly well with a simple backup strategy. Again - they aren't a business where losing a few hours/days of customer data is potentially business ending. --- And to be blunt - I can make the same risk avoidance claims about a lot of things that would simply get me laughed out of the room. "The lead researcher shouldn't be allowed in a car because it might crash!" "The lab work must be done in a bomb shelter in case of war or tornados!" "No one on the team can eat red meat because it increases the risk of heart attack!" and on and on and on... Simply saying "There's risk" is not sufficient - you must still make a compelling argument that the cost of avoiding that risk is justified, and you're not doing that.
- noobermin 4y ago
  May be consider that your use case and the average scientist's use case isn't the same? What works for you won't work for them and vise versa? What you consider a risk, I wouldn't? Consider the following, I have never considered applying meltdown or spectre mitigations if it makes my code run slower because I plain don't care, assuming anyone even peeks at what my simulations doing, whoopdeedo, I don't care. I won't do that on my laptop I use to buy shit off amazon with, but the workstation I have control of? I don't care. I DO care if my simulation will take 10 days instead of a week. My use case isn't yours because my needs aren't yours. Not everything maps across domains.
- vjk800 4y ago
  The point is, there's not need for everything to be 100% reliable in this context. If a fire destroys everything and their computational resources is unavailable for a few days, that's somewhat okay. Not ideal, but not a catastrophic loss either. Even data loss is no catastrophic - at worst it means redoing one or two weeks worth of computations. Some sort of 80/20 principle is at works here. Most of the costs in professional cloud solutions comes from making the infrastructure 99.99% reliable instead of 99% reliable. It is totally worth it if you have millions of customers that expect a certain level of reliability, but a complete overkill if the worst case scenario from a system failure is some graduate student having to redo a few days worth of computations (which probably had to be redone several times anyway because of some bug in the code or something).
secabeen 4y ago
Thankfully, only a small part of the academic research enterprise involves human subjects, HIPAA, and all that. Neither fruit flies nor quarks have privacy rights.
- dmicah 4y ago
  Research involving human subjects (psychology, cognitive neuroscience, behavioral economics, etc.) requires institutional review board approval and informed consent, etc. but mostly doesn't involve HIPAA either.
  - charcircuit 4y ago
    That is not a law.
    - icedchai 4y ago
      There are actually laws around such things. You can read about them here: https://www.hhs.gov/ohrp/index.html https://www.hhs.gov/ohrp/index.html
      - charcircuit 4y ago
        That only applies to government / government sponsored research. That is different from someone doing their own research
        icedchai 4y ago
        This is true, but to say it is "not a law", as you did, completely unqualified, is incorrect. If the research project is connected with a government grant (and many are) you need to pay attention to those laws. Many universities also have their own policies you need to follow, regardless. (Requiring informed consent and protecting people's privacy seems like a good thing.)
        4y ago

by ptero 4y ago

crazygringo 4y ago

> And it works just fine for them. Until it doesn't because there's a fire or huge power surge or whatever. That's the point -- there's a lot of risk they're not taking into account, and by focusing on the "it works just fine for them", you're cherry picking the ones that didn't suffer disaster.

withinboredom 4y ago
Ummm. I’ve def been unable to do anything for entire days because our AWS region went down and we had to rebuild the database from scratch. AWS goes down, you twiddle your thumbs and the people you report to are going to be asking why, for how long, etc. and you can’t give them an answer until AWS comes back to see how fubar things are. When your own hardware rack goes down. You know the problem, how much it costs to fix it, and when it will come back up; usually within a few hours (or minutes) of it going down. Do things catch fire, yes. But I think you’re over-estimating how often. In my entire life, I’ve had a single SATA connector catch fire and it just melted plastic before going out.
- crazygringo 4y ago
  I'm not talking about temporary outages, I'm talking about data loss. With AWS it's extremely easy to keep an up-to-date database backup in a different region. And it's great that you haven't personally encountered disaster, but of course once again that's cherry-picking. And it's not just a component overheating, it's the whole closet on fire, it's a broken ceiling sprinkler system going off, it's a hurricane, it's whatever.
  - withinboredom 4y ago
    So was I also talking about data loss. Not everything can be replicated, but backups can and were made. For the rest, there’s insurance. Most calculations done in a research setting are dependent upon that research surviving. If there’s a fire and the whole building goes down, those calculations are probably worthless now too. Hell, most companies probably can’t survive their own building/factory burning down.
  - FpUser 4y ago
    >"With AWS it's extremely easy to keep an up-to-date database backup in a different region" It is just as extremely easy on Hetzner or on premises
    - monkmartinez 4y ago
      I would say even easier on prem as you don't need to wade 15 layers deep to do anything. Since I have moved to hosting my own stuff at my house, I have learned that connecting a monitor and keyboard to a 'sever' is awesome for productivity. I know where everything is, its fast as hell, and everything is locked down. Monitoring temps, adjusting and configuring hardware is just better in every imaginable way. Need more RAM, Storage, Compute? Slap those puppies in there and send it. For home gamers like myself, it's has become a no brainer with advances in tunneling, docker, and cheap prices on Ebay.
kijin 4y ago
Even that depends on what you're doing. Most scientists aren't running apps that require several 9's of availability, connect to an irreplaceable customer database, etc. An outage, or even permanent loss of hardware, might not be a big problem if you're running easily repeatable computations on data of which you have multiple copies. At worst, you might have to copy some data from an external hard drive and redo a few weeks' worth of computations.
billythemaniam 4y ago
The counterpoint to that point is that a significant percentage of scientific computing doesn't care about any of that. They are unlikely to have enough hardware to cause a fire and they don't care about outages or even data loss in many cases. As others have said, it depends on the specifics of the research. In the cases where that stuff matters, the cloud would be better option.
- Fomite 4y ago
  This. If my lab-level server failed tomorrow, I'd be annoyed, order another one, and start the simulations again.
ptero 4y ago
> there's a lot of risk they're not taking into account I see it the other way: experimental scientists operate with unreliable systems all the time: fickle systems, soldered one-time setups, shared lab space, etc. Computing is just one more thing that is not 100% reliable (but way more reliable than some other equipment), and usb data sticks serve as a good enough data backup.
- mangecoeur 4y ago
  Or your university might have it's own backup system. We have a massive central tape-based archive that you can run nightly backups to.
horsawlarway 4y ago
I'd counter by saying I think you're over-estimating how valuable mitigating that risk is to this crowd. I'd further say that you're probably over-estimating how valuable mitigating that risk is to anyone, although there are a few limited set of customers that genuinely do care. There are few places I can think of that would benefit more by avoiding cloud costs than scientific computing... They often have limited budgets that are driven by grants, not derived by providing online services (computer going down does not impact bottom line). They have real computation needs that mean hardware is unlikely to sit idle. There is no compelling reason to "scale" in the way that a company might need to in order to handle additional unexpected load from customers or hit marketing campaigns. Basically... the only meaningful offering from the cloud is likely preventing data loss, and this can be done fairly well with a simple backup strategy. Again - they aren't a business where losing a few hours/days of customer data is potentially business ending. --- And to be blunt - I can make the same risk avoidance claims about a lot of things that would simply get me laughed out of the room. "The lead researcher shouldn't be allowed in a car because it might crash!" "The lab work must be done in a bomb shelter in case of war or tornados!" "No one on the team can eat red meat because it increases the risk of heart attack!" and on and on and on... Simply saying "There's risk" is not sufficient - you must still make a compelling argument that the cost of avoiding that risk is justified, and you're not doing that.
noobermin 4y ago
May be consider that your use case and the average scientist's use case isn't the same? What works for you won't work for them and vise versa? What you consider a risk, I wouldn't? Consider the following, I have never considered applying meltdown or spectre mitigations if it makes my code run slower because I plain don't care, assuming anyone even peeks at what my simulations doing, whoopdeedo, I don't care. I won't do that on my laptop I use to buy shit off amazon with, but the workstation I have control of? I don't care. I DO care if my simulation will take 10 days instead of a week. My use case isn't yours because my needs aren't yours. Not everything maps across domains.
vjk800 4y ago
The point is, there's not need for everything to be 100% reliable in this context. If a fire destroys everything and their computational resources is unavailable for a few days, that's somewhat okay. Not ideal, but not a catastrophic loss either. Even data loss is no catastrophic - at worst it means redoing one or two weeks worth of computations. Some sort of 80/20 principle is at works here. Most of the costs in professional cloud solutions comes from making the infrastructure 99.99% reliable instead of 99% reliable. It is totally worth it if you have millions of customers that expect a certain level of reliability, but a complete overkill if the worst case scenario from a system failure is some graduate student having to redo a few days worth of computations (which probably had to be redone several times anyway because of some bug in the code or something).

secabeen 4y ago

Thankfully, only a small part of the academic research enterprise involves human subjects, HIPAA, and all that. Neither fruit flies nor quarks have privacy rights.

dmicah 4y ago
Research involving human subjects (psychology, cognitive neuroscience, behavioral economics, etc.) requires institutional review board approval and informed consent, etc. but mostly doesn't involve HIPAA either.
- charcircuit 4y ago
  That is not a law.
  - icedchai 4y ago
    There are actually laws around such things. You can read about them here: https://www.hhs.gov/ohrp/index.html https://www.hhs.gov/ohrp/index.html
    - charcircuit 4y ago
      That only applies to government / government sponsored research. That is different from someone doing their own research
      - icedchai 4y ago
        This is true, but to say it is "not a law", as you did, completely unqualified, is incorrect. If the research project is connected with a government grant (and many are) you need to pay attention to those laws. Many universities also have their own policies you need to follow, regardless. (Requiring informed consent and protecting people's privacy seems like a good thing.)
        4y ago