6 ms·

Fermilab is 'doomed' without management overhaul claims whistleblower report

79 points by f1shy 2y ago

cut3 2y ago
An investigation shows a problem with leadership and leadership responds by saying the report isnt correct. Classic.
elashri 2y ago
I would like to also point out that the decision to cut the development of scientific Linux came after pressure from fermilab administration. They had the classical, why not just rely on Redhat CentOS and save the efforts. Ironically after the plans were announced and the community was shocked. In less than a year RedHat announced that they will stop developing CentOS. And now instead of making a review of their decision and see the actual needs and think about cutting the reliance on RedHat, they went fully dependent on RHEL (one can say that CERN takes part of the blame too).
- shepardrtc 2y ago
  I wonder if there were any nice dinners had by the administration and Red Hat sales.
- OutOfHere 2y ago
  They could've just switched to Debian variants with a little effort. More newly, since 2021, we have Alma Linux and Rocky Linux that are RHEL based.
  - jonathanspw 2y ago
    Fermi did swap to AlmaLinux.
    - catherinecodes 2y ago
      AlmaLinux is a perfect replacement for scientific agencies. It's got the fastest turnaround time for security patches, an open community, and sustainable funding sources.
- cmdrk 2y ago
  As someone working in scientific computing, I seem to recall rumors from the grapevine that the Department of Energy and/or reviewers essentially said, "CentOS is identical to Scientific Linux. Why are we paying for this?" and pushed hard to sunset it. It's really sad because Fermi Linux was one of the first (if not _the_ first) free RH rebuilds out there back in the day. Huge props to Connie Sieh and the maintainers that followed for pulling that off. Really wonderful folks. It does certainly feel convenient for RH that the end of SL coincided with radical changes to CentOS.
johnohara 2y ago
> The issues at Fermilab are, however, also hindering the DOE, which earlier this year called for bids on the contract to operate the lab. The University of Chicago and the URA have submitted a contract bid together with other partners. Associated Universities, Inc., which runs the US-based 100 m-diameter Green Bank Telescope and the Atacama Large Millimeter/submillimeter Array in Chile, has also thrown its hat in the ring. The DOE says it will announce the winner of the contract by 30 September. Just give "us" the contract and we'll address all the problems that have come to the fore since the request for bids was announced. Does it ever end?
pfdietz 2y ago
The central problem is the bread and butter of Fermilab, operating an accelerator to push the frontiers of physics, is reaching its limit. At some point one must admit the place has discovered what it's going to discover and decide to shut the place down. This problem is facing particle physics as a whole, world wide, since accelerators have become increasingly expensive as the needed energy has increased.
- twelfthnight 2y ago
  I don't know all the details, but I think they are starting to do novel work with the accelerator to produce neutrinos in an experiment called Dune. So there actually is important stuff being done that isn't just superceded by Cern.
  - pfdietz 2y ago
    Yes, they are squeezing as much juice as they can out of the lemon. But there's diminishing returns. It can't continue forever, and the budget will be under inexorable pressure as the scientific returns become increasingly marginal. This is a problem with science, particularly particle physics: discovering is like mining. Eventually the "ore body" of new phenomena is used up. The period of rapid discovery (up to say the mid 1970s, with a decreasing trickle to 2012 with the discovery of the Higgs Boson) in particle physics will look very atypical and transitory in historical hindsight.
    - twelfthnight 2y ago
      That sounds right to me. I don't work in physics, but have many colleagues who have left it for that reason. ~The said, I think Dune still does seem useful. Hard to say whether the cost overruns are a lack of good management or if the experiment is just a quagmire, though. Guess I lean towards the scientists here that it's the former~ EDIT: Doing some research it seems the Hyper K experiment might just be better, and funding might be better directed there than Fermilab.
    - elashri 2y ago
OutOfHere 2y ago
Fermilab doesn't pay its employees fairly. As such, the only non-scientist engineer employees that stay beyond a few years are ones that would struggle to get a job elsewhere. This is a good way to breed dysfunction. As for management, if they're not scientists doing science, at least half-time, they shouldn't be there.
- mnky9800n 2y ago
  This is a problem across the hard sciences particularly ones with a computational focus. The salaries are, simply put, pathetic. Even at private labs I have friends who make 80-100k USD when industry would pay the same job twice as much. And it's not like 100k is the same 100k it was 5-10 years ago. It's wild that nothing is being done to increase salaries in science. Everyone just seems to bury their heads in the sand and hope someone else will fix it.
  - dr-detroit 2y ago
    [dead]
- rincebrain 2y ago
  I've always described it as people who valued stability over anything else - not just in the sense of "would not get a job elsewhere", but also "values knowing they're not going to get fired in the next decade no matter what", whether that's because they're there for the benefits or the retirement plan or w/e.
  - OutOfHere 2y ago
    You may not get fired, but how long could you survive without a raise? Not long. Any raises they give are about 0.01%. The benefits aren't extraordinary in any sense. In fact, they don't give a pension, whereas even a local government does.
    - rincebrain 2y ago
      Some people survive for quite some time, depending on where they are. But notably, if they're linked to academic institutions, you and yours might get significant benefits around continuing education.
mikewarot 2y ago
Should it ever come to pass that FermiLab were shut down, the real-estate developers would grab it in a heart beat. It's a massive block of prime west Chicagoland Real Estate.
- Kon-Peki 2y ago
  The chance of that happening is close to nil. The property is owned by the US Government and would likely get transferred to the US Forest Service to be managed as a tallgrass prairie, like Midewin [1] (large amounts of open space within Fermilab are already being managed as a tallgrass prairie [2]). [1] https://www.fs.usda.gov/midewin https://www.fs.usda.gov/midewin [2] https://ecology.fnal.gov https://ecology.fnal.gov
- wenc 2y ago
  Legality aside, I used to work in Hinsdale and I’ve visited Fermilab in Batavia. I could be all wrong about this but the location didn’t seem that desirable. Batavia is far away from major highways and feels like a forgotten sleepy town out in the boonies. I remember feeling, “who would ever want to live here except physicists who wanted to isolate themselves?” This is in contrast to Argonne National Lab (also operated by UChicago) down the street (well 30 mi, 40 mins away) in Lemont. It’s next to the I-55 and reachable from populated burbs like Naperville and Westmont, and even Chicago. Food options abound in neighboring suburbs. Now that’s prime real estate.
  - Kon-Peki 2y ago
    Argonne is completely surrounded by (and was carved out of) DuPage County Forest Preserve land. They'll want it back if the government is ever done with it. The forest preserve district will simply remove all the roads on their land and leave it inaccessible.
  - mikewarot 2y ago
    You haven't been down Kirk road in the past few years, have you? There are so many McMansions across the street in land that used to be farms.
OutOfHere 2y ago
A place like Fermilab should in theory be minting money making clever physics gadgets, machines, and services with real world value using 50% of its specialist scientific capacity. Once bootstrapped, this would fix so many of its problems to fund the other 50% that's fundamental research.
- OldGuyInTheClub 2y ago
  They're good at outreach. See their YT videos with Don Lincoln. More merch? A Patreon?
  - OutOfHere 2y ago
    They need more than two or three bucks to run a lab.
  - photonthug 2y ago
    In a perfect world we could afford to have lots of basic research institutions that dream wild dreams 100% of the time. In the current context tho maybe it’s not crazy to ask for 10-20% of time/cost to be applied. That’s not the same as suggesting merch or a bake sale, especially if there’s a route to more sustainable self funding, which idea I would think is appealing to any scientific institution that’s beholden to a political one. Besides, we have lots of urgent problems. Pure basic research is always going to be good/necessary, but whether it seems mostly noble or mostly ridiculous depends on whether the house is burning.
    - OldGuyInTheClub 2y ago
      I'd say that a staggering amount of work at Fermilab is already applied and/or engineering. That's a compliment. Theorists get the PR but building and running those enormous experiments is no joke.
mturmon 2y ago
Another roundup from Science magazine: https://www.science.org/content/article/trouble-and-strife-deepen-famed-u-s-particle-physics-lab https://www.science.org/content/article/trouble-and-strife-d...
floxy 2y ago
Anyone know the current state of the art on "tabletop" particle accelerators? https://newatlas.com/berkely-world-record-compact-particle-accelerator/35118/ https://newatlas.com/berkely-world-record-compact-particle-a... https://news.stanford.edu/stories/2019/09/precision-physics-tabletop-experiments https://news.stanford.edu/stories/2019/09/precision-physics-...

79 points by f1shy 2y ago

cut3 2y ago

An investigation shows a problem with leadership and leadership responds by saying the report isnt correct. Classic.

I would like to also point out that the decision to cut the development of scientific Linux came after pressure from fermilab administration. They had the classical, why not just rely on Redhat CentOS and save the efforts. Ironically after the plans were announced and the community was shocked. In less than a year RedHat announced that they will stop developing CentOS. And now instead of making a review of their decision and see the actual needs and think about cutting the reliance on RedHat, they went fully dependent on RHEL (one can say that CERN takes part of the blame too).

shepardrtc 2y ago
I wonder if there were any nice dinners had by the administration and Red Hat sales.
OutOfHere 2y ago
They could've just switched to Debian variants with a little effort. More newly, since 2021, we have Alma Linux and Rocky Linux that are RHEL based.
- jonathanspw 2y ago
  Fermi did swap to AlmaLinux.
  - catherinecodes 2y ago
    AlmaLinux is a perfect replacement for scientific agencies. It's got the fastest turnaround time for security patches, an open community, and sustainable funding sources.
cmdrk 2y ago
As someone working in scientific computing, I seem to recall rumors from the grapevine that the Department of Energy and/or reviewers essentially said, "CentOS is identical to Scientific Linux. Why are we paying for this?" and pushed hard to sunset it. It's really sad because Fermi Linux was one of the first (if not _the_ first) free RH rebuilds out there back in the day. Huge props to Connie Sieh and the maintainers that followed for pulling that off. Really wonderful folks. It does certainly feel convenient for RH that the end of SL coincided with radical changes to CentOS.

johnohara 2y ago

> The issues at Fermilab are, however, also hindering the DOE, which earlier this year called for bids on the contract to operate the lab. The University of Chicago and the URA have submitted a contract bid together with other partners. Associated Universities, Inc., which runs the US-based 100 m-diameter Green Bank Telescope and the Atacama Large Millimeter/submillimeter Array in Chile, has also thrown its hat in the ring. The DOE says it will announce the winner of the contract by 30 September. Just give "us" the contract and we'll address all the problems that have come to the fore since the request for bids was announced. Does it ever end?

pfdietz 2y ago

The central problem is the bread and butter of Fermilab, operating an accelerator to push the frontiers of physics, is reaching its limit. At some point one must admit the place has discovered what it's going to discover and decide to shut the place down. This problem is facing particle physics as a whole, world wide, since accelerators have become increasingly expensive as the needed energy has increased.

twelfthnight 2y ago
I don't know all the details, but I think they are starting to do novel work with the accelerator to produce neutrinos in an experiment called Dune. So there actually is important stuff being done that isn't just superceded by Cern.
- pfdietz 2y ago
  Yes, they are squeezing as much juice as they can out of the lemon. But there's diminishing returns. It can't continue forever, and the budget will be under inexorable pressure as the scientific returns become increasingly marginal. This is a problem with science, particularly particle physics: discovering is like mining. Eventually the "ore body" of new phenomena is used up. The period of rapid discovery (up to say the mid 1970s, with a decreasing trickle to 2012 with the discovery of the Higgs Boson) in particle physics will look very atypical and transitory in historical hindsight.
  - twelfthnight 2y ago
    That sounds right to me. I don't work in physics, but have many colleagues who have left it for that reason. ~The said, I think Dune still does seem useful. Hard to say whether the cost overruns are a lack of good management or if the experiment is just a quagmire, though. Guess I lean towards the scientists here that it's the former~ EDIT: Doing some research it seems the Hyper K experiment might just be better, and funding might be better directed there than Fermilab.
  - elashri 2y ago

OutOfHere 2y ago

Fermilab doesn't pay its employees fairly. As such, the only non-scientist engineer employees that stay beyond a few years are ones that would struggle to get a job elsewhere. This is a good way to breed dysfunction. As for management, if they're not scientists doing science, at least half-time, they shouldn't be there.

mnky9800n 2y ago
This is a problem across the hard sciences particularly ones with a computational focus. The salaries are, simply put, pathetic. Even at private labs I have friends who make 80-100k USD when industry would pay the same job twice as much. And it's not like 100k is the same 100k it was 5-10 years ago. It's wild that nothing is being done to increase salaries in science. Everyone just seems to bury their heads in the sand and hope someone else will fix it.
- dr-detroit 2y ago
  [dead]
rincebrain 2y ago
I've always described it as people who valued stability over anything else - not just in the sense of "would not get a job elsewhere", but also "values knowing they're not going to get fired in the next decade no matter what", whether that's because they're there for the benefits or the retirement plan or w/e.
- OutOfHere 2y ago
  You may not get fired, but how long could you survive without a raise? Not long. Any raises they give are about 0.01%. The benefits aren't extraordinary in any sense. In fact, they don't give a pension, whereas even a local government does.
  - rincebrain 2y ago
    Some people survive for quite some time, depending on where they are. But notably, if they're linked to academic institutions, you and yours might get significant benefits around continuing education.

mikewarot 2y ago

Should it ever come to pass that FermiLab were shut down, the real-estate developers would grab it in a heart beat. It's a massive block of prime west Chicagoland Real Estate.

Kon-Peki 2y ago
The chance of that happening is close to nil. The property is owned by the US Government and would likely get transferred to the US Forest Service to be managed as a tallgrass prairie, like Midewin [1] (large amounts of open space within Fermilab are already being managed as a tallgrass prairie [2]). [1] https://www.fs.usda.gov/midewin https://www.fs.usda.gov/midewin [2] https://ecology.fnal.gov https://ecology.fnal.gov
wenc 2y ago
Legality aside, I used to work in Hinsdale and I’ve visited Fermilab in Batavia. I could be all wrong about this but the location didn’t seem that desirable. Batavia is far away from major highways and feels like a forgotten sleepy town out in the boonies. I remember feeling, “who would ever want to live here except physicists who wanted to isolate themselves?” This is in contrast to Argonne National Lab (also operated by UChicago) down the street (well 30 mi, 40 mins away) in Lemont. It’s next to the I-55 and reachable from populated burbs like Naperville and Westmont, and even Chicago. Food options abound in neighboring suburbs. Now that’s prime real estate.
- Kon-Peki 2y ago
  Argonne is completely surrounded by (and was carved out of) DuPage County Forest Preserve land. They'll want it back if the government is ever done with it. The forest preserve district will simply remove all the roads on their land and leave it inaccessible.
- mikewarot 2y ago
  You haven't been down Kirk road in the past few years, have you? There are so many McMansions across the street in land that used to be farms.

OutOfHere 2y ago

A place like Fermilab should in theory be minting money making clever physics gadgets, machines, and services with real world value using 50% of its specialist scientific capacity. Once bootstrapped, this would fix so many of its problems to fund the other 50% that's fundamental research.

OldGuyInTheClub 2y ago
They're good at outreach. See their YT videos with Don Lincoln. More merch? A Patreon?
- OutOfHere 2y ago
  They need more than two or three bucks to run a lab.
- photonthug 2y ago
  In a perfect world we could afford to have lots of basic research institutions that dream wild dreams 100% of the time. In the current context tho maybe it’s not crazy to ask for 10-20% of time/cost to be applied. That’s not the same as suggesting merch or a bake sale, especially if there’s a route to more sustainable self funding, which idea I would think is appealing to any scientific institution that’s beholden to a political one. Besides, we have lots of urgent problems. Pure basic research is always going to be good/necessary, but whether it seems mostly noble or mostly ridiculous depends on whether the house is burning.
  - OldGuyInTheClub 2y ago
    I'd say that a staggering amount of work at Fermilab is already applied and/or engineering. That's a compliment. Theorists get the PR but building and running those enormous experiments is no joke.

mturmon 2y ago

Another roundup from Science magazine: https://www.science.org/content/article/trouble-and-strife-deepen-famed-u-s-particle-physics-lab https://www.science.org/content/article/trouble-and-strife-d...

floxy 2y ago

Anyone know the current state of the art on "tabletop" particle accelerators? https://newatlas.com/berkely-world-record-compact-particle-accelerator/35118/ https://newatlas.com/berkely-world-record-compact-particle-a... https://news.stanford.edu/stories/2019/09/precision-physics-tabletop-experiments https://news.stanford.edu/stories/2019/09/precision-physics-...