5 ms·

The NASA engineer who made the James Webb Space Telescope work (2022)

93 points by EndXA 3y ago

neonate 3y ago
https://archive.md/AEzFQ https://archive.md/AEzFQ
dang 3y ago
Related: Greg Robinson fixed NASA’s James Webb Space Telescope, reluctantly - https://news.ycombinator.com/item?id=32109760 https://news.ycombinator.com/item?id=32109760 - July 2022 (199 comments)
WalterBright 3y ago
The article seems a little short on specifics of what he did. I'd like to know more!
- icapybara 3y ago
  It sounds like he was/is a personable engineering manager that paid attention to detail. Making sure parts match the drawings, bringing in outsiders to have a second look at things, etc.
  - bryananderson 3y ago
    Basically this, and it’s a bigger deal than it may sound. Competent technical management is a scarce commodity worth its weight in gold around NASA Goddard. A lot of (both frontline and upper) management has no technical background and it shows. JWST was a shambles when I interned there in 2014. Eternally delayed and broken due to nonsensical decisions by management with absolutely zero engineering knowledge. Sometime in the 90s they had decided to save money on devs by “auto-generating” the code from UML diagrams using this thing called Rational Rose. In the mid-2010s they were still trying to fix the code. I never quite figured out if it was an urban legend, but there was also a story around Goddard that when JSWT did its acoustic testing (to simulate launch conditions), a bunch of screws fell off—and when they put them back, they were left with several extra screws that nobody ever figured out the place for. So that’s the state the program was in. Greg Robinson seems to be an engineering manager who’s actually highly competent at both the engineering and the management, and that’s exactly what the program needed to turn around. Honestly, when I was there I thought there was very little chance this thing would actually work, if it ever launched at all, so major props to him and the team. I saw this in way more programs than just JWST. Word would get around among us contractors that X program had real technical management and Y didn’t, and we’d all try to get onto X. If you got stuck on Y, you’d spend most of your time doing paperwork and PowerPoint instead of coding. It happened in the civil service too. Most of the branches in the software division had non-technical leadership and seemed to do little of note, but the few with technical leaders did incredible work. I remember one time a previously-good branch had its head depart (promoted?) and the new head wasn’t technical at all, and over a few months most of the good civil servants transferred somewhere else. At one point it seemed like most of the progress in the software division was being driven by one rather obscure-sounding branch, simply because that was the one with sound technical leadership and thus all the good engineers.
    - wil421 3y ago
      Why are there so many non technical managers/directors in charge of engineering or IT in Government type jobs? My wife sells to state and local governments. 90% of the time the people in charge have no idea about IT, SecOps, Software Development etc. Their only qualification is they have engineering or security in their title.
      - otteromkram 3y ago
        Gatekeeping, probably. (Only half joking; the cult of personality in IT can be pretty toxic.) Plus, most people in tech got into their field because they like tech. Managing people isn't nearly as fun or rewarding to them.
      - BurningFrog 3y ago
        There is less pressure to perform well in government, so it becomes less prioritized than in organizations that may cease to exist if they perform badly.
      - TheOtherHobbes 3y ago
        It's not just government jobs.
        borg16 3y ago
        amen.
      - borg16 3y ago
        > Why are there so many non technical managers/directors in charge of engineering or IT in Government type jobs? due to product/program/project managers making a case for themselves to lead these teams. it might work in most of the cases, but clearly does not in engineering intensive projects - because the lens required to look at and deliver these projects is more than input/output, deadlines/deliverables. understanding the subject is necessary to make smart tradeoffs (common scenario in projects that are as large scale as this one), but few of these folks do. engineering enterprises that have often become successful owe it in their early days to majority of management having technical expertise. this is not a need once the companies become large, but such attitude does show up as inefficiency holes, if one knows where to look.
- man4 3y ago
  [flagged]
- hbarka 3y ago
  We have a fascination to find a hero or villain as a singular explanation. It makes for a a good article or Michael Lewis book. The JWST might have also succeeded because of the failures learned from the Hubble telescope.
man4 3y ago
[flagged]
troelsSteegin 3y ago
The related [0] NYTimes article mentions one pattern for fixes, which was finding gaps between design and execution: "For the screws that popped off during the shake test, it turned out that the engineering drawings did not specify how much torque needed to be applied. That was left to the contractor, Northrop Grumman, to decide, and they were not tight enough. “You should have a specification to make sure it’s right,” Mr. Robinson said. The review board released its report, noting a series of issues, and made 32 recommendations. NASA followed all of them, Mr. Robinson said. One of the recommendations was performing an audit of the entire spacecraft to identify “embedded problems” — mistakes that occurred without anyone noticing. The engineers checked the drawings and specifications. They looked at the purchase requests to make sure that what was ordered matched the specifications and that the suppliers provided the correct items. “There were multiple teams set up, led by the most experienced people,” Mr. Robinson said. “They really dug into the paperwork.”" [0] https://news.ycombinator.com/item?id=32109760 https://news.ycombinator.com/item?id=32109760
- TheJoeMan 3y ago
  I have found it's very easy to scapegoat the "engineers" for situations like this, but in the long term not the cause of the problems nor the actual solution. For example, screws have standard torques published, and for small common diameters is left implicit. The matter is, Northrop's blue collar assemblers didn't particularly care and just installed the screw however, and now the contractor is looking to shift blame. If the drawing did actually say "12 inch pounds" or however, it's still a coin flip if the screw was actually that tight, and the end result of the shake test would be identical. For an analogy, say the website designers just specified "add a button" and you went and made the button fluorescent yellow with white text. Then the design fails the color contrast test. Will you say it's entirely the designer's fault for not specifying the button color? Digging into the paperwork to the extent described may have the highest assurance level, but it is extremely expensive. This spacecraft in particular was quite over budget and most likely only able to be carried over the finish line by pure government inertia.
  - pas 3y ago
    > Northrop's blue collar assemblers ... they are not factory assembly line workers, right? > over budget this is the classic problem of too small overall budget. too few launches, not enough money, everything has to be perfect, because everyone wants their pet science/dream project to succeed. so overall efficiency is abysmal. but the data gained for/through the scientific mission is nice.
deleted 3y ago
[deleted]
justrealist 3y ago
I'm impressed by the meticulous detail here, but I still have to believe that if the original goal was to build 10 of these, (one every 2 years) and accept a 60% fail rate for each, we'd have gotten more telescopes for the same money in the end.
- OrvalWintermute 3y ago
  I don't believe Science Mission Directorate (SMD), which funds science related missions that JPL and GSFC compete for, has ever been so flush with cash as to be able to develop a JWST constellation of that size. We'd prolly need to raid the HEO budget and kill off some too-big-to-fail-missions in Texas to get the money for that....
ck2 3y ago
There is a great episode on PBS about putting JWST together and the hurricane it went through (literally) https://www.pbs.org/video/ultimate-space-telescope-gunryt/ https://www.pbs.org/video/ultimate-space-telescope-gunryt/ follow-up after launch https://www.pbs.org/video/new-eye-on-the-universe-zvzqn1/ https://www.pbs.org/video/new-eye-on-the-universe-zvzqn1/

93 points by EndXA 3y ago

https://archive.md/AEzFQ https://archive.md/AEzFQ

dang 3y ago

Related: Greg Robinson fixed NASA’s James Webb Space Telescope, reluctantly - https://news.ycombinator.com/item?id=32109760 https://news.ycombinator.com/item?id=32109760 - July 2022 (199 comments)

WalterBright 3y ago

The article seems a little short on specifics of what he did. I'd like to know more!

icapybara 3y ago
It sounds like he was/is a personable engineering manager that paid attention to detail. Making sure parts match the drawings, bringing in outsiders to have a second look at things, etc.
- bryananderson 3y ago
  Basically this, and it’s a bigger deal than it may sound. Competent technical management is a scarce commodity worth its weight in gold around NASA Goddard. A lot of (both frontline and upper) management has no technical background and it shows. JWST was a shambles when I interned there in 2014. Eternally delayed and broken due to nonsensical decisions by management with absolutely zero engineering knowledge. Sometime in the 90s they had decided to save money on devs by “auto-generating” the code from UML diagrams using this thing called Rational Rose. In the mid-2010s they were still trying to fix the code. I never quite figured out if it was an urban legend, but there was also a story around Goddard that when JSWT did its acoustic testing (to simulate launch conditions), a bunch of screws fell off—and when they put them back, they were left with several extra screws that nobody ever figured out the place for. So that’s the state the program was in. Greg Robinson seems to be an engineering manager who’s actually highly competent at both the engineering and the management, and that’s exactly what the program needed to turn around. Honestly, when I was there I thought there was very little chance this thing would actually work, if it ever launched at all, so major props to him and the team. I saw this in way more programs than just JWST. Word would get around among us contractors that X program had real technical management and Y didn’t, and we’d all try to get onto X. If you got stuck on Y, you’d spend most of your time doing paperwork and PowerPoint instead of coding. It happened in the civil service too. Most of the branches in the software division had non-technical leadership and seemed to do little of note, but the few with technical leaders did incredible work. I remember one time a previously-good branch had its head depart (promoted?) and the new head wasn’t technical at all, and over a few months most of the good civil servants transferred somewhere else. At one point it seemed like most of the progress in the software division was being driven by one rather obscure-sounding branch, simply because that was the one with sound technical leadership and thus all the good engineers.
  - wil421 3y ago
    Why are there so many non technical managers/directors in charge of engineering or IT in Government type jobs? My wife sells to state and local governments. 90% of the time the people in charge have no idea about IT, SecOps, Software Development etc. Their only qualification is they have engineering or security in their title.
    - otteromkram 3y ago
      Gatekeeping, probably. (Only half joking; the cult of personality in IT can be pretty toxic.) Plus, most people in tech got into their field because they like tech. Managing people isn't nearly as fun or rewarding to them.
    - BurningFrog 3y ago
      There is less pressure to perform well in government, so it becomes less prioritized than in organizations that may cease to exist if they perform badly.
    - TheOtherHobbes 3y ago
      It's not just government jobs.
      - borg16 3y ago
        amen.
    - borg16 3y ago
      > Why are there so many non technical managers/directors in charge of engineering or IT in Government type jobs? due to product/program/project managers making a case for themselves to lead these teams. it might work in most of the cases, but clearly does not in engineering intensive projects - because the lens required to look at and deliver these projects is more than input/output, deadlines/deliverables. understanding the subject is necessary to make smart tradeoffs (common scenario in projects that are as large scale as this one), but few of these folks do. engineering enterprises that have often become successful owe it in their early days to majority of management having technical expertise. this is not a need once the companies become large, but such attitude does show up as inefficiency holes, if one knows where to look.
man4 3y ago
[flagged]
hbarka 3y ago
We have a fascination to find a hero or villain as a singular explanation. It makes for a a good article or Michael Lewis book. The JWST might have also succeeded because of the failures learned from the Hubble telescope.

man4 3y ago

[flagged]

troelsSteegin 3y ago

The related [0] NYTimes article mentions one pattern for fixes, which was finding gaps between design and execution: "For the screws that popped off during the shake test, it turned out that the engineering drawings did not specify how much torque needed to be applied. That was left to the contractor, Northrop Grumman, to decide, and they were not tight enough. “You should have a specification to make sure it’s right,” Mr. Robinson said. The review board released its report, noting a series of issues, and made 32 recommendations. NASA followed all of them, Mr. Robinson said. One of the recommendations was performing an audit of the entire spacecraft to identify “embedded problems” — mistakes that occurred without anyone noticing. The engineers checked the drawings and specifications. They looked at the purchase requests to make sure that what was ordered matched the specifications and that the suppliers provided the correct items. “There were multiple teams set up, led by the most experienced people,” Mr. Robinson said. “They really dug into the paperwork.”" [0] https://news.ycombinator.com/item?id=32109760 https://news.ycombinator.com/item?id=32109760

TheJoeMan 3y ago
I have found it's very easy to scapegoat the "engineers" for situations like this, but in the long term not the cause of the problems nor the actual solution. For example, screws have standard torques published, and for small common diameters is left implicit. The matter is, Northrop's blue collar assemblers didn't particularly care and just installed the screw however, and now the contractor is looking to shift blame. If the drawing did actually say "12 inch pounds" or however, it's still a coin flip if the screw was actually that tight, and the end result of the shake test would be identical. For an analogy, say the website designers just specified "add a button" and you went and made the button fluorescent yellow with white text. Then the design fails the color contrast test. Will you say it's entirely the designer's fault for not specifying the button color? Digging into the paperwork to the extent described may have the highest assurance level, but it is extremely expensive. This spacecraft in particular was quite over budget and most likely only able to be carried over the finish line by pure government inertia.
- pas 3y ago
  > Northrop's blue collar assemblers ... they are not factory assembly line workers, right? > over budget this is the classic problem of too small overall budget. too few launches, not enough money, everything has to be perfect, because everyone wants their pet science/dream project to succeed. so overall efficiency is abysmal. but the data gained for/through the scientific mission is nice.

deleted 3y ago

[deleted]

justrealist 3y ago

I'm impressed by the meticulous detail here, but I still have to believe that if the original goal was to build 10 of these, (one every 2 years) and accept a 60% fail rate for each, we'd have gotten more telescopes for the same money in the end.

OrvalWintermute 3y ago
I don't believe Science Mission Directorate (SMD), which funds science related missions that JPL and GSFC compete for, has ever been so flush with cash as to be able to develop a JWST constellation of that size. We'd prolly need to raid the HEO budget and kill off some too-big-to-fail-missions in Texas to get the money for that....

ck2 3y ago

There is a great episode on PBS about putting JWST together and the hurricane it went through (literally) https://www.pbs.org/video/ultimate-space-telescope-gunryt/ https://www.pbs.org/video/ultimate-space-telescope-gunryt/ follow-up after launch https://www.pbs.org/video/new-eye-on-the-universe-zvzqn1/ https://www.pbs.org/video/new-eye-on-the-universe-zvzqn1/