3 ms·

I started in HPC about 2 years ago on a ~500 node cluster at a Fortune 100 company. I was really just looking for a job where I was doing Linux 100% of the time

I started in HPC about 2 years ago on a ~500 node cluster at a Fortune 100 company. I was really just looking for a job where I was doing Linux 100% of the time, and it's been fun so far. But it wasn't what I thought it would be. I guess I expected to be doing more performance oriented work, analyzing numbers and trying to get every last bit of performance out of the cluster. To be honest, they didn't even have any kind of monitoring running. I set some up, and it doesn't really get used. Once in a while we get questions from management about "how busy is the cluster", to justify budgets and that sort of thing. Most of my 'optimization' work ends up being things like making sure people aren't (usually unknowingly) requesting 384 CPUs when their script only uses 16, testing software to see what # of CPU's it works with before you see a degradation, etc. I've only had the Intel profiler open twice. And I've found that most of the job is really just helping researchers and such with their work. Typically running either a commercial or open-source program, troubleshooting it, or getting some code written by another team on another cluster and getting it built and running on yours. Slogging through terrible Python code. Trying to get a C++ project built on a more modern cluster in a CentOS 7 environment. It can be fun in a way. I've worked with different languages over the years so I enjoy trying to get things working, digging through crashes and stack traces. And working with such large machines, your sense of normal gets twisted when you're on a server with 'only' 128GB of RAM or 20TB of disk. It's a little scary when you know the results of some of this stuff are being used in the real world, and the people running the simulations aren't even doing things right. Incorrect code, mixed up source code, not using the data they thing they are, I once found a huge bug that had existed for 3 years. Doesn't this invalidate all the work you've done on this subject? The one drawback I find is that a lot of HPC jobs want you do have a masters degree. Even to just run the cluster. Doesn't make sense to me, I'm not writing the software you're running, we aren't running some state of the art, TOP500 cluster. We're just getting a bunch of machines networked together and running some code.

throwawaaarrgh 3y ago
I always found that funny too. A business who needs a powerful computing solution can come up with some amazingly robust stuff, whereas science/research just buys a big mainframe and hopes it works.
- s_Hogg 3y ago
  I was working in a company that had been spun out of a university until recently and it was shocking how hopeless the researchers were. I've always been critical of how poor the job security in academia is but you'd think it's still too much given how slapdash some of the crap you see is. We basically had to reinvent their product from the ground up, awful.
  - danparsonson 3y ago
    This is probably a naive question but isn't that the point of having developers on staff? The researchers aren't coders and vice versa, so having researchers produce prototypes that are productized by engineers makes sense to me.
    - rramadass 3y ago
      Exactly! This is how it should be. Researchers/Scientists with their hard earned PhDs should only concentrate on doing cutting-edge "researchy" stuff. It is hard enough that they should not be asked to learn all the intricacies/problems inherent in Software Development. That is the domain of a "Professional Software Engineer". There is now in fact a new class called "Research Software Engineer" who are Software Developers working in Research developing code specific to their needs - https://www.nature.com/articles/d41586-022-01516-2 https://www.nature.com/articles/d41586-022-01516-2 and https://en.wikipedia.org/wiki/Research_software_engineering https://en.wikipedia.org/wiki/Research_software_engineering
  - m-ee 3y ago
    I've had very similar experiences working with former researchers including at a university spinout. Mechanical rather than CS. It was perplexing how they still carried the elitism that industry was mostly for people who can't hack it in academia given the quality of their work. Would be unacceptable coming from a new hire PD engineer at Apple yet you're demanding respect because you used to lead a whole lab apparently producing rubbish?
justin66 3y ago
> The one drawback I find is that a lot of HPC jobs want you do have a masters degree. Is it possible that pretty much any specialization, outside of the most common ones, engages in a lot of gatekeeping? I remember how difficult it appeared to be after I graduated to break into embedded systems (I never did). I persisted until I realized it doesn't even pay very well, comparatively.
- johnnyanmac 3y ago
  Yes, but to varying degrees. I imagine the whole "fortune 500" deal probably gatekept more than it really needed to. While I don't think any specialization NEEDS a masters since 2-3 years of industry work in that field will do just as much 90% of the tie (a few may need PhD's, mostly for R&D labs), some can justify it more than others. It's also cultural. From what I hear, the east cost US cares a lot more about prestige than the west coast that focuses a lot more on performance.
od0 3y ago
would love to connect and talk more about HPC - let me know if you'd be up for a chat :)
- bluedino 3y ago
  Emails in my profile, feel free to reach out

throwawaaarrgh 3y ago

I always found that funny too. A business who needs a powerful computing solution can come up with some amazingly robust stuff, whereas science/research just buys a big mainframe and hopes it works.

s_Hogg 3y ago
I was working in a company that had been spun out of a university until recently and it was shocking how hopeless the researchers were. I've always been critical of how poor the job security in academia is but you'd think it's still too much given how slapdash some of the crap you see is. We basically had to reinvent their product from the ground up, awful.
- danparsonson 3y ago
  This is probably a naive question but isn't that the point of having developers on staff? The researchers aren't coders and vice versa, so having researchers produce prototypes that are productized by engineers makes sense to me.
  - rramadass 3y ago
    Exactly! This is how it should be. Researchers/Scientists with their hard earned PhDs should only concentrate on doing cutting-edge "researchy" stuff. It is hard enough that they should not be asked to learn all the intricacies/problems inherent in Software Development. That is the domain of a "Professional Software Engineer". There is now in fact a new class called "Research Software Engineer" who are Software Developers working in Research developing code specific to their needs - https://www.nature.com/articles/d41586-022-01516-2 https://www.nature.com/articles/d41586-022-01516-2 and https://en.wikipedia.org/wiki/Research_software_engineering https://en.wikipedia.org/wiki/Research_software_engineering
- m-ee 3y ago
  I've had very similar experiences working with former researchers including at a university spinout. Mechanical rather than CS. It was perplexing how they still carried the elitism that industry was mostly for people who can't hack it in academia given the quality of their work. Would be unacceptable coming from a new hire PD engineer at Apple yet you're demanding respect because you used to lead a whole lab apparently producing rubbish?

justin66 3y ago

> The one drawback I find is that a lot of HPC jobs want you do have a masters degree. Is it possible that pretty much any specialization, outside of the most common ones, engages in a lot of gatekeeping? I remember how difficult it appeared to be after I graduated to break into embedded systems (I never did). I persisted until I realized it doesn't even pay very well, comparatively.

johnnyanmac 3y ago
Yes, but to varying degrees. I imagine the whole "fortune 500" deal probably gatekept more than it really needed to. While I don't think any specialization NEEDS a masters since 2-3 years of industry work in that field will do just as much 90% of the tie (a few may need PhD's, mostly for R&D labs), some can justify it more than others. It's also cultural. From what I hear, the east cost US cares a lot more about prestige than the west coast that focuses a lot more on performance.

od0 3y ago

would love to connect and talk more about HPC - let me know if you'd be up for a chat :)

bluedino 3y ago
Emails in my profile, feel free to reach out