6 ms·

I, too, am a software engineer dealing with researcher code on a daily basis, and I'm also the person that oversees most of our research compute hardware [sysad

I, too, am a software engineer dealing with researcher code on a daily basis, and I'm also the person that oversees most of our research compute hardware [sysadmin in a past life]. Some of my other favorite gems: * "I need a terabyte of RAM" * "50k samples ran in the blink of an eye. 100k ran in an hour. 1 million took a week. I'm about to do the full 50 million observations, I expect it'll take ten days" * "This code is always slow... that's a lot of math it does" * "Profiler? Never heard of such a thing" * "Big-O? What's that?" * "I only have ten days' money, and it took me that long. Afraid I can't afford your time to dig into this" * "AWS will fix this" Generally I'm dealing with the intersection of two or more of these. I do think, though, that the last paragraph is a bit unfair. Literally everyone I work with is incredibly smart; just their background isn't the same as mine. Things that I find incredibly tough are idiotic to them ["Can't I just take the average and be done with it?" "no"]. So, for most of the people I work with, it's not "common sense"; they literally lack the frame of reference to even know what's awry. I never hold it against anyone, and nor should you. My favorite example is from many years ago, a guy who had some code that took a month, "because that's just complicated math, it's how long that takes". I ran it through a profiler, and after about ten minutes work, it ran in seconds. I came out looking like a hero.

arthurcolle 5y ago
Terabyte of RAM is my favorite because I choose to believe he had a perfect data model and was literally telling you exactly what he needed :D In my recent work doing financial modeling (options, cryptos, other nonsense), I pretty much just try to cache absolutely everything with some arbitrary 1 week TTL and then continue to cache new data every day. Cache hits cause a 10% bump in the current TTL time delta. Kinda heuristic but whatever, works in most cases. Any other funny stories? I always love these engineering horror stories. Have you looked at Julia at all? Would be interesting to hear if there's been an uptick in usage from the research community from a veteran research code surgeon!
- chunkyks 5y ago
  The mathematician that designed the model I referred to at the end ended up passing it on to me. I've now built something of an empire of people and projects using it, "just" by making better tools to manage its inputs, outputs, visualisation and automation. I was stupidly excited to recently do more work on that same model - I took one of its internal datastructures and turned it from a simple queue into a priority queue using a couple different queueing metrics. In both cases, I had run cases out to a scale where I was able to actually show going from O(1) to O(log(n)) performance - but got beautiful wins because the fastest work is work you never have to do. Not often I can empirically show traditional CS101 stuff in a real-world chart.
  - dirkt 5y ago
    Your work sounds awesome. (And you probably mean O(n) and not O(1), otherwise O(log(n)) isn't really an improvement).
    - chunkyks 5y ago
      No, that's the point; the previous algorithm was O(1) [a simple queue] but that led to doing a huge amount of extra work. By using a smarter queue that "cost" an extra O(log(n)), we could skip work that didn't need doing in the first place.
chunkyks 5y ago
Oh, and not to tar with toooo broad a brush: One guy contacted me the other day wanting a terabyte of memory. He'd actually done all his homework and had a weirdly interesting use case. I'm actually pretty excited to explore getting his problem solved; it's likely that a terabyte is about the right amount to get him over the finish line, and he's converged on this algorithm precisely because it has fairly horrifying memory needs, but trades off to give a nice big-O runtime.
- arthurcolle 5y ago
  I've heard that some financial institutions use a YARC supercomputer with 256TB of RAM to do daily "full market watch" of client activity to determine insider trading for compliance purposes. I can't even imagine the power draw of such a system or the heat generation. Fucking galaxy brain engineering. At that point you need like custom motherboards and special buses, right?
  - chunkyks 5y ago
    Finance guys tend to be able to afford to have the right people to be sure of that on hand; they also are specifically in the business of cost-benefit analysis. I have no insight or information this; but sometimes you can throw money at a problem to make it go away, and that's literally the best deal open to you. If custom fab solves a problem more cheaply than the 25 senior software engineers it would otherwise take, then custom fab it is.
  - pinewurst 5y ago
    That’s about a 2,000 node Cray, something like 14 cabinets which isn’t that big of an XC configuration.
  - jsjohnst 5y ago
    > I can't even imagine the power draw of such a system or the heat generation. I’m unfamiliar with the machine specified in your example, but I’ve seen in person HPC clusters with more RAM than that, so I’ll use that to give you some napkin math calculations. Dell R730 with 24x64gb ECC DIMMs = 1536gb RAM. To get to 256TB, you’d need about 170 of them (aka about 5 racks). Power draw varies depending on a number of factors, but let’s be very conservative and take a worse case and say 750W per 1U server. That’s 127,500W, which is just over 600A at 208V, so say a long block of houses with their A/Cs cranking in the summer. Now as to the heat generation, each R730 puts out up to 2850 BTU/hr, so for all 170 of them, that’s just shy of 500,000 BTU/hr. Continuing the comparison to homes, that’s 5-6 home furnaces worth of heat generation per hour. I don’t have any photos handy of the 64gb DIMMs referenced above, but here’s a photo of 130 16gb ECC DIMMs (aka just over 2TB): https://www.dropbox.com/s/ww1fwdn7o9se14z/2020-06-15%2020.39.31.jpg?dl=0 https://www.dropbox.com/s/ww1fwdn7o9se14z/2020-06-15%2020.39... If instead of the above based on a real world HPC machine, you based it on current gen Dell R750s which can hold 32 256gb LRDIMMs totaling 8TB per machine, you could get to by with just over a single rack of machines for 256TB, albeit for a much higher price (256gb LRDIMMs cost as much each as roughly the price of a used R730 complete w/ 1.5TB of RAM). The cluster of R750s with that much RAM configured as indicated would be well into the 8 figure range.
arthurcolle 5y ago
What kind of profilers are you using? If you can list a mapping between language and profiler name, that would be sweet
- chunkyks 5y ago
  The stuff I've used in recent months: - For Java, personally I like the netbeans profiler - Octave's built in profiler gets the job done: https://octave.org/doc/v6.2.0/Profiling.html - I use JetBrain's tooling a lot. - I like the profiler in Rider for VB.NET and C# - The one in PyCharm is ... fine. Python profiling is ugly, PyCharm does a clean job handling it. - For R, Rprof is ... fine: https://stat.ethz.ch/R-manual/R-devel/library/utils/html/Rprof.html - STATA's built in profiler does as good a job as possible given it's a weird language For various other things, where a profiler doesn't really exist per se: - SAS can be configured to output runtime after each proc. There's no cost, you should do it - SAS proc SQL has an undocumented _method flag: http://support.sas.com/techsup/technote/ts553.html - Past "EXPLAIN QUERY PLAN", one of the tools I build ends up doing a simple visualisation of SQLite's explain query plan output for *every* query it runs. There's no way to *not* get that estimate Those are the tools I can think of that I've used in the last few months.
  - arthurcolle 5y ago
    I appreciate you taking the time to write this out. I'm curious about the lack of any mention of C/C++ tooling - I thought there are some mathematicians who write stuff in C/C++ and leverage libs like GNU SL (Scientific Library) etc. I've written a few custom pricers and needed to use those libs and they were pretty efficient, and didn't need to resort to anything crazy to run them in production. Curious if you ever find stuff written in those languages, as opposed to the gamut you just mention, which are classical "research tools" (language/IDE combinations that let you not need to think about the language specifically.
    - chunkyks 5y ago
      I don't cross paths with a lot of C or C++ where profiling is a concern. I use gprof for that, but I haven't needed it in a few years. None of the folks I work with reach for C or C++ as a weapon in their quiver. I do cross paths with a fair bit of FORTRAN, but in my experience, generally if someone's writing FORTRAN, they already know the deal with performance. Fun fact: When I interviewed for my job [12 years ago], I interviewed with a guy using SQLite and FLTK in a C++ model. I'd been using SQLite and FLTK for some recent personal projects; in the interview, I solved a bunch of performance problems he'd been having for a couple years. I interviewed on Monday, and started working there two days later.
brundolf 5y ago
That sounds like a pretty satisfying job. How'd you find it?
- chunkyks 5y ago
  This place has been an astonishingly good fit for me, yeah. I was laid off from my last job, and a friend passed along my resume at an opportune moment. That was twelve years ago, and I'm still here. My group has a perpetually open posting and are always looking for people: https://rand.wd5.myworkdayjobs.com/en-US/External_Career_Site/job/Santa-Monica-CA-Greater-Los-Angeles-Area/Research-Software-Engineer_R628 https://rand.wd5.myworkdayjobs.com/en-US/External_Career_Sit... [If the link doesn't work, go to rand.org, search jobs, then search for "research software engineer"] EDIT: That link is to a specific posting. Our perpetual one is going back up any day now, once we get through the machinations.
esalman 5y ago
I share the same feeling. I am a PhD student with working knowledge of application development. My skills have been quite useful to the rest of the lab as they transitioned to using HPC systems. > * "I need a terabyte of RAM" A lot of our work involve running algorithms such as PCA and ICA on thousands of biomedical imaging data. It is totally normal for us to allocation 1TB RAM. We have 3 nodes with 1.5TB RAM in our cluster. Aapproximation algorithms do exist and are used for resource optimization though.
- chunkyks 5y ago
  To be clear, sometimes a terabyte of RAM is the right solution. But it's pretty common to find someone needing that simply needs a better algorithm. [or, recently, a mathematical identity that meant the task could be done streamed from disk]
  - dgb99 5y ago
    Tbh if you need a terrabyte of RAM to simply run your experiment then you aren’t doing it right. We haven’t reached a level where an algorithm would require at least a terrabyte of RAM just to get baseline results.

arthurcolle 5y ago

Terabyte of RAM is my favorite because I choose to believe he had a perfect data model and was literally telling you exactly what he needed :D In my recent work doing financial modeling (options, cryptos, other nonsense), I pretty much just try to cache absolutely everything with some arbitrary 1 week TTL and then continue to cache new data every day. Cache hits cause a 10% bump in the current TTL time delta. Kinda heuristic but whatever, works in most cases. Any other funny stories? I always love these engineering horror stories. Have you looked at Julia at all? Would be interesting to hear if there's been an uptick in usage from the research community from a veteran research code surgeon!

chunkyks 5y ago
The mathematician that designed the model I referred to at the end ended up passing it on to me. I've now built something of an empire of people and projects using it, "just" by making better tools to manage its inputs, outputs, visualisation and automation. I was stupidly excited to recently do more work on that same model - I took one of its internal datastructures and turned it from a simple queue into a priority queue using a couple different queueing metrics. In both cases, I had run cases out to a scale where I was able to actually show going from O(1) to O(log(n)) performance - but got beautiful wins because the fastest work is work you never have to do. Not often I can empirically show traditional CS101 stuff in a real-world chart.
- dirkt 5y ago
  Your work sounds awesome. (And you probably mean O(n) and not O(1), otherwise O(log(n)) isn't really an improvement).
  - chunkyks 5y ago
    No, that's the point; the previous algorithm was O(1) [a simple queue] but that led to doing a huge amount of extra work. By using a smarter queue that "cost" an extra O(log(n)), we could skip work that didn't need doing in the first place.

chunkyks 5y ago

Oh, and not to tar with toooo broad a brush: One guy contacted me the other day wanting a terabyte of memory. He'd actually done all his homework and had a weirdly interesting use case. I'm actually pretty excited to explore getting his problem solved; it's likely that a terabyte is about the right amount to get him over the finish line, and he's converged on this algorithm precisely because it has fairly horrifying memory needs, but trades off to give a nice big-O runtime.

arthurcolle 5y ago
I've heard that some financial institutions use a YARC supercomputer with 256TB of RAM to do daily "full market watch" of client activity to determine insider trading for compliance purposes. I can't even imagine the power draw of such a system or the heat generation. Fucking galaxy brain engineering. At that point you need like custom motherboards and special buses, right?
- chunkyks 5y ago
  Finance guys tend to be able to afford to have the right people to be sure of that on hand; they also are specifically in the business of cost-benefit analysis. I have no insight or information this; but sometimes you can throw money at a problem to make it go away, and that's literally the best deal open to you. If custom fab solves a problem more cheaply than the 25 senior software engineers it would otherwise take, then custom fab it is.
- pinewurst 5y ago
  That’s about a 2,000 node Cray, something like 14 cabinets which isn’t that big of an XC configuration.
- jsjohnst 5y ago
  > I can't even imagine the power draw of such a system or the heat generation. I’m unfamiliar with the machine specified in your example, but I’ve seen in person HPC clusters with more RAM than that, so I’ll use that to give you some napkin math calculations. Dell R730 with 24x64gb ECC DIMMs = 1536gb RAM. To get to 256TB, you’d need about 170 of them (aka about 5 racks). Power draw varies depending on a number of factors, but let’s be very conservative and take a worse case and say 750W per 1U server. That’s 127,500W, which is just over 600A at 208V, so say a long block of houses with their A/Cs cranking in the summer. Now as to the heat generation, each R730 puts out up to 2850 BTU/hr, so for all 170 of them, that’s just shy of 500,000 BTU/hr. Continuing the comparison to homes, that’s 5-6 home furnaces worth of heat generation per hour. I don’t have any photos handy of the 64gb DIMMs referenced above, but here’s a photo of 130 16gb ECC DIMMs (aka just over 2TB): https://www.dropbox.com/s/ww1fwdn7o9se14z/2020-06-15%2020.39.31.jpg?dl=0 https://www.dropbox.com/s/ww1fwdn7o9se14z/2020-06-15%2020.39... If instead of the above based on a real world HPC machine, you based it on current gen Dell R750s which can hold 32 256gb LRDIMMs totaling 8TB per machine, you could get to by with just over a single rack of machines for 256TB, albeit for a much higher price (256gb LRDIMMs cost as much each as roughly the price of a used R730 complete w/ 1.5TB of RAM). The cluster of R750s with that much RAM configured as indicated would be well into the 8 figure range.

arthurcolle 5y ago

What kind of profilers are you using? If you can list a mapping between language and profiler name, that would be sweet

chunkyks 5y ago
The stuff I've used in recent months: - For Java, personally I like the netbeans profiler - Octave's built in profiler gets the job done: https://octave.org/doc/v6.2.0/Profiling.html - I use JetBrain's tooling a lot. - I like the profiler in Rider for VB.NET and C# - The one in PyCharm is ... fine. Python profiling is ugly, PyCharm does a clean job handling it. - For R, Rprof is ... fine: https://stat.ethz.ch/R-manual/R-devel/library/utils/html/Rprof.html - STATA's built in profiler does as good a job as possible given it's a weird language For various other things, where a profiler doesn't really exist per se: - SAS can be configured to output runtime after each proc. There's no cost, you should do it - SAS proc SQL has an undocumented _method flag: http://support.sas.com/techsup/technote/ts553.html - Past "EXPLAIN QUERY PLAN", one of the tools I build ends up doing a simple visualisation of SQLite's explain query plan output for *every* query it runs. There's no way to *not* get that estimate Those are the tools I can think of that I've used in the last few months.
- arthurcolle 5y ago
  I appreciate you taking the time to write this out. I'm curious about the lack of any mention of C/C++ tooling - I thought there are some mathematicians who write stuff in C/C++ and leverage libs like GNU SL (Scientific Library) etc. I've written a few custom pricers and needed to use those libs and they were pretty efficient, and didn't need to resort to anything crazy to run them in production. Curious if you ever find stuff written in those languages, as opposed to the gamut you just mention, which are classical "research tools" (language/IDE combinations that let you not need to think about the language specifically.
  - chunkyks 5y ago
    I don't cross paths with a lot of C or C++ where profiling is a concern. I use gprof for that, but I haven't needed it in a few years. None of the folks I work with reach for C or C++ as a weapon in their quiver. I do cross paths with a fair bit of FORTRAN, but in my experience, generally if someone's writing FORTRAN, they already know the deal with performance. Fun fact: When I interviewed for my job [12 years ago], I interviewed with a guy using SQLite and FLTK in a C++ model. I'd been using SQLite and FLTK for some recent personal projects; in the interview, I solved a bunch of performance problems he'd been having for a couple years. I interviewed on Monday, and started working there two days later.

brundolf 5y ago

That sounds like a pretty satisfying job. How'd you find it?

chunkyks 5y ago
This place has been an astonishingly good fit for me, yeah. I was laid off from my last job, and a friend passed along my resume at an opportune moment. That was twelve years ago, and I'm still here. My group has a perpetually open posting and are always looking for people: https://rand.wd5.myworkdayjobs.com/en-US/External_Career_Site/job/Santa-Monica-CA-Greater-Los-Angeles-Area/Research-Software-Engineer_R628 https://rand.wd5.myworkdayjobs.com/en-US/External_Career_Sit... [If the link doesn't work, go to rand.org, search jobs, then search for "research software engineer"] EDIT: That link is to a specific posting. Our perpetual one is going back up any day now, once we get through the machinations.

esalman 5y ago

I share the same feeling. I am a PhD student with working knowledge of application development. My skills have been quite useful to the rest of the lab as they transitioned to using HPC systems. > * "I need a terabyte of RAM" A lot of our work involve running algorithms such as PCA and ICA on thousands of biomedical imaging data. It is totally normal for us to allocation 1TB RAM. We have 3 nodes with 1.5TB RAM in our cluster. Aapproximation algorithms do exist and are used for resource optimization though.

chunkyks 5y ago
To be clear, sometimes a terabyte of RAM is the right solution. But it's pretty common to find someone needing that simply needs a better algorithm. [or, recently, a mathematical identity that meant the task could be done streamed from disk]
- dgb99 5y ago
  Tbh if you need a terrabyte of RAM to simply run your experiment then you aren’t doing it right. We haven’t reached a level where an algorithm would require at least a terrabyte of RAM just to get baseline results.