5 ms·

Oh hey, I worked on this. Paper with all the details here: https://arxiv.org/pdf/1801.10277.pdf https://arxiv.org/pdf/1801.10277.pdf. Happy to answer questions.

philipkglass 8y ago
Finishing the whole job in under 15 minutes is impressive but also a bit suspicious, in a way. Back when I used a top 10 HPC facility at a national lab I saw a lot of jobs in the queue that could have run just fine on business class servers without the fancy, expensive interconnects needed for Grand Challenge problems. People didn't run their jobs on smaller machines because there was money in the budget to build a world class Top 10 computing resource but not money to buy less exotic hardware matched to typical jobs. Individual research groups didn't really get their typical compute needs measured or surveyed. These are obviously leading questions, but: - Are there significant research advantages to super-fast turnaround (less than 15 minutes, enabled by massive parallelism) in this domain? - Do you feel like a massively parallel system with Xeon Phi nodes is a good match for this problem? Or did the code get optimized to run at high scale on Cori Phase II because that's where you were given compute resources? Finally, a bit less provocative: Does this approach effectively scale down to e.g. a university that can afford a large storage array and beefy commercial servers (optionally equipped with Phi or other accelerators), but doesn't have HPC resources that are contenders for the Top 500? Or do you really need things that smaller systems can't deliver, like many terabytes of memory in distributed global arrays?
- KenoFischer 8y ago
  | Are there significant research advantages to super-fast turnaround (less than 15 minutes, enabled by massive parallelism) in this domain? No, the 15 minute turn around time is not important given the dataset we have at the moment, but showing that it was possible to do was considered important from a science perspective, because of the upcoming LSST telescope. LSST will generate an amount of data equivalent to the full dataset we had available every 3-4 days, so being able to scale this up far enough to accommodate that as well as future planned extensions to the algorithm, necessitated showing scale. The actual science runs by the project are usually done on a few hundred nodes over a couple of hours. | Do you feel like a massively parallel system with Xeon Phi nodes is a good match for this problem? Or did the code get optimized to run at high scale on Cori Phase II because that's where you were given compute resources? Cori Phase II worked well for this problem, though I wouldn't be surprised if GPUs wouldn't have been a better fit (though harder to program of course and at the time the Julia GPU infrastructure probably wasn't quite ready yet - even KNL was a struggle since LLVM was still in the process of completing support for it). The Celeste project is still ongoing (working on science goals more so than extra parallelism or performance improvements at the moment), but I wouldn't be surprised if there was an attempt to run on Summit at some point, especially now that Julia's GPU compiler is much more mature. One of the biggest problems we failed to anticipate actually was getting the data from disk to compute units quickly enough. Early in the project we crashed the interconnect on the machine, so for the challenge run we weren't allowed to do anything other than pull the data directly from disk (lest we bring down the machine again while other challenge runs were ongoing). I haven't really looked at the interconnect on Summit, so I can't say how well it would handle that. | Does this approach effectively scale down to e.g. a university that can afford a large storage array and beefy commercial servers Yes, it scales fairly well. In fact you could probably do it with spot instances on a public could fairly well. The biggest thing would once again be getting the data to the compute units quick enough. That's quite demanding on the network (and ideally you want to pre-stage the data in memory). Certainly it's feasible to do this on a large-ish university cluster on the SDSS data set in a few hours. Probably less feasible on LSST data once that comes online, but maybe by that point improvements in computation speed and storage speed will have made up for that and it'll become feasible again.
  - pmalynin 8y ago
    >No, the 15 minute turn around time is not important given the dataset we have at the moment, but showing that it was possible to do was considered important from a science perspective, because of the upcoming LSST telescope. LSST will generate an amount of data equivalent to the full dataset we had available every 3-4 days, so being able to scale this up far enough to accommodate that as well as future planned extensions to the algorithm, necessitated showing scale. The actual science runs by the project are usually done on a few hundred nodes over a couple of hours. I've heard that the folks working on the EHT array need months to crunch numbers. Could something like this be used to speed up that process? Or is there some other reason that would prohibit that. P.S. I want pictures of black holes.
    - KenoFischer 8y ago
      I don't know. I'd imagine that the folks working on the EHT already are making use of plenty of HPC for image reconstruction. It's quite a different problem from the Celeste application of course, so this work isn't directly applicable, but if they ever wanted to rewrite their code in Julia, they should give us a shout ;).
- deleted 8y ago
  [deleted]
noobermin 8y ago
Very low key question that might be in your paper but I'm lazy: Did you use one of the compilation options for this or just the JIT compilation (if direct compilation is available, I'll be honest, I'm not that abreast of Julia developments). One of the key things for more computational simulation tools (as opposed to just analysis) is we find that the awesome compilers for C and Fortran codes are a huge boon, especially on systems that have ones that improve on stock intel (not to mention gcc) by a good factor. You lose that if you're just using the jit for the sake of using a "nicer" language, I guess. I had similar sentiments to the other cat who asked about the 15 minute run, but I guess it's fun to show it "can be done." I was planning to play a little in Julia and this shows it can be worth it.
- KenoFischer 8y ago
  The JIT compiler was active, but most of the computationally heavy parts were ahead of time compiled. The compiler had a number of tweaks to increase the quality of the generated code (though at the expense of a good amount of compile time). Generating the ahead of time image probably took a good 5-10 minutes (much more if we had tried to do it on the KNL nodes that we actually ran on), so it was infeasible and would have been a waste of time to try to do so in a JITted fashion. Some JIT compilation probably still happened (for code paths that weren't in the pre-compilation trace, e.g. logging code or exception code paths hit by a few processes).
  - noobermin 8y ago
    For some reason, my brain forgot you did this on KNL. I guess even with cross compilation and then passing it to the platform compilers and it performing well is a sign it's good. Anyway, it's great you guys got this through especially with a tool like Julia. At least in my corner of academia, people utilizing "untested tools" unnecessarily raises the ire and skepticism of peers (for rather stupid reasons). I appreciate you guys "making the first move" which makes it easier for the rest of us looking to branch out and use these tools for our own codes.
    - KenoFischer 8y ago
mannermachine 8y ago
This is really neat, I used to do research in Bayesian networks, I had no idea they were being applied to astronomy, is their use widespread?
- KenoFischer 8y ago
  I worked on the computational side of this, so while I understand the science, I don't have a very good overview on the state of the art in astronomy. However, my understanding from the statisticians and astronomers on the project is that variational inference had not been attempted on this scale before (in Astronomy or otherwise).
SiempreViernes 8y ago
Does this make it possible to get closer to David Hogg's and Dustin Lang's "theory of everything"? (Is that the same D Lang as in the Celeste paper?) That is the idea that a catalogue is released as a complete model of the sky, rather than as a table of intensities and coordinates. (https://arxiv.org/pdf/0810.3851.pdf https://arxiv.org/pdf/0810.3851.pdf)
- KenoFischer 8y ago
  I don't feel qualified to speculate on the implications for astronomy because that's not my field. However, Celeste is indeed a generative model and you can draw an image sample from the Celeste model and compare it to the original (there's a few examples in the Celeste issue tracker - https://github.com/jeff-regier/Celeste.jl/issues/625 https://github.com/jeff-regier/Celeste.jl/issues/625 is one I found on a quick look but there's more), so I think it qualifies in that sense. Of course you may want a more sophisticated model than Celeste for a complete model of the sky.
petermcneeley 8y ago
If I wanted to create a code challenge that would be equivalent to the difficulty of this problem could I simply generate a series of images from 3d point lights and add noise? Is the changing location of earth in space factored into these calculations?
- KenoFischer 8y ago
  The data is publicly available (as is the code), so you could just use it. The data itself is organized by region of the sky being imaged, so that calculation has already been done for us (and at the distance the stars/galaxies are at the difference in distance due to the Earth orbit doesn't matter). In the actual application, both the noise and the psf are modeled based on the actual telescope, so just adding arbitrary noise is a bit of a different problem, but probably a good start. Do note also that Celeste is a generative model (including modeling the noise), so you could just use it to generate a bunch of images for you.
  - petermcneeley 8y ago
    I see you have the forward function to render out "color" given the properties of the stars that contribute to this region in space. All you have to do is find the highest probable property values that would generate that set of pixels. Sounds extremely parallizable which is probably why you managed to get the most out of the xeon phi.
    - KenoFischer 8y ago
      It's not quite that easy because of a) the noise (you're fitting distributions not values) and b) the fact that multiple light sources affect a given pixel, so you have to be a bit clever. But yes, you can get parallelism by spatial partition.
C1sc0cat 8y ago
Cool I miss working in technical computing sigh I recall when having a cluster of 17 supermini's was a really big thing :-) of course I monitored it using a dial up 110 baud portable print only terminal. Don't miss the poor pay though

philipkglass 8y ago

Finishing the whole job in under 15 minutes is impressive but also a bit suspicious, in a way. Back when I used a top 10 HPC facility at a national lab I saw a lot of jobs in the queue that could have run just fine on business class servers without the fancy, expensive interconnects needed for Grand Challenge problems. People didn't run their jobs on smaller machines because there was money in the budget to build a world class Top 10 computing resource but not money to buy less exotic hardware matched to typical jobs. Individual research groups didn't really get their typical compute needs measured or surveyed. These are obviously leading questions, but: - Are there significant research advantages to super-fast turnaround (less than 15 minutes, enabled by massive parallelism) in this domain? - Do you feel like a massively parallel system with Xeon Phi nodes is a good match for this problem? Or did the code get optimized to run at high scale on Cori Phase II because that's where you were given compute resources? Finally, a bit less provocative: Does this approach effectively scale down to e.g. a university that can afford a large storage array and beefy commercial servers (optionally equipped with Phi or other accelerators), but doesn't have HPC resources that are contenders for the Top 500? Or do you really need things that smaller systems can't deliver, like many terabytes of memory in distributed global arrays?

KenoFischer 8y ago
| Are there significant research advantages to super-fast turnaround (less than 15 minutes, enabled by massive parallelism) in this domain? No, the 15 minute turn around time is not important given the dataset we have at the moment, but showing that it was possible to do was considered important from a science perspective, because of the upcoming LSST telescope. LSST will generate an amount of data equivalent to the full dataset we had available every 3-4 days, so being able to scale this up far enough to accommodate that as well as future planned extensions to the algorithm, necessitated showing scale. The actual science runs by the project are usually done on a few hundred nodes over a couple of hours. | Do you feel like a massively parallel system with Xeon Phi nodes is a good match for this problem? Or did the code get optimized to run at high scale on Cori Phase II because that's where you were given compute resources? Cori Phase II worked well for this problem, though I wouldn't be surprised if GPUs wouldn't have been a better fit (though harder to program of course and at the time the Julia GPU infrastructure probably wasn't quite ready yet - even KNL was a struggle since LLVM was still in the process of completing support for it). The Celeste project is still ongoing (working on science goals more so than extra parallelism or performance improvements at the moment), but I wouldn't be surprised if there was an attempt to run on Summit at some point, especially now that Julia's GPU compiler is much more mature. One of the biggest problems we failed to anticipate actually was getting the data from disk to compute units quickly enough. Early in the project we crashed the interconnect on the machine, so for the challenge run we weren't allowed to do anything other than pull the data directly from disk (lest we bring down the machine again while other challenge runs were ongoing). I haven't really looked at the interconnect on Summit, so I can't say how well it would handle that. | Does this approach effectively scale down to e.g. a university that can afford a large storage array and beefy commercial servers Yes, it scales fairly well. In fact you could probably do it with spot instances on a public could fairly well. The biggest thing would once again be getting the data to the compute units quick enough. That's quite demanding on the network (and ideally you want to pre-stage the data in memory). Certainly it's feasible to do this on a large-ish university cluster on the SDSS data set in a few hours. Probably less feasible on LSST data once that comes online, but maybe by that point improvements in computation speed and storage speed will have made up for that and it'll become feasible again.
- pmalynin 8y ago
  >No, the 15 minute turn around time is not important given the dataset we have at the moment, but showing that it was possible to do was considered important from a science perspective, because of the upcoming LSST telescope. LSST will generate an amount of data equivalent to the full dataset we had available every 3-4 days, so being able to scale this up far enough to accommodate that as well as future planned extensions to the algorithm, necessitated showing scale. The actual science runs by the project are usually done on a few hundred nodes over a couple of hours. I've heard that the folks working on the EHT array need months to crunch numbers. Could something like this be used to speed up that process? Or is there some other reason that would prohibit that. P.S. I want pictures of black holes.
  - KenoFischer 8y ago
    I don't know. I'd imagine that the folks working on the EHT already are making use of plenty of HPC for image reconstruction. It's quite a different problem from the Celeste application of course, so this work isn't directly applicable, but if they ever wanted to rewrite their code in Julia, they should give us a shout ;).
deleted 8y ago
[deleted]

noobermin 8y ago

Very low key question that might be in your paper but I'm lazy: Did you use one of the compilation options for this or just the JIT compilation (if direct compilation is available, I'll be honest, I'm not that abreast of Julia developments). One of the key things for more computational simulation tools (as opposed to just analysis) is we find that the awesome compilers for C and Fortran codes are a huge boon, especially on systems that have ones that improve on stock intel (not to mention gcc) by a good factor. You lose that if you're just using the jit for the sake of using a "nicer" language, I guess. I had similar sentiments to the other cat who asked about the 15 minute run, but I guess it's fun to show it "can be done." I was planning to play a little in Julia and this shows it can be worth it.

KenoFischer 8y ago
The JIT compiler was active, but most of the computationally heavy parts were ahead of time compiled. The compiler had a number of tweaks to increase the quality of the generated code (though at the expense of a good amount of compile time). Generating the ahead of time image probably took a good 5-10 minutes (much more if we had tried to do it on the KNL nodes that we actually ran on), so it was infeasible and would have been a waste of time to try to do so in a JITted fashion. Some JIT compilation probably still happened (for code paths that weren't in the pre-compilation trace, e.g. logging code or exception code paths hit by a few processes).
- noobermin 8y ago
  For some reason, my brain forgot you did this on KNL. I guess even with cross compilation and then passing it to the platform compilers and it performing well is a sign it's good. Anyway, it's great you guys got this through especially with a tool like Julia. At least in my corner of academia, people utilizing "untested tools" unnecessarily raises the ire and skepticism of peers (for rather stupid reasons). I appreciate you guys "making the first move" which makes it easier for the rest of us looking to branch out and use these tools for our own codes.
  - KenoFischer 8y ago

mannermachine 8y ago

This is really neat, I used to do research in Bayesian networks, I had no idea they were being applied to astronomy, is their use widespread?

KenoFischer 8y ago
I worked on the computational side of this, so while I understand the science, I don't have a very good overview on the state of the art in astronomy. However, my understanding from the statisticians and astronomers on the project is that variational inference had not been attempted on this scale before (in Astronomy or otherwise).

SiempreViernes 8y ago

Does this make it possible to get closer to David Hogg's and Dustin Lang's "theory of everything"? (Is that the same D Lang as in the Celeste paper?) That is the idea that a catalogue is released as a complete model of the sky, rather than as a table of intensities and coordinates. (https://arxiv.org/pdf/0810.3851.pdf https://arxiv.org/pdf/0810.3851.pdf)

KenoFischer 8y ago
I don't feel qualified to speculate on the implications for astronomy because that's not my field. However, Celeste is indeed a generative model and you can draw an image sample from the Celeste model and compare it to the original (there's a few examples in the Celeste issue tracker - https://github.com/jeff-regier/Celeste.jl/issues/625 https://github.com/jeff-regier/Celeste.jl/issues/625 is one I found on a quick look but there's more), so I think it qualifies in that sense. Of course you may want a more sophisticated model than Celeste for a complete model of the sky.

petermcneeley 8y ago

If I wanted to create a code challenge that would be equivalent to the difficulty of this problem could I simply generate a series of images from 3d point lights and add noise? Is the changing location of earth in space factored into these calculations?

KenoFischer 8y ago
The data is publicly available (as is the code), so you could just use it. The data itself is organized by region of the sky being imaged, so that calculation has already been done for us (and at the distance the stars/galaxies are at the difference in distance due to the Earth orbit doesn't matter). In the actual application, both the noise and the psf are modeled based on the actual telescope, so just adding arbitrary noise is a bit of a different problem, but probably a good start. Do note also that Celeste is a generative model (including modeling the noise), so you could just use it to generate a bunch of images for you.
- petermcneeley 8y ago
  I see you have the forward function to render out "color" given the properties of the stars that contribute to this region in space. All you have to do is find the highest probable property values that would generate that set of pixels. Sounds extremely parallizable which is probably why you managed to get the most out of the xeon phi.
  - KenoFischer 8y ago
    It's not quite that easy because of a) the noise (you're fitting distributions not values) and b) the fact that multiple light sources affect a given pixel, so you have to be a bit clever. But yes, you can get parallelism by spatial partition.

C1sc0cat 8y ago

Cool I miss working in technical computing sigh I recall when having a cluster of 17 supermini's was a really big thing :-) of course I monitored it using a dial up 110 baud portable print only terminal. Don't miss the poor pay though