8 ms·

Parallel Supercomputing for Astronomy with Julia

161 points by SiempreViernes 8y ago

KenoFischer 8y ago
Oh hey, I worked on this. Paper with all the details here: https://arxiv.org/pdf/1801.10277.pdf https://arxiv.org/pdf/1801.10277.pdf. Happy to answer questions.
- philipkglass 8y ago
  Finishing the whole job in under 15 minutes is impressive but also a bit suspicious, in a way. Back when I used a top 10 HPC facility at a national lab I saw a lot of jobs in the queue that could have run just fine on business class servers without the fancy, expensive interconnects needed for Grand Challenge problems. People didn't run their jobs on smaller machines because there was money in the budget to build a world class Top 10 computing resource but not money to buy less exotic hardware matched to typical jobs. Individual research groups didn't really get their typical compute needs measured or surveyed. These are obviously leading questions, but: - Are there significant research advantages to super-fast turnaround (less than 15 minutes, enabled by massive parallelism) in this domain? - Do you feel like a massively parallel system with Xeon Phi nodes is a good match for this problem? Or did the code get optimized to run at high scale on Cori Phase II because that's where you were given compute resources? Finally, a bit less provocative: Does this approach effectively scale down to e.g. a university that can afford a large storage array and beefy commercial servers (optionally equipped with Phi or other accelerators), but doesn't have HPC resources that are contenders for the Top 500? Or do you really need things that smaller systems can't deliver, like many terabytes of memory in distributed global arrays?
  - KenoFischer 8y ago
    | Are there significant research advantages to super-fast turnaround (less than 15 minutes, enabled by massive parallelism) in this domain? No, the 15 minute turn around time is not important given the dataset we have at the moment, but showing that it was possible to do was considered important from a science perspective, because of the upcoming LSST telescope. LSST will generate an amount of data equivalent to the full dataset we had available every 3-4 days, so being able to scale this up far enough to accommodate that as well as future planned extensions to the algorithm, necessitated showing scale. The actual science runs by the project are usually done on a few hundred nodes over a couple of hours. | Do you feel like a massively parallel system with Xeon Phi nodes is a good match for this problem? Or did the code get optimized to run at high scale on Cori Phase II because that's where you were given compute resources? Cori Phase II worked well for this problem, though I wouldn't be surprised if GPUs wouldn't have been a better fit (though harder to program of course and at the time the Julia GPU infrastructure probably wasn't quite ready yet - even KNL was a struggle since LLVM was still in the process of completing support for it). The Celeste project is still ongoing (working on science goals more so than extra parallelism or performance improvements at the moment), but I wouldn't be surprised if there was an attempt to run on Summit at some point, especially now that Julia's GPU compiler is much more mature. One of the biggest problems we failed to anticipate actually was getting the data from disk to compute units quickly enough. Early in the project we crashed the interconnect on the machine, so for the challenge run we weren't allowed to do anything other than pull the data directly from disk (lest we bring down the machine again while other challenge runs were ongoing). I haven't really looked at the interconnect on Summit, so I can't say how well it would handle that. | Does this approach effectively scale down to e.g. a university that can afford a large storage array and beefy commercial servers Yes, it scales fairly well. In fact you could probably do it with spot instances on a public could fairly well. The biggest thing would once again be getting the data to the compute units quick enough. That's quite demanding on the network (and ideally you want to pre-stage the data in memory). Certainly it's feasible to do this on a large-ish university cluster on the SDSS data set in a few hours. Probably less feasible on LSST data once that comes online, but maybe by that point improvements in computation speed and storage speed will have made up for that and it'll become feasible again.
    - pmalynin 8y ago
      >No, the 15 minute turn around time is not important given the dataset we have at the moment, but showing that it was possible to do was considered important from a science perspective, because of the upcoming LSST telescope. LSST will generate an amount of data equivalent to the full dataset we had available every 3-4 days, so being able to scale this up far enough to accommodate that as well as future planned extensions to the algorithm, necessitated showing scale. The actual science runs by the project are usually done on a few hundred nodes over a couple of hours. I've heard that the folks working on the EHT array need months to crunch numbers. Could something like this be used to speed up that process? Or is there some other reason that would prohibit that. P.S. I want pictures of black holes.
      - KenoFischer 8y ago
        I don't know. I'd imagine that the folks working on the EHT already are making use of plenty of HPC for image reconstruction. It's quite a different problem from the Celeste application of course, so this work isn't directly applicable, but if they ever wanted to rewrite their code in Julia, they should give us a shout ;).
  - deleted 8y ago
    [deleted]
- noobermin 8y ago
  Very low key question that might be in your paper but I'm lazy: Did you use one of the compilation options for this or just the JIT compilation (if direct compilation is available, I'll be honest, I'm not that abreast of Julia developments). One of the key things for more computational simulation tools (as opposed to just analysis) is we find that the awesome compilers for C and Fortran codes are a huge boon, especially on systems that have ones that improve on stock intel (not to mention gcc) by a good factor. You lose that if you're just using the jit for the sake of using a "nicer" language, I guess. I had similar sentiments to the other cat who asked about the 15 minute run, but I guess it's fun to show it "can be done." I was planning to play a little in Julia and this shows it can be worth it.
  - KenoFischer 8y ago
    The JIT compiler was active, but most of the computationally heavy parts were ahead of time compiled. The compiler had a number of tweaks to increase the quality of the generated code (though at the expense of a good amount of compile time). Generating the ahead of time image probably took a good 5-10 minutes (much more if we had tried to do it on the KNL nodes that we actually ran on), so it was infeasible and would have been a waste of time to try to do so in a JITted fashion. Some JIT compilation probably still happened (for code paths that weren't in the pre-compilation trace, e.g. logging code or exception code paths hit by a few processes).
    - noobermin 8y ago
      For some reason, my brain forgot you did this on KNL. I guess even with cross compilation and then passing it to the platform compilers and it performing well is a sign it's good. Anyway, it's great you guys got this through especially with a tool like Julia. At least in my corner of academia, people utilizing "untested tools" unnecessarily raises the ire and skepticism of peers (for rather stupid reasons). I appreciate you guys "making the first move" which makes it easier for the rest of us looking to branch out and use these tools for our own codes.
      - KenoFischer 8y ago
- mannermachine 8y ago
  This is really neat, I used to do research in Bayesian networks, I had no idea they were being applied to astronomy, is their use widespread?
  - KenoFischer 8y ago
    I worked on the computational side of this, so while I understand the science, I don't have a very good overview on the state of the art in astronomy. However, my understanding from the statisticians and astronomers on the project is that variational inference had not been attempted on this scale before (in Astronomy or otherwise).
- SiempreViernes 8y ago
  Does this make it possible to get closer to David Hogg's and Dustin Lang's "theory of everything"? (Is that the same D Lang as in the Celeste paper?) That is the idea that a catalogue is released as a complete model of the sky, rather than as a table of intensities and coordinates. (https://arxiv.org/pdf/0810.3851.pdf https://arxiv.org/pdf/0810.3851.pdf)
  - KenoFischer 8y ago
    I don't feel qualified to speculate on the implications for astronomy because that's not my field. However, Celeste is indeed a generative model and you can draw an image sample from the Celeste model and compare it to the original (there's a few examples in the Celeste issue tracker - https://github.com/jeff-regier/Celeste.jl/issues/625 https://github.com/jeff-regier/Celeste.jl/issues/625 is one I found on a quick look but there's more), so I think it qualifies in that sense. Of course you may want a more sophisticated model than Celeste for a complete model of the sky.
- petermcneeley 8y ago
  If I wanted to create a code challenge that would be equivalent to the difficulty of this problem could I simply generate a series of images from 3d point lights and add noise? Is the changing location of earth in space factored into these calculations?
  - KenoFischer 8y ago
    The data is publicly available (as is the code), so you could just use it. The data itself is organized by region of the sky being imaged, so that calculation has already been done for us (and at the distance the stars/galaxies are at the difference in distance due to the Earth orbit doesn't matter). In the actual application, both the noise and the psf are modeled based on the actual telescope, so just adding arbitrary noise is a bit of a different problem, but probably a good start. Do note also that Celeste is a generative model (including modeling the noise), so you could just use it to generate a bunch of images for you.
    - petermcneeley 8y ago
      I see you have the forward function to render out "color" given the properties of the stars that contribute to this region in space. All you have to do is find the highest probable property values that would generate that set of pixels. Sounds extremely parallizable which is probably why you managed to get the most out of the xeon phi.
      - KenoFischer 8y ago
        It's not quite that easy because of a) the noise (you're fitting distributions not values) and b) the fact that multiple light sources affect a given pixel, so you have to be a bit clever. But yes, you can get parallelism by spatial partition.
- C1sc0cat 8y ago
  Cool I miss working in technical computing sigh I recall when having a cluster of 17 supermini's was a really big thing :-) of course I monitored it using a dial up 110 baud portable print only terminal. Don't miss the poor pay though
qualsiasi 8y ago
I would be curious to know what would be the elapsed time on the same machine (and problem) using more "enterprisey" languages like Java, C#, ...
- noobermin 8y ago
  Tbh, I just don't see any usage of Java or C# on HPC systems. Python is used a bit but in embarrassingly parallel mode. Part of it might just be cultural/biases but it just isn't used, especially since there isn't much of a need to create and manage a complex software architecture that enterprise systems leverage.
- KenoFischer 8y ago
  I'm not sure what the state of AVX-512 codegen was in the CLR/JVM back in early 2017, but given that LLVM's was still in buggy early stages, I would guess that it was a WIP at best so performance was probably not ideal (we had to work quite hard to get decent quality code generation). On KNL, you're either running AVX512 all the time or you're pretty much dead in terms of performance.
- jabl 8y ago
  I work in HPC, and, well, Java and C# are more or less irrelevant in that space. Well, there is a little bit of Java, some people are using things like Hadoop or Spark, but otherwise, no. "Hard-core" computing is almost all C/C++/Fortran (and of course CUDA for GPU's etc.). Python and R are fairly popular, but in those cases (hopefully) most of the heavy lifting is done by library code (again, C/C++/Fortran, or increasingly CUDA via ML libraries such as tensorflow) rather than the interpreter. Julia is very promising in this space, as it offers a solution to the "two-language" problem. I'm hopeful for Julia to make more of an impact, but it's of course a slow process. (There is a (tiny) bit of ASM, but that's more or less exclusively done for widely used performance-critical libraries like BLAS, or FFT, not for application code.)
  - antpls 8y ago
    Is Julia a JIT compiler? How is Julia different than Pypy or even the JVM? There is also the new GraalVM in the landscape. I don't see the plus value in Julia compared to Python or Java. It will still be slower than C/C++, probably less portable, and all the legacy libraries have to be rewritten in Julia. If Julia is only a new syntax, to me Python is already very simple. If Julia is a JIT compiler, why not participate to already available compilers? Julia is still there, so I guess it adds value, but I don't know where to place that effort in the grand scheme of things.
jgamman 8y ago
wow - 28 people on the about page and the only woman is the diversity director.
- KenoFischer 8y ago
  Yep, that's something we need to fix. So far we've done most of our hiring from the open source community, which is unfortunately overwhelmingly male. We've done some work to try to increase diversity in the community and Jane graciously agreed to take a break from her PhD studies at Caltech to help with outreach as part of that (also thanks to the Sloan Foundation for funding the corresponding effort). Clearly a lot more work is needed on that front, but I'm hoping the work that Jane and others have been doing this past year have helped on the community side. Of course there's work to do on the company side independently, but I imagine the community will continue to be an important hiring channel, so I consider improving diversity there as a necessary ingredient.
  - chimpburger 8y ago
    Nothing needs to be fixed. Make the opportunity available to women but do not enforce a gender quota. Candidates should only be selected on merit. Equal opportunity is good. Forcing equality of outcome is BAD.
    - lern_too_spel 8y ago
      Nobody said anything about quotas except you. The GP said they were working on sourcing their candidates differently.
      - chimpburger 8y ago
        @lern_too_spel: I know they didn't, but they did not say they were not considering a gender quota.
        SiempreViernes 8y ago
adyavanapalli 8y ago
"Peta floating point operations per second per second" XD
kwertzzz 8y ago
This is really a great achievement. I am wondering how much communication was necessary between the cores during the computation or is the problem a so called "embarrassingly parallel" workload where the work could be split into independent tasks?
sgt101 8y ago
I found it funny to compare a paper by my colleague Andrew and this talk. Ok we used gibbs sampling and a teeny weeny sad old hadoop cluster and we were looking a phone lines and not stars, but Bayesian inference to make a single catalogue... Ours does what's this made of and where is this thing, rather than "is it a star or a galaxy" Also ours doesn't run in 15 minutes! https://rd.springer.com/content/pdf/10.1007%2F978-3-319-71078-5_17.pdf https://rd.springer.com/content/pdf/10.1007%2F978-3-319-7107...

Oh hey, I worked on this. Paper with all the details here: https://arxiv.org/pdf/1801.10277.pdf https://arxiv.org/pdf/1801.10277.pdf. Happy to answer questions.

philipkglass 8y ago
Finishing the whole job in under 15 minutes is impressive but also a bit suspicious, in a way. Back when I used a top 10 HPC facility at a national lab I saw a lot of jobs in the queue that could have run just fine on business class servers without the fancy, expensive interconnects needed for Grand Challenge problems. People didn't run their jobs on smaller machines because there was money in the budget to build a world class Top 10 computing resource but not money to buy less exotic hardware matched to typical jobs. Individual research groups didn't really get their typical compute needs measured or surveyed. These are obviously leading questions, but: - Are there significant research advantages to super-fast turnaround (less than 15 minutes, enabled by massive parallelism) in this domain? - Do you feel like a massively parallel system with Xeon Phi nodes is a good match for this problem? Or did the code get optimized to run at high scale on Cori Phase II because that's where you were given compute resources? Finally, a bit less provocative: Does this approach effectively scale down to e.g. a university that can afford a large storage array and beefy commercial servers (optionally equipped with Phi or other accelerators), but doesn't have HPC resources that are contenders for the Top 500? Or do you really need things that smaller systems can't deliver, like many terabytes of memory in distributed global arrays?
- KenoFischer 8y ago
  | Are there significant research advantages to super-fast turnaround (less than 15 minutes, enabled by massive parallelism) in this domain? No, the 15 minute turn around time is not important given the dataset we have at the moment, but showing that it was possible to do was considered important from a science perspective, because of the upcoming LSST telescope. LSST will generate an amount of data equivalent to the full dataset we had available every 3-4 days, so being able to scale this up far enough to accommodate that as well as future planned extensions to the algorithm, necessitated showing scale. The actual science runs by the project are usually done on a few hundred nodes over a couple of hours. | Do you feel like a massively parallel system with Xeon Phi nodes is a good match for this problem? Or did the code get optimized to run at high scale on Cori Phase II because that's where you were given compute resources? Cori Phase II worked well for this problem, though I wouldn't be surprised if GPUs wouldn't have been a better fit (though harder to program of course and at the time the Julia GPU infrastructure probably wasn't quite ready yet - even KNL was a struggle since LLVM was still in the process of completing support for it). The Celeste project is still ongoing (working on science goals more so than extra parallelism or performance improvements at the moment), but I wouldn't be surprised if there was an attempt to run on Summit at some point, especially now that Julia's GPU compiler is much more mature. One of the biggest problems we failed to anticipate actually was getting the data from disk to compute units quickly enough. Early in the project we crashed the interconnect on the machine, so for the challenge run we weren't allowed to do anything other than pull the data directly from disk (lest we bring down the machine again while other challenge runs were ongoing). I haven't really looked at the interconnect on Summit, so I can't say how well it would handle that. | Does this approach effectively scale down to e.g. a university that can afford a large storage array and beefy commercial servers Yes, it scales fairly well. In fact you could probably do it with spot instances on a public could fairly well. The biggest thing would once again be getting the data to the compute units quick enough. That's quite demanding on the network (and ideally you want to pre-stage the data in memory). Certainly it's feasible to do this on a large-ish university cluster on the SDSS data set in a few hours. Probably less feasible on LSST data once that comes online, but maybe by that point improvements in computation speed and storage speed will have made up for that and it'll become feasible again.
  - pmalynin 8y ago
    >No, the 15 minute turn around time is not important given the dataset we have at the moment, but showing that it was possible to do was considered important from a science perspective, because of the upcoming LSST telescope. LSST will generate an amount of data equivalent to the full dataset we had available every 3-4 days, so being able to scale this up far enough to accommodate that as well as future planned extensions to the algorithm, necessitated showing scale. The actual science runs by the project are usually done on a few hundred nodes over a couple of hours. I've heard that the folks working on the EHT array need months to crunch numbers. Could something like this be used to speed up that process? Or is there some other reason that would prohibit that. P.S. I want pictures of black holes.
    - KenoFischer 8y ago
      I don't know. I'd imagine that the folks working on the EHT already are making use of plenty of HPC for image reconstruction. It's quite a different problem from the Celeste application of course, so this work isn't directly applicable, but if they ever wanted to rewrite their code in Julia, they should give us a shout ;).
- deleted 8y ago
  [deleted]
noobermin 8y ago
Very low key question that might be in your paper but I'm lazy: Did you use one of the compilation options for this or just the JIT compilation (if direct compilation is available, I'll be honest, I'm not that abreast of Julia developments). One of the key things for more computational simulation tools (as opposed to just analysis) is we find that the awesome compilers for C and Fortran codes are a huge boon, especially on systems that have ones that improve on stock intel (not to mention gcc) by a good factor. You lose that if you're just using the jit for the sake of using a "nicer" language, I guess. I had similar sentiments to the other cat who asked about the 15 minute run, but I guess it's fun to show it "can be done." I was planning to play a little in Julia and this shows it can be worth it.
- KenoFischer 8y ago
  The JIT compiler was active, but most of the computationally heavy parts were ahead of time compiled. The compiler had a number of tweaks to increase the quality of the generated code (though at the expense of a good amount of compile time). Generating the ahead of time image probably took a good 5-10 minutes (much more if we had tried to do it on the KNL nodes that we actually ran on), so it was infeasible and would have been a waste of time to try to do so in a JITted fashion. Some JIT compilation probably still happened (for code paths that weren't in the pre-compilation trace, e.g. logging code or exception code paths hit by a few processes).
  - noobermin 8y ago
    For some reason, my brain forgot you did this on KNL. I guess even with cross compilation and then passing it to the platform compilers and it performing well is a sign it's good. Anyway, it's great you guys got this through especially with a tool like Julia. At least in my corner of academia, people utilizing "untested tools" unnecessarily raises the ire and skepticism of peers (for rather stupid reasons). I appreciate you guys "making the first move" which makes it easier for the rest of us looking to branch out and use these tools for our own codes.
    - KenoFischer 8y ago
mannermachine 8y ago
This is really neat, I used to do research in Bayesian networks, I had no idea they were being applied to astronomy, is their use widespread?
- KenoFischer 8y ago
  I worked on the computational side of this, so while I understand the science, I don't have a very good overview on the state of the art in astronomy. However, my understanding from the statisticians and astronomers on the project is that variational inference had not been attempted on this scale before (in Astronomy or otherwise).
SiempreViernes 8y ago
Does this make it possible to get closer to David Hogg's and Dustin Lang's "theory of everything"? (Is that the same D Lang as in the Celeste paper?) That is the idea that a catalogue is released as a complete model of the sky, rather than as a table of intensities and coordinates. (https://arxiv.org/pdf/0810.3851.pdf https://arxiv.org/pdf/0810.3851.pdf)
- KenoFischer 8y ago
  I don't feel qualified to speculate on the implications for astronomy because that's not my field. However, Celeste is indeed a generative model and you can draw an image sample from the Celeste model and compare it to the original (there's a few examples in the Celeste issue tracker - https://github.com/jeff-regier/Celeste.jl/issues/625 https://github.com/jeff-regier/Celeste.jl/issues/625 is one I found on a quick look but there's more), so I think it qualifies in that sense. Of course you may want a more sophisticated model than Celeste for a complete model of the sky.
petermcneeley 8y ago
If I wanted to create a code challenge that would be equivalent to the difficulty of this problem could I simply generate a series of images from 3d point lights and add noise? Is the changing location of earth in space factored into these calculations?
- KenoFischer 8y ago
  The data is publicly available (as is the code), so you could just use it. The data itself is organized by region of the sky being imaged, so that calculation has already been done for us (and at the distance the stars/galaxies are at the difference in distance due to the Earth orbit doesn't matter). In the actual application, both the noise and the psf are modeled based on the actual telescope, so just adding arbitrary noise is a bit of a different problem, but probably a good start. Do note also that Celeste is a generative model (including modeling the noise), so you could just use it to generate a bunch of images for you.
  - petermcneeley 8y ago
    I see you have the forward function to render out "color" given the properties of the stars that contribute to this region in space. All you have to do is find the highest probable property values that would generate that set of pixels. Sounds extremely parallizable which is probably why you managed to get the most out of the xeon phi.
    - KenoFischer 8y ago
      It's not quite that easy because of a) the noise (you're fitting distributions not values) and b) the fact that multiple light sources affect a given pixel, so you have to be a bit clever. But yes, you can get parallelism by spatial partition.
C1sc0cat 8y ago
Cool I miss working in technical computing sigh I recall when having a cluster of 17 supermini's was a really big thing :-) of course I monitored it using a dial up 110 baud portable print only terminal. Don't miss the poor pay though

qualsiasi 8y ago

I would be curious to know what would be the elapsed time on the same machine (and problem) using more "enterprisey" languages like Java, C#, ...

noobermin 8y ago
Tbh, I just don't see any usage of Java or C# on HPC systems. Python is used a bit but in embarrassingly parallel mode. Part of it might just be cultural/biases but it just isn't used, especially since there isn't much of a need to create and manage a complex software architecture that enterprise systems leverage.
KenoFischer 8y ago
I'm not sure what the state of AVX-512 codegen was in the CLR/JVM back in early 2017, but given that LLVM's was still in buggy early stages, I would guess that it was a WIP at best so performance was probably not ideal (we had to work quite hard to get decent quality code generation). On KNL, you're either running AVX512 all the time or you're pretty much dead in terms of performance.
jabl 8y ago
I work in HPC, and, well, Java and C# are more or less irrelevant in that space. Well, there is a little bit of Java, some people are using things like Hadoop or Spark, but otherwise, no. "Hard-core" computing is almost all C/C++/Fortran (and of course CUDA for GPU's etc.). Python and R are fairly popular, but in those cases (hopefully) most of the heavy lifting is done by library code (again, C/C++/Fortran, or increasingly CUDA via ML libraries such as tensorflow) rather than the interpreter. Julia is very promising in this space, as it offers a solution to the "two-language" problem. I'm hopeful for Julia to make more of an impact, but it's of course a slow process. (There is a (tiny) bit of ASM, but that's more or less exclusively done for widely used performance-critical libraries like BLAS, or FFT, not for application code.)
- antpls 8y ago
  Is Julia a JIT compiler? How is Julia different than Pypy or even the JVM? There is also the new GraalVM in the landscape. I don't see the plus value in Julia compared to Python or Java. It will still be slower than C/C++, probably less portable, and all the legacy libraries have to be rewritten in Julia. If Julia is only a new syntax, to me Python is already very simple. If Julia is a JIT compiler, why not participate to already available compilers? Julia is still there, so I guess it adds value, but I don't know where to place that effort in the grand scheme of things.

jgamman 8y ago

wow - 28 people on the about page and the only woman is the diversity director.

KenoFischer 8y ago
Yep, that's something we need to fix. So far we've done most of our hiring from the open source community, which is unfortunately overwhelmingly male. We've done some work to try to increase diversity in the community and Jane graciously agreed to take a break from her PhD studies at Caltech to help with outreach as part of that (also thanks to the Sloan Foundation for funding the corresponding effort). Clearly a lot more work is needed on that front, but I'm hoping the work that Jane and others have been doing this past year have helped on the community side. Of course there's work to do on the company side independently, but I imagine the community will continue to be an important hiring channel, so I consider improving diversity there as a necessary ingredient.
- chimpburger 8y ago
  Nothing needs to be fixed. Make the opportunity available to women but do not enforce a gender quota. Candidates should only be selected on merit. Equal opportunity is good. Forcing equality of outcome is BAD.
  - lern_too_spel 8y ago
    Nobody said anything about quotas except you. The GP said they were working on sourcing their candidates differently.
    - chimpburger 8y ago
      @lern_too_spel: I know they didn't, but they did not say they were not considering a gender quota.
      - SiempreViernes 8y ago

adyavanapalli 8y ago

"Peta floating point operations per second per second" XD

kwertzzz 8y ago

This is really a great achievement. I am wondering how much communication was necessary between the cores during the computation or is the problem a so called "embarrassingly parallel" workload where the work could be split into independent tasks?

sgt101 8y ago

I found it funny to compare a paper by my colleague Andrew and this talk. Ok we used gibbs sampling and a teeny weeny sad old hadoop cluster and we were looking a phone lines and not stars, but Bayesian inference to make a single catalogue... Ours does what's this made of and where is this thing, rather than "is it a star or a galaxy" Also ours doesn't run in 15 minutes! https://rd.springer.com/content/pdf/10.1007%2F978-3-319-71078-5_17.pdf https://rd.springer.com/content/pdf/10.1007%2F978-3-319-7107...