8 ms·

LIGO Gravitational Wave Data in iPython Jupyter Notebooks

209 points by Mizza 11y ago

jeffjose 11y ago
As a python fanboy this makes me really happy. Great job, folks at PSF and Jupyter
ck2 11y ago
For a moment I thought "notebook" meant like laptop. And I was like, wow that is amazing, they crowdsourced data from all over the world. Then I remembered the sensitivity of the instruments used and how dumb I was. But it really would be cool if oneday people had smartphones so advanced they could contribute to worldwide collection of data that needed a huge area to sample. I think they are talking about doing that for earthquake alerts.
- madvoid 11y ago
  WeatherSignal is doing something along those lines for weather data: http://weathersignal.com/about/ http://weathersignal.com/about/ I'm excited to see where this kind of tech goes.
- saboot 11y ago
  http://crayfis.io/ http://crayfis.io/ Use your smartphone camera to create a global network for detecting cosmic ray showers!
Xcelerate 11y ago
As someone who keeps getting comments about my "weird" toolkit for doing scientific research (Jupyter/Julia), this made my day.
- gaur 11y ago
  What would people rather have you use? Excel?
  - joshvm 11y ago
    You would be surprised at the number of Excel plots in the literature. Most astrophysicists use IDL or IRAF, I've never seen anyone use Matlab. The benefits are tons of functions specifically tailored for astro data analysis. Python is gathering momentum though, there are libraries like sunpy and astropy. Plenty of IDL fanatics are floored when you can show them just how easy it is to process data with the Numpy stack. I'm _not_ an astrophysicist, but I work with a lot of them. I use Jupyter notebook daily. It's the perfect balance between REPL python and standalone scripts. My typical workflow is to hack something together in a notebook, which lets you iterate very quickly, then once I'm happy I freeze the code into a module.
    - gaur 11y ago
      Haha, I forgot about IDL. My very first coding project as an undergrad was in IDL, until one day a new grad student showed up to the lab and said, "WTF? Do this in python." When looking at plots in papers, there are always little giveaways for what program it was made in. Plot has horizontal gridlines, but no vertical gridlines: Excel. Plot is typeset with Arial, size 4: Matlab. Plot looks like it was sent through a fax a few times: IDL.
      - cdelsolar 11y ago
        Did the LIGO paper use matplotlib? I thought it did but that would be too cool.
    - whorleater 11y ago
      I'm an astrophysics/CS student, and IDL has been on life support at my university for the last 3 years. Everything is either done through Fortran 90 or C/C++ for simulations code, and the data processing is done through Python with some combination of GNUplot/matplotlib/yt, or maybe R.
      - joshvm 11y ago
        Oh good God how could I forget about Fortran and GNUplot. F90 though? Luxury. All your code should be in F77 :P . I've worked on a few image reduction pipelines which were all in C++.
daveguy 11y ago
Note the sample code is in Python 2.7 not 3.x. In general you should avoid 3.x for scientific computing as it is slower and less supported in academia.
- erispoe 11y ago
  That's interesting, do you have a good source on that? I switched to 3.x for development but I do academic projects too.
  - joshvm 11y ago
    I don't know about speed, most of the modules where speed matters, like Numpy, rely on calling C routines somewhere down the line. The version of Python shouldn't make a significant difference. Numpy, Scipy and Pandas work with Python 3, Jupyter certainly does. That covers probably 80% of scientific grunt work. For specialist applications, OpenCV 3.0 works, as does scikit-learn/image. I can't speak for other development like web though. I think the main problem is that if you're using someone else's code in academia, it's likely to be written in 2.7 and you would have to go through and update everything that's not back-compatible.
- gaur 11y ago
  This is a self-fulfilling prophecy. Someone tries to write an analysis routine in 3.x, but then is quickly told "nobody in science uses 3.x, because of <vague reasons>". Result: everyone sticks to 2.7, so nobody ever even tries to build against 3.x when writing packages.
- babo 11y ago
  I switched to python3 after measured a large ETL process converted. As scientific packages are rely on FFI and the actual code is running outside of python I'm puzzled with your comment about speed.
- fonnesbeck 11y ago
  Neither of these assertions are true. I use Py3 to teach graduate-level scientific computing and machine learning with great success, and the scientific Python community is making a very strong push for adoption. All major packages in the scientific stack now support Python 3.
wiz21 11y ago
Donald Knuth is probably swimming in joy right now. This so literate programming (or should I say literate research)...
Mizza 11y ago
If you liked this, you may like this surprisingly fascinating talk on the color scheme in Matplotlib used in this paper: https://www.youtube.com/watch?v=xAoljeRJ3lU&feature=youtu.be https://www.youtube.com/watch?v=xAoljeRJ3lU&feature=youtu.be
- cscheid 11y ago
  This talk is great! And if you watched it all the way to the 6:30 mark, where he says "you can't have negative amounts of light", then you'll really enjoy chimerical colors: https://en.wikipedia.org/wiki/Impossible_color https://en.wikipedia.org/wiki/Impossible_color The trick is you exploit retinal fatigue in order to perceive colors both "above 100%" and "below 0%". Wild stuff.
po84 11y ago
https://github.com/minrk/ligo-binder https://github.com/minrk/ligo-binder Re-run the analysis yourself on Jupyter using Binder. Click the launch binder button.
- qhoc 11y ago
  Just curious on binder specific here. I could never got it to work out-of-the-box. In this example, it gave me "ImportError: No module named 'seaborn'" error. Now I could do something like !sudo pip install seaborn but still... Did you guys have same experience?
melted 11y ago
Makes for a nice demo of Python Notebook capabilities, especially when it comes to graphing.
anotheryou 11y ago
can anyone tell me how the figured out where the black holes are? with just 2 "ears" I'd expect to be only able to determen a circle, but here is the picture they released: http://content.screencast.com/users/cougarten/folders/Jing/media/64d98a70-7b90-45b6-ac97-2518aa657171/2016-02-11_1650.png http://content.screencast.com/users/cougarten/folders/Jing/m... Do i see a warped circle, or the bottom part of one? In the latter case I wonder how they found out. Are the L-shape sensors capable of seeing some direction depending on which way the phases shift first/last? If you'd build one of the sensors in reverse you'd see a reversed signal, no? Given that they will have optimized the orientations of both stations this is probably how it worked and I'm seeing just part of a circle, right? That one lonely blob might be an unlikely mirrored version.

209 points by Mizza 11y ago

As a python fanboy this makes me really happy. Great job, folks at PSF and Jupyter

ck2 11y ago

For a moment I thought "notebook" meant like laptop. And I was like, wow that is amazing, they crowdsourced data from all over the world. Then I remembered the sensitivity of the instruments used and how dumb I was. But it really would be cool if oneday people had smartphones so advanced they could contribute to worldwide collection of data that needed a huge area to sample. I think they are talking about doing that for earthquake alerts.

madvoid 11y ago
WeatherSignal is doing something along those lines for weather data: http://weathersignal.com/about/ http://weathersignal.com/about/ I'm excited to see where this kind of tech goes.
saboot 11y ago
http://crayfis.io/ http://crayfis.io/ Use your smartphone camera to create a global network for detecting cosmic ray showers!

Xcelerate 11y ago

As someone who keeps getting comments about my "weird" toolkit for doing scientific research (Jupyter/Julia), this made my day.

gaur 11y ago
What would people rather have you use? Excel?
- joshvm 11y ago
  You would be surprised at the number of Excel plots in the literature. Most astrophysicists use IDL or IRAF, I've never seen anyone use Matlab. The benefits are tons of functions specifically tailored for astro data analysis. Python is gathering momentum though, there are libraries like sunpy and astropy. Plenty of IDL fanatics are floored when you can show them just how easy it is to process data with the Numpy stack. I'm _not_ an astrophysicist, but I work with a lot of them. I use Jupyter notebook daily. It's the perfect balance between REPL python and standalone scripts. My typical workflow is to hack something together in a notebook, which lets you iterate very quickly, then once I'm happy I freeze the code into a module.
  - gaur 11y ago
    Haha, I forgot about IDL. My very first coding project as an undergrad was in IDL, until one day a new grad student showed up to the lab and said, "WTF? Do this in python." When looking at plots in papers, there are always little giveaways for what program it was made in. Plot has horizontal gridlines, but no vertical gridlines: Excel. Plot is typeset with Arial, size 4: Matlab. Plot looks like it was sent through a fax a few times: IDL.
    - cdelsolar 11y ago
      Did the LIGO paper use matplotlib? I thought it did but that would be too cool.
  - whorleater 11y ago
    I'm an astrophysics/CS student, and IDL has been on life support at my university for the last 3 years. Everything is either done through Fortran 90 or C/C++ for simulations code, and the data processing is done through Python with some combination of GNUplot/matplotlib/yt, or maybe R.
    - joshvm 11y ago
      Oh good God how could I forget about Fortran and GNUplot. F90 though? Luxury. All your code should be in F77 :P . I've worked on a few image reduction pipelines which were all in C++.

daveguy 11y ago

Note the sample code is in Python 2.7 not 3.x. In general you should avoid 3.x for scientific computing as it is slower and less supported in academia.

erispoe 11y ago
That's interesting, do you have a good source on that? I switched to 3.x for development but I do academic projects too.
- joshvm 11y ago
  I don't know about speed, most of the modules where speed matters, like Numpy, rely on calling C routines somewhere down the line. The version of Python shouldn't make a significant difference. Numpy, Scipy and Pandas work with Python 3, Jupyter certainly does. That covers probably 80% of scientific grunt work. For specialist applications, OpenCV 3.0 works, as does scikit-learn/image. I can't speak for other development like web though. I think the main problem is that if you're using someone else's code in academia, it's likely to be written in 2.7 and you would have to go through and update everything that's not back-compatible.
gaur 11y ago
This is a self-fulfilling prophecy. Someone tries to write an analysis routine in 3.x, but then is quickly told "nobody in science uses 3.x, because of <vague reasons>". Result: everyone sticks to 2.7, so nobody ever even tries to build against 3.x when writing packages.
babo 11y ago
I switched to python3 after measured a large ETL process converted. As scientific packages are rely on FFI and the actual code is running outside of python I'm puzzled with your comment about speed.
fonnesbeck 11y ago
Neither of these assertions are true. I use Py3 to teach graduate-level scientific computing and machine learning with great success, and the scientific Python community is making a very strong push for adoption. All major packages in the scientific stack now support Python 3.

wiz21 11y ago

Donald Knuth is probably swimming in joy right now. This so literate programming (or should I say literate research)...

Mizza 11y ago

If you liked this, you may like this surprisingly fascinating talk on the color scheme in Matplotlib used in this paper: https://www.youtube.com/watch?v=xAoljeRJ3lU&feature=youtu.be https://www.youtube.com/watch?v=xAoljeRJ3lU&feature=youtu.be

cscheid 11y ago
This talk is great! And if you watched it all the way to the 6:30 mark, where he says "you can't have negative amounts of light", then you'll really enjoy chimerical colors: https://en.wikipedia.org/wiki/Impossible_color https://en.wikipedia.org/wiki/Impossible_color The trick is you exploit retinal fatigue in order to perceive colors both "above 100%" and "below 0%". Wild stuff.

po84 11y ago

https://github.com/minrk/ligo-binder https://github.com/minrk/ligo-binder Re-run the analysis yourself on Jupyter using Binder. Click the launch binder button.

qhoc 11y ago
Just curious on binder specific here. I could never got it to work out-of-the-box. In this example, it gave me "ImportError: No module named 'seaborn'" error. Now I could do something like !sudo pip install seaborn but still... Did you guys have same experience?

melted 11y ago

Makes for a nice demo of Python Notebook capabilities, especially when it comes to graphing.

anotheryou 11y ago

can anyone tell me how the figured out where the black holes are? with just 2 "ears" I'd expect to be only able to determen a circle, but here is the picture they released: http://content.screencast.com/users/cougarten/folders/Jing/media/64d98a70-7b90-45b6-ac97-2518aa657171/2016-02-11_1650.png http://content.screencast.com/users/cougarten/folders/Jing/m... Do i see a warped circle, or the bottom part of one? In the latter case I wonder how they found out. Are the L-shape sensors capable of seeing some direction depending on which way the phases shift first/last? If you'd build one of the sensors in reverse you'd see a reversed signal, no? Given that they will have optimized the orientations of both stations this is probably how it worked and I'm seeing just part of a circle, right? That one lonely blob might be an unlikely mirrored version.