9 ms·

Scientific Visualization: Python + Matplotlib (2021)

177 points by danso 2y ago

jszymborski 2y ago
Love Mr. Rougier's open books. I supported this one while it was in development and I wasn't disappointment by the final product. I've long awaited for him to finish "Python & OpenGL for Scientific Visualization" [0] but I'll take this in the mean time :P [0] https://www.labri.fr/perso/nrougier/python-opengl/#python-opengl-for-scientific-visualization https://www.labri.fr/perso/nrougier/python-opengl/#python-op...
- astrolx 2y ago
  Very nice guy too! Met him at a workshop recently and got a hardcopy of the book
- rossant 2y ago
  I second this! Nicolas and I have been working hard lately to try to advance the field of scientific visualization in Python — stay tuned!
- Iwan-Zotow 2y ago
  Looks good, thanks
aborsy 2y ago
Nice book. But frankly Tikz and pgfplots are better for scientific visualization, unless your plot is too complex (in which case you have to settle for a lower quality plot).
- jtrueb 2y ago
  How do you use those outside of Latex? Lots of figures are coming from Matlab or Python, where Latex is a non-starter. Is there a cookbook to pollinate?
  - aborsy 2y ago
    You export figure in PDF or some other format.
- adityamwagh 2y ago
  TikZ has a steep learning curve and felt so convoluted when I tried to use it.
  - whatever1 2y ago
    Great use case for llms?
    - semi-extrinsic 2y ago
      I would actually be very surprised if that is true. tikz is one of those examples (like e.g. M4 macros or assembly code or OpenSCAD) where producing working code requires significant mental arithmetic. Most skilled humans need to write down intermediate calculations on a piece of paper. If you don't believe me, look at the "tikz unicorn" of GPT4 demo fame. Then try to replicate that with the currently available "degraded because of safety" GPT4 versions. Then ask yourself how on Earth you could use that to produce publication quality graphics.
      - svantana 2y ago
        I tried the "Draw a unicorn in TiKZ" in chatGPT now, and it actually looks even better than in that paper, and the code is very clean too. Not surprising as the quality of outputs from chatbots have steadily improved in the ~17 months since.
  - atrettel 2y ago
    TikZ does have a steep learning curve and can get pretty convoluted, but once you get a hang of it, you have a tremendous amount of control over what you are doing. I sometimes feel like TikZ is more like TeX instead of LaTeX given how primitive it can be, but it can be quite powerful if you give it enough time.
- setopt 2y ago
  Matplotlib is very easy to customize via “style sheets”, so you can make it match any style you want. Note that you can even turn on the “usetex” option and provide a preamble to compile all text/equations via TeX.
  - aborsy 2y ago
    No doubt it’s very customizable. But the display quality seems to me not as good as even matlab. At least with defaults, which is a bit barebones. You have to work on it . The syntax of python is also a bit verbose. Like, why do we need plt.show(), when I plot a figure obviously I want to plot it! Now I know, the point is, see matlab, gnuplot etc. Compare with the default plot of gnuplot. The limitation of latex is you can’t do much calculation with data, and memory usage. Don’t get me wrong, I still sometimes go to python for plotting. But it’s general purpose programming language, and for specific applications there are custom tools.
    - setopt 2y ago
      Don’t get me wrong either – I personally prefer Gnuplot by quite a margin, Matplotlib is indeed too verbose. These days I use Matplotlib more mainly because I use Python more. I’m just saying that it doesn’t take too much configuration to make publication-quality plots in Matplotlib either, by e.g. enabling the TeX integration and setting up a reusable stylesheet file. The style sheet format is actually very reasonable, and several examples ship with Matplotlib. There’s also the Seaborn package if you want some nicer default stylesheets. (Side note, the same goes for Gnuplot… As shown by e.g. gnuplotting.org you can definitely create publication-quality plots there as well.)
      - davrosthedalek 2y ago
        gnuplot + tikz terminal. I have yet to see better looking figures. And technically, it's one of the rare if not the only one solution that can give you bibtex resolved citations in figure labels.... (Don't do it, the journals hate it)
    - Keysh 2y ago
- bsdz 2y ago
  You can use matplotlib's pgf backend for direct use in latex. That said, not sure I agree that's better for scientific viz. Perhaps in the past. Also tikz/pgf is much more fiddly and has it's limitations too (imho).
lemursage 2y ago
I really like the book's subchapter on colours, wish it was even more elaborated on. Colours are one of the subtle things I so often find difficult to get right. As to using matplotlib in published research: when I started out as an undergrad, everybody in the research team used OriginLab for plotting -- my impression of it then was pretty good. At some point, I started using matplotlib + Latex + science plots and it caught on, mostly because there's no need to shift all the data around to a separate programme. Scienceplots package does heavy lifting with fonts and styling for specific journals, so it's just a matter of designing the right plot geometry and information density [1]. [1] an obligatory Tufte citation.
fecal_henge 2y ago
As an Engineer (but in a scientific field) I just love Matplotlib. I moved a number of years ago from Matlab and even Excel (love how I still see papers with that mid grey background and full magenta on figures). I'm a poor data analyst and programmer but have always managed to do what I need, mostly just time series and scatter plots. Sometimes a little more involved for my ability: such as a set of heat maps with common scaling and some tiles omitted... I see some comments about the quality of the rendering and the look, but I think the default is really good. Nice proportionality of text, lineweight etc IMO. Anyway I'll get to my point. I really can't get any value out of the Matplotlib official documentation. I'm not going to criticize it, just say that its not compatible with my brain. On this basis, good quality and accessible literature like this is very well received.
- analog31 2y ago
  I find the same thing about the official docs. I can't really put my finger on the reason, except that it might be due to the vastness of MPL and the need for the docs to auto-update with each revision cycle. When I do find useful docs, they're usually in the form of an example that I can use as a starting point. Referring to the docs comes later if needed. The consequence is that I probably use only 1% of MPL's capabilities, but that's already more than I could ever have imagined. I've only been using Google Copilot for a few weeks, so it's too soon to know if that's how I'll deal with the situation in the longer term.
wodenokoto 2y ago
Maybe I can’t see the forest for the trees but is there an html version available?
nofunsir 2y ago
Python ^ ( matplotlib v pyqtgraph ) is my goto
masterkram 2y ago
For the folks who don't like the default style of matplotlib, fonts, colors etc... I made a stylesheet for my plots based on tailwindcss which you can use: https://www.bruderer.ai/blog/matplotlib https://www.bruderer.ai/blog/matplotlib
- moandcompany 2y ago
  Additionally, Seaborn (https://seaborn.pydata.org/ https://seaborn.pydata.org/) is a great mention for people that want to use Matplotlib with better default aesthetics, amongst other conveniences: "Seaborn is a Python data visualization library based on matplotlib. It provides a high-level interface for drawing attractive and informative statistical graphics."
  - slashdave 2y ago
    Better is subjective. Not everyone likes Seaborn's defaults.
weinzierl 2y ago
My weapon of choice is still R + dyplr + ggplot2 - mostly because I have been using it for so long and know it by heart. I'd love to try something new, but don't feel the whole Python world is it. Is there any modern take - doesn't have to be production ready but should show a promising future? Anything from a more modern ecosystem, like Rust or zig, maybe?
- fluidcruft 2y ago
  It probably depends on your interests but I'm excited about the efforts using GPU rendering for interactivity (it's related to vispy but I forget the latest status of whether vispy is the current focus... I think the last I remember is someone was prototyping a vulkan-based system).
  - fluidcruft 2y ago
    Edit: the vulkan-based system I was thinking of is datovis
- cauthon 2y ago
  Why on earth would you want to do exploratory data analysis in a compiled language
  - weinzierl 2y ago
    Not necessarily in Rust itself. Actually I like R as a language. I was more thinking something along the lines of Typist. A modern take of a proven concept, possibly with a Rust inspired scripting language. https://typst.app/docs/reference/scripting https://typst.app/docs/reference/scripting
- slashdave 2y ago
  Is ggplot2 still using hyphens for minus signs? A pet peeve of mine.
- jmount 2y ago
  plotnine is a really good grammar of graphics at this point.
huem0n 2y ago
Please for the love of reproducibly, interactivity, and not having 200 system dependencies use plotly instead of matplotlib.
- patrick451 2y ago
  Plotly is only useful if you need to display low hundreds of data points. For anything more, it becomes so slow it's quite literally unusable.
- cactusfrog 2y ago
  Interactivity is not desirable in all situations. For static figures, matplotlib is still king.
- KeplerBoy 2y ago
  Can plotly generate publication ready vector graphics with Latex texts?
- mistrial9 2y ago
  plotly has had ten years to take over the world?
plaidfuji 2y ago
I work with chemical and materials companies on full stack data capabilities and interactive viz / dashboarding is a recurring pain point. Frequently there are larger scale processes (beyond a single engineer’s or scientist’s scope) that still require scientific-level interactive viz that the likes of PowerBI or Tableau can’t (or won’t?) provide… if your company even has a subscription. Things like being able to dynamically re-group/nest variables and recalculate statistical tests, dual axis capabilities, automatic sig fig number reformatting, just all kinds of quality-of-life features that in some cases are possible but in most cases are considered extreme edge cases and require too much manual config / aren’t templatizable. Of course on the other end you’ve got the whole Python/matplotlib/seaborn/bokeh/plotly/vega/altair “ecosystem” (although it’s more of a swamp if you ask me), which require someone to maintain Python code and a means to stand up an internal server. Not to mention that most use cases require significant customization. Plotly Dash always seems somewhat promising but as someone below mentioned it’s actually kind of slow? Every time I try it I’m just kind of underwhelmed. I hear ggplot in R is good but I’ve never used R and it’s hard to get a critical mass of people in a company behind R so that’s kind of off the table. The only programs that really get the aesthetics of scientific plotting right without a ton of customization are JMP, Origin, and Igor Pro (props if you’ve heard of it), but these are all desktop apps… although JMP is starting to make a push into cloud-hosted stuff. I guess all that is to say if anyone is interested in starting a company in this space, let me know.
- hantusk 2y ago
  I think https://github.com/uwdata/mosaic https://github.com/uwdata/mosaic is really promising here. See the example https://idl.uw.edu/mosaic/examples/linear-regression.html https://idl.uw.edu/mosaic/examples/linear-regression.html where the user can recalculate a linear regression based on their selection. You'd still need to implement any custom selection widgets, data transformations (like other statistical tests) etc. still missing, but i like the technical design to build on top off. It uses https://github.com/observablehq/plot https://github.com/observablehq/plot under the hood, which aims to have just as flexible a grammar as ggplot (already quite capable) but with interactive features (built by the creator of d3 and uses it under its hood).
- cactusfrog 2y ago
morotter 2y ago
I can recommend proplot (https://github.com/proplot-dev/proplot https://github.com/proplot-dev/proplot) as a ”beautifier” wrapper for Matplotlib—particularly useful for scientific publications
- s1291 2y ago
  SciencePlots is another option: https://github.com/garrettj403/SciencePlots https://github.com/garrettj403/SciencePlots
IshKebab 2y ago
Some of these examples are surprisingly decent. When I tried matplotlib I found it was ugly by default; I guess it has improved a lot. For publication quality graphs I tend to use GLE. Not sure why it isn't better known, but check out the examples here: https://glx.sourceforge.io/examples/2dplots/index.html https://glx.sourceforge.io/examples/2dplots/index.html

177 points by danso 2y ago

Love Mr. Rougier's open books. I supported this one while it was in development and I wasn't disappointment by the final product. I've long awaited for him to finish "Python & OpenGL for Scientific Visualization" [0] but I'll take this in the mean time :P [0] https://www.labri.fr/perso/nrougier/python-opengl/#python-opengl-for-scientific-visualization https://www.labri.fr/perso/nrougier/python-opengl/#python-op...

astrolx 2y ago
Very nice guy too! Met him at a workshop recently and got a hardcopy of the book
rossant 2y ago
I second this! Nicolas and I have been working hard lately to try to advance the field of scientific visualization in Python — stay tuned!
Iwan-Zotow 2y ago
Looks good, thanks

aborsy 2y ago

Nice book. But frankly Tikz and pgfplots are better for scientific visualization, unless your plot is too complex (in which case you have to settle for a lower quality plot).

jtrueb 2y ago
How do you use those outside of Latex? Lots of figures are coming from Matlab or Python, where Latex is a non-starter. Is there a cookbook to pollinate?
- aborsy 2y ago
  You export figure in PDF or some other format.
adityamwagh 2y ago
TikZ has a steep learning curve and felt so convoluted when I tried to use it.
- whatever1 2y ago
  Great use case for llms?
  - semi-extrinsic 2y ago
    I would actually be very surprised if that is true. tikz is one of those examples (like e.g. M4 macros or assembly code or OpenSCAD) where producing working code requires significant mental arithmetic. Most skilled humans need to write down intermediate calculations on a piece of paper. If you don't believe me, look at the "tikz unicorn" of GPT4 demo fame. Then try to replicate that with the currently available "degraded because of safety" GPT4 versions. Then ask yourself how on Earth you could use that to produce publication quality graphics.
    - svantana 2y ago
      I tried the "Draw a unicorn in TiKZ" in chatGPT now, and it actually looks even better than in that paper, and the code is very clean too. Not surprising as the quality of outputs from chatbots have steadily improved in the ~17 months since.
- atrettel 2y ago
  TikZ does have a steep learning curve and can get pretty convoluted, but once you get a hang of it, you have a tremendous amount of control over what you are doing. I sometimes feel like TikZ is more like TeX instead of LaTeX given how primitive it can be, but it can be quite powerful if you give it enough time.
setopt 2y ago
Matplotlib is very easy to customize via “style sheets”, so you can make it match any style you want. Note that you can even turn on the “usetex” option and provide a preamble to compile all text/equations via TeX.
- aborsy 2y ago
  No doubt it’s very customizable. But the display quality seems to me not as good as even matlab. At least with defaults, which is a bit barebones. You have to work on it . The syntax of python is also a bit verbose. Like, why do we need plt.show(), when I plot a figure obviously I want to plot it! Now I know, the point is, see matlab, gnuplot etc. Compare with the default plot of gnuplot. The limitation of latex is you can’t do much calculation with data, and memory usage. Don’t get me wrong, I still sometimes go to python for plotting. But it’s general purpose programming language, and for specific applications there are custom tools.
  - setopt 2y ago
    Don’t get me wrong either – I personally prefer Gnuplot by quite a margin, Matplotlib is indeed too verbose. These days I use Matplotlib more mainly because I use Python more. I’m just saying that it doesn’t take too much configuration to make publication-quality plots in Matplotlib either, by e.g. enabling the TeX integration and setting up a reusable stylesheet file. The style sheet format is actually very reasonable, and several examples ship with Matplotlib. There’s also the Seaborn package if you want some nicer default stylesheets. (Side note, the same goes for Gnuplot… As shown by e.g. gnuplotting.org you can definitely create publication-quality plots there as well.)
    - davrosthedalek 2y ago
      gnuplot + tikz terminal. I have yet to see better looking figures. And technically, it's one of the rare if not the only one solution that can give you bibtex resolved citations in figure labels.... (Don't do it, the journals hate it)
  - Keysh 2y ago
bsdz 2y ago
You can use matplotlib's pgf backend for direct use in latex. That said, not sure I agree that's better for scientific viz. Perhaps in the past. Also tikz/pgf is much more fiddly and has it's limitations too (imho).

lemursage 2y ago

I really like the book's subchapter on colours, wish it was even more elaborated on. Colours are one of the subtle things I so often find difficult to get right. As to using matplotlib in published research: when I started out as an undergrad, everybody in the research team used OriginLab for plotting -- my impression of it then was pretty good. At some point, I started using matplotlib + Latex + science plots and it caught on, mostly because there's no need to shift all the data around to a separate programme. Scienceplots package does heavy lifting with fonts and styling for specific journals, so it's just a matter of designing the right plot geometry and information density [1]. [1] an obligatory Tufte citation.

fecal_henge 2y ago

As an Engineer (but in a scientific field) I just love Matplotlib. I moved a number of years ago from Matlab and even Excel (love how I still see papers with that mid grey background and full magenta on figures). I'm a poor data analyst and programmer but have always managed to do what I need, mostly just time series and scatter plots. Sometimes a little more involved for my ability: such as a set of heat maps with common scaling and some tiles omitted... I see some comments about the quality of the rendering and the look, but I think the default is really good. Nice proportionality of text, lineweight etc IMO. Anyway I'll get to my point. I really can't get any value out of the Matplotlib official documentation. I'm not going to criticize it, just say that its not compatible with my brain. On this basis, good quality and accessible literature like this is very well received.

analog31 2y ago
I find the same thing about the official docs. I can't really put my finger on the reason, except that it might be due to the vastness of MPL and the need for the docs to auto-update with each revision cycle. When I do find useful docs, they're usually in the form of an example that I can use as a starting point. Referring to the docs comes later if needed. The consequence is that I probably use only 1% of MPL's capabilities, but that's already more than I could ever have imagined. I've only been using Google Copilot for a few weeks, so it's too soon to know if that's how I'll deal with the situation in the longer term.

wodenokoto 2y ago

Maybe I can’t see the forest for the trees but is there an html version available?

nofunsir 2y ago

Python ^ ( matplotlib v pyqtgraph ) is my goto

masterkram 2y ago

For the folks who don't like the default style of matplotlib, fonts, colors etc... I made a stylesheet for my plots based on tailwindcss which you can use: https://www.bruderer.ai/blog/matplotlib https://www.bruderer.ai/blog/matplotlib

moandcompany 2y ago
Additionally, Seaborn (https://seaborn.pydata.org/ https://seaborn.pydata.org/) is a great mention for people that want to use Matplotlib with better default aesthetics, amongst other conveniences: "Seaborn is a Python data visualization library based on matplotlib. It provides a high-level interface for drawing attractive and informative statistical graphics."
- slashdave 2y ago
  Better is subjective. Not everyone likes Seaborn's defaults.

weinzierl 2y ago

My weapon of choice is still R + dyplr + ggplot2 - mostly because I have been using it for so long and know it by heart. I'd love to try something new, but don't feel the whole Python world is it. Is there any modern take - doesn't have to be production ready but should show a promising future? Anything from a more modern ecosystem, like Rust or zig, maybe?

fluidcruft 2y ago
It probably depends on your interests but I'm excited about the efforts using GPU rendering for interactivity (it's related to vispy but I forget the latest status of whether vispy is the current focus... I think the last I remember is someone was prototyping a vulkan-based system).
- fluidcruft 2y ago
  Edit: the vulkan-based system I was thinking of is datovis
cauthon 2y ago
Why on earth would you want to do exploratory data analysis in a compiled language
- weinzierl 2y ago
  Not necessarily in Rust itself. Actually I like R as a language. I was more thinking something along the lines of Typist. A modern take of a proven concept, possibly with a Rust inspired scripting language. https://typst.app/docs/reference/scripting https://typst.app/docs/reference/scripting
slashdave 2y ago
Is ggplot2 still using hyphens for minus signs? A pet peeve of mine.
jmount 2y ago
plotnine is a really good grammar of graphics at this point.

huem0n 2y ago

Please for the love of reproducibly, interactivity, and not having 200 system dependencies use plotly instead of matplotlib.

patrick451 2y ago
Plotly is only useful if you need to display low hundreds of data points. For anything more, it becomes so slow it's quite literally unusable.
cactusfrog 2y ago
Interactivity is not desirable in all situations. For static figures, matplotlib is still king.
KeplerBoy 2y ago
Can plotly generate publication ready vector graphics with Latex texts?
mistrial9 2y ago
plotly has had ten years to take over the world?

plaidfuji 2y ago

I work with chemical and materials companies on full stack data capabilities and interactive viz / dashboarding is a recurring pain point. Frequently there are larger scale processes (beyond a single engineer’s or scientist’s scope) that still require scientific-level interactive viz that the likes of PowerBI or Tableau can’t (or won’t?) provide… if your company even has a subscription. Things like being able to dynamically re-group/nest variables and recalculate statistical tests, dual axis capabilities, automatic sig fig number reformatting, just all kinds of quality-of-life features that in some cases are possible but in most cases are considered extreme edge cases and require too much manual config / aren’t templatizable. Of course on the other end you’ve got the whole Python/matplotlib/seaborn/bokeh/plotly/vega/altair “ecosystem” (although it’s more of a swamp if you ask me), which require someone to maintain Python code and a means to stand up an internal server. Not to mention that most use cases require significant customization. Plotly Dash always seems somewhat promising but as someone below mentioned it’s actually kind of slow? Every time I try it I’m just kind of underwhelmed. I hear ggplot in R is good but I’ve never used R and it’s hard to get a critical mass of people in a company behind R so that’s kind of off the table. The only programs that really get the aesthetics of scientific plotting right without a ton of customization are JMP, Origin, and Igor Pro (props if you’ve heard of it), but these are all desktop apps… although JMP is starting to make a push into cloud-hosted stuff. I guess all that is to say if anyone is interested in starting a company in this space, let me know.

hantusk 2y ago
I think https://github.com/uwdata/mosaic https://github.com/uwdata/mosaic is really promising here. See the example https://idl.uw.edu/mosaic/examples/linear-regression.html https://idl.uw.edu/mosaic/examples/linear-regression.html where the user can recalculate a linear regression based on their selection. You'd still need to implement any custom selection widgets, data transformations (like other statistical tests) etc. still missing, but i like the technical design to build on top off. It uses https://github.com/observablehq/plot https://github.com/observablehq/plot under the hood, which aims to have just as flexible a grammar as ggplot (already quite capable) but with interactive features (built by the creator of d3 and uses it under its hood).
cactusfrog 2y ago

morotter 2y ago

I can recommend proplot (https://github.com/proplot-dev/proplot https://github.com/proplot-dev/proplot) as a ”beautifier” wrapper for Matplotlib—particularly useful for scientific publications

s1291 2y ago
SciencePlots is another option: https://github.com/garrettj403/SciencePlots https://github.com/garrettj403/SciencePlots

IshKebab 2y ago

Some of these examples are surprisingly decent. When I tried matplotlib I found it was ugly by default; I guess it has improved a lot. For publication quality graphs I tend to use GLE. Not sure why it isn't better known, but check out the examples here: https://glx.sourceforge.io/examples/2dplots/index.html https://glx.sourceforge.io/examples/2dplots/index.html