7 ms·

Recording and visualising the 20k system calls it takes to "import seaborn"

111 points by sYnfo 3y ago

dmwilcox 3y ago
I've been writing python for going on 20 years now and while it was a good language to cut my teeth on thus sort of analysis brings only horror. Many thanks to the author for dropping into plain view. I'm going to go back to learning more C and Forth... And shake my fist at passing clouds :)
- GuestHNUser 3y ago
  ImPlot is small and worth checking out if you don't want to make the plotting functions yourself. https://github.com/epezent/implot https://github.com/epezent/implot
- axchizhov 3y ago
  So, instead of printing a pretty plot in 2 lines of code, you will be... making these 20k syscalls yourself? You have such unusual hobby, my friend!
  - rising-sky 3y ago
    Ha!
  - hnlmorg 3y ago
    Python will do a lot under the hood that a hand-rolled C solution wouldn’t. So I wouldn’t expect the C equivalent to make the same number of syscalls as Python.
  - t8sr 3y ago
    It doesn’t take 20k syscalls to print a plot, the 20k syscalls is for the import call. I would hope that drawing plots takes a lot less. To engage with your point: loading a dynamic library in a regular language takes significantly less than 20k syscalls. Probably 20-40 for C on Linux. Python is uniquely inefficient. On most plots comparing resource use by different languages, in order to even show python together with regular languages like Java and C, either you use the log scale, or everything but Python is shown as a single point. Of course, most people use Python to glue together stuff written in C, so it’s not that big of a deal, but it becomes a problem when people forget pure Python code is literally hundreds or thousands times slower than a “regular” program doing the same thing.
    - 3abiton 3y ago
      Modularity and customization come at a cost. Python is the systemd of computer languages. But it is not trying to sell itself under the KISS banner.
      - t8sr 3y ago
        Oh absolutely, for some tasks Python is amazing. I use Jupyter notebooks a lot, for example, and the flexibility is an incredible feature. It just worries me when I sometimes see those same Jupyter notebooks running in production, crunching 100s of terabytes of data. Maybe I’m wrong, but I didn’t get the impression everyone realizes exactly how wasteful that is. I guess AWS credits are easy to come by. One thing Google did well back in the day, was making resource costs report in SWE/hours, the idea being that you see if you should go and rewrite something. If it cost 100 SWE/h to run, and it only took you a day to cut that in half, you should do it.
        bb88 3y ago
        Numpy is competitive with optimized C/C++. So even if it's running in a Jupyter notebook, it's still going to be insanely fast.
        GuestHNUser 3y ago
        > Numpy is competitive with optimized C/C++ Can you cite a source/example for that? I cannot imagine an optimized C program that doesn't blow python with numpy out of the water. Even a poorly written C program is likely to be 2x faster simply because it doesn't have to round trip operations from C to python and back.
        bb88 3y ago
        I feel like this is google-able, no? I found some metrics after 30 seconds of googling.
        3y ago
- chx 3y ago
  Forth?? Now, that's a name I haven't heard in a long time. A long time.
- pdonis 3y ago
  > I'm going to go back to learning more C and Forth Why would you expect that to decrease the number of syscalls you need? The syscalls are there because the program needs the OS to do things. That need is driven by the application domain, not by the programming language you use.
  - floating-io 3y ago
    Unless the programming language you use happens to perform 20,000 system calls before it ever even runs a line of your actual code...
  - pixelesque 3y ago
    Python's module importing / $PYTHONPATH lookup/traversal is incredibly inefficient, especially with cold FS caches... I've worked at places where we've significantly patched the logic (in a way which breaks compatibility in some cases, so couldn't be up-streamed) which makes Python startup / module loading with hundreds of paths in $PYTHONPATH orders of magnitude faster...
  - Jtsummers 3y ago
    > The syscalls are there because the program needs the OS to do things. Maybe some of them are. Many of those syscalls are there because Python (not the core program someone is creating, but rather its platform) needs the OS to do things. Importing an empty python file takes 28 syscalls (30 measured by their tool, but the last two are closing out the trace not actually related to the import). 29 syscalls if you have any text in it (presumably more for larger files). The logical equivalent in C for many portions of the import process in Python happen at compile + linker time, not during execution. So while it might not be a pleasant experience to develop, a C equivalent of many Python programs would involve far fewer syscalls at execution time.
- josephg 3y ago
  Yeah. I recently worked on a small web project being developed at a university. The project is written in flask, and it presents a reasonably simple UI on top of some data living in a mysql database. When I started on the project, page loads often took 10 seconds or more. The web application is used by about 20 people and that was enough to bring their single beefy server to its knees. Someone in NY tried scraping the site the other week and the site became completely unresponsive. They resorted to banning the IP to keep the website up. The reasons it was slow were all the usual culprits - a misused ORM being the main one. It’s a nice language, but I really felt like I’d been transported back in time a few decades working in it. It feels like I’m using a computer from the 90s where performance choices matter again because the language is so slow. And where dependency management is a circus of half working tools and half hearted attempts at versioning. Packages conflict with one another. Some “pinned” package versions have apparently rusted and won’t actually install on my computer. And the system to install packages locally was obviously bolted on, badly, long after the horse had left the gate. It reminds me of working in C in the early 2000s. I never thought I’d say this but it makes server side JavaScript with npm look positively modern and fast by comparison.
  - amluto 3y ago
    I once worked on a small web project, at a university, in Python, using WSGI IIRC. It loaded a lot faster than any of the big expensive apps the university had written. Well, there was one exception. The little import statement to import the Oracle database client took maybe 15 seconds. MySQL for the win :) (I would not recommend MySQL for new applications today, although I might recommend it over Oracle…)
    - tomrod 3y ago
      Oracle EEE'd MySQL. MariaDB these days. Or postgres along with the rest of the singularity.
  - deleted 3y ago
    [deleted]
  - throwaway828 3y ago
shadycuz 3y ago
I also wrote a somewhat similar tool. I call it deep-ast. It's pretty flexible in what it can track. I used it when refactoring some code in urllib3, to see what Exceptions could get raised along a given code path. https://github.com/DontShaveTheYak/deep-ast https://github.com/DontShaveTheYak/deep-ast
albertgoeswoof 3y ago
Today I asked a devops engineer to tell me how much time a long (3 seconds avg) api call was spending on database queries, application logic, and network etc. He couldn’t understand the request and instead opened up the azure console and recommended we increase the number cpu cores / memory if performance is an issue. I look at posts like this and cry.
- tomrod 3y ago
  Your DevOps engineer needs to learn more about observability. Jaeger or similar could be helpful. Also, with DevOps pushing out traditional administrators, companies are often spending way more on infra than needed.
- jiggawatts 3y ago
  I keep telling people that a hundred buses will let you take a hundred times more people, but nobody will get to their destination a hundred times faster. Inevitably this comment is followed by quiet blinking as they digest this and then this question: “Are you saying we need to scale up one hundred times bigger?” Sigh…
  - QuercusMax 3y ago
    Really makes one wonder what kind of thought process these people go through, and what kind of education they actually had.
    - teddyh 3y ago
      On two occasions I have been asked, — “Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?” In one case a member of the Upper, and in the other a member of the Lower, House put this question. I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question. — Charles Babbage, Passages from the Life of a Philosopher (1864), chapter 5, Difference Engine No. 1
- turtlebits 3y ago
ok123456 3y ago
If it's a large project, I'll use local imports to defer this cost only around where I'm plotting. That way, if I have another entry point that only does computation or is part of a larger system like a web application, it won't have this sort of overhead.
nsm 3y ago
Related to this, if you set the env var `PYTHONPROFILEIMPORTTIME=1` or run python with `-X importtime`, it will print out the cumulative and self times to import various modules. There is then this neat tool to visualize the data. https://kmichel.github.io/python-importtime-graph/ https://kmichel.github.io/python-importtime-graph/ Highly recommend to find the worst imports affecting your program startup time. In general, the python community values tend towards functionality over performance. For example, large modules (looking at networkx here) will often import a bunch of there submodules in their __init__.py, which means all modules now end up loaded even if you didn't need them. I've never tried https://pyoxidizer.readthedocs.io/en/stable/oxidized_importer.html https://pyoxidizer.readthedocs.io/en/stable/oxidized_importe..., but it compiles all the imports into one, memory mapped file, that _may_ speed up the importing. Having everything compiled to bytecode also helps a bunch.
- lights0123 3y ago
  At the other extreme, checkpointing the whole process once all imports have been resolved and restoring it for every execution can be used for frequently-run tools: https://github.com/albertz/python-preloaded https://github.com/albertz/python-preloaded
mrnonchalant 3y ago
Very cool! I liked the instruction counting article as well.
rldjbpin 3y ago
https://archive.is/XVWSO https://archive.is/XVWSO Aggressive anti-adblock plugin used.

111 points by sYnfo 3y ago

I've been writing python for going on 20 years now and while it was a good language to cut my teeth on thus sort of analysis brings only horror. Many thanks to the author for dropping into plain view. I'm going to go back to learning more C and Forth... And shake my fist at passing clouds :)

GuestHNUser 3y ago
ImPlot is small and worth checking out if you don't want to make the plotting functions yourself. https://github.com/epezent/implot https://github.com/epezent/implot
axchizhov 3y ago
So, instead of printing a pretty plot in 2 lines of code, you will be... making these 20k syscalls yourself? You have such unusual hobby, my friend!
- rising-sky 3y ago
  Ha!
- hnlmorg 3y ago
  Python will do a lot under the hood that a hand-rolled C solution wouldn’t. So I wouldn’t expect the C equivalent to make the same number of syscalls as Python.
- t8sr 3y ago
  It doesn’t take 20k syscalls to print a plot, the 20k syscalls is for the import call. I would hope that drawing plots takes a lot less. To engage with your point: loading a dynamic library in a regular language takes significantly less than 20k syscalls. Probably 20-40 for C on Linux. Python is uniquely inefficient. On most plots comparing resource use by different languages, in order to even show python together with regular languages like Java and C, either you use the log scale, or everything but Python is shown as a single point. Of course, most people use Python to glue together stuff written in C, so it’s not that big of a deal, but it becomes a problem when people forget pure Python code is literally hundreds or thousands times slower than a “regular” program doing the same thing.
  - 3abiton 3y ago
    Modularity and customization come at a cost. Python is the systemd of computer languages. But it is not trying to sell itself under the KISS banner.
    - t8sr 3y ago
      Oh absolutely, for some tasks Python is amazing. I use Jupyter notebooks a lot, for example, and the flexibility is an incredible feature. It just worries me when I sometimes see those same Jupyter notebooks running in production, crunching 100s of terabytes of data. Maybe I’m wrong, but I didn’t get the impression everyone realizes exactly how wasteful that is. I guess AWS credits are easy to come by. One thing Google did well back in the day, was making resource costs report in SWE/hours, the idea being that you see if you should go and rewrite something. If it cost 100 SWE/h to run, and it only took you a day to cut that in half, you should do it.
      - bb88 3y ago
        Numpy is competitive with optimized C/C++. So even if it's running in a Jupyter notebook, it's still going to be insanely fast.
        GuestHNUser 3y ago
        > Numpy is competitive with optimized C/C++ Can you cite a source/example for that? I cannot imagine an optimized C program that doesn't blow python with numpy out of the water. Even a poorly written C program is likely to be 2x faster simply because it doesn't have to round trip operations from C to python and back.
        bb88 3y ago
        I feel like this is google-able, no? I found some metrics after 30 seconds of googling.
        3y ago
chx 3y ago
Forth?? Now, that's a name I haven't heard in a long time. A long time.
pdonis 3y ago
> I'm going to go back to learning more C and Forth Why would you expect that to decrease the number of syscalls you need? The syscalls are there because the program needs the OS to do things. That need is driven by the application domain, not by the programming language you use.
- floating-io 3y ago
  Unless the programming language you use happens to perform 20,000 system calls before it ever even runs a line of your actual code...
- pixelesque 3y ago
  Python's module importing / $PYTHONPATH lookup/traversal is incredibly inefficient, especially with cold FS caches... I've worked at places where we've significantly patched the logic (in a way which breaks compatibility in some cases, so couldn't be up-streamed) which makes Python startup / module loading with hundreds of paths in $PYTHONPATH orders of magnitude faster...
- Jtsummers 3y ago
  > The syscalls are there because the program needs the OS to do things. Maybe some of them are. Many of those syscalls are there because Python (not the core program someone is creating, but rather its platform) needs the OS to do things. Importing an empty python file takes 28 syscalls (30 measured by their tool, but the last two are closing out the trace not actually related to the import). 29 syscalls if you have any text in it (presumably more for larger files). The logical equivalent in C for many portions of the import process in Python happen at compile + linker time, not during execution. So while it might not be a pleasant experience to develop, a C equivalent of many Python programs would involve far fewer syscalls at execution time.
josephg 3y ago
Yeah. I recently worked on a small web project being developed at a university. The project is written in flask, and it presents a reasonably simple UI on top of some data living in a mysql database. When I started on the project, page loads often took 10 seconds or more. The web application is used by about 20 people and that was enough to bring their single beefy server to its knees. Someone in NY tried scraping the site the other week and the site became completely unresponsive. They resorted to banning the IP to keep the website up. The reasons it was slow were all the usual culprits - a misused ORM being the main one. It’s a nice language, but I really felt like I’d been transported back in time a few decades working in it. It feels like I’m using a computer from the 90s where performance choices matter again because the language is so slow. And where dependency management is a circus of half working tools and half hearted attempts at versioning. Packages conflict with one another. Some “pinned” package versions have apparently rusted and won’t actually install on my computer. And the system to install packages locally was obviously bolted on, badly, long after the horse had left the gate. It reminds me of working in C in the early 2000s. I never thought I’d say this but it makes server side JavaScript with npm look positively modern and fast by comparison.
- amluto 3y ago
  I once worked on a small web project, at a university, in Python, using WSGI IIRC. It loaded a lot faster than any of the big expensive apps the university had written. Well, there was one exception. The little import statement to import the Oracle database client took maybe 15 seconds. MySQL for the win :) (I would not recommend MySQL for new applications today, although I might recommend it over Oracle…)
  - tomrod 3y ago
    Oracle EEE'd MySQL. MariaDB these days. Or postgres along with the rest of the singularity.
- deleted 3y ago
  [deleted]
- throwaway828 3y ago

shadycuz 3y ago

I also wrote a somewhat similar tool. I call it deep-ast. It's pretty flexible in what it can track. I used it when refactoring some code in urllib3, to see what Exceptions could get raised along a given code path. https://github.com/DontShaveTheYak/deep-ast https://github.com/DontShaveTheYak/deep-ast

albertgoeswoof 3y ago

Today I asked a devops engineer to tell me how much time a long (3 seconds avg) api call was spending on database queries, application logic, and network etc. He couldn’t understand the request and instead opened up the azure console and recommended we increase the number cpu cores / memory if performance is an issue. I look at posts like this and cry.

tomrod 3y ago
Your DevOps engineer needs to learn more about observability. Jaeger or similar could be helpful. Also, with DevOps pushing out traditional administrators, companies are often spending way more on infra than needed.
jiggawatts 3y ago
I keep telling people that a hundred buses will let you take a hundred times more people, but nobody will get to their destination a hundred times faster. Inevitably this comment is followed by quiet blinking as they digest this and then this question: “Are you saying we need to scale up one hundred times bigger?” Sigh…
- QuercusMax 3y ago
  Really makes one wonder what kind of thought process these people go through, and what kind of education they actually had.
  - teddyh 3y ago
    On two occasions I have been asked, — “Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?” In one case a member of the Upper, and in the other a member of the Lower, House put this question. I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question. — Charles Babbage, Passages from the Life of a Philosopher (1864), chapter 5, Difference Engine No. 1
turtlebits 3y ago

ok123456 3y ago

If it's a large project, I'll use local imports to defer this cost only around where I'm plotting. That way, if I have another entry point that only does computation or is part of a larger system like a web application, it won't have this sort of overhead.

nsm 3y ago

Related to this, if you set the env var `PYTHONPROFILEIMPORTTIME=1` or run python with `-X importtime`, it will print out the cumulative and self times to import various modules. There is then this neat tool to visualize the data. https://kmichel.github.io/python-importtime-graph/ https://kmichel.github.io/python-importtime-graph/ Highly recommend to find the worst imports affecting your program startup time. In general, the python community values tend towards functionality over performance. For example, large modules (looking at networkx here) will often import a bunch of there submodules in their __init__.py, which means all modules now end up loaded even if you didn't need them. I've never tried https://pyoxidizer.readthedocs.io/en/stable/oxidized_importer.html https://pyoxidizer.readthedocs.io/en/stable/oxidized_importe..., but it compiles all the imports into one, memory mapped file, that _may_ speed up the importing. Having everything compiled to bytecode also helps a bunch.

lights0123 3y ago
At the other extreme, checkpointing the whole process once all imports have been resolved and restoring it for every execution can be used for frequently-run tools: https://github.com/albertz/python-preloaded https://github.com/albertz/python-preloaded

mrnonchalant 3y ago

Very cool! I liked the instruction counting article as well.

rldjbpin 3y ago

https://archive.is/XVWSO https://archive.is/XVWSO Aggressive anti-adblock plugin used.