3 ms·

> Reasonably simple > 35 minutes to compile What kind of CPU are we talking about here!?

pjmlp 3y ago
Most likely a single or dual core CPU. I have similar compilation times with Rust on an old Asus 1215B, where 8GB and SSD hardly help the compile the world from scratch cargo model, when starting a new project.
- kgwgk 3y ago
  reasonably simple notebook != compile the world from scratch
  - pjmlp 3y ago
    It is, when the libraries aren't shipped as native libraries, and one depends on the slow LLVM to compile them.
  - eklavya 3y ago
    But that needs to happen everytime rustc is updated.
  - currymj 3y ago
    if a small piece of code depends on a few big, complex packages then depending on how things work out, due to the JIT model, you might have needed to essentially recompile all these dependencies at runtime every time. now there are increasingly better precompilation tools to avoid this.
  - deleted 3y ago
    [deleted]
  - ChrisRackauckas 3y ago
    IIRC there was a bug triggering precompilation each time you re-instantiated manifests, and Pluto notebooks keep a manifest of all packages that they used (in order to have full reproducibility, so it doesn't necessarily match your global environment), so Pluto notebooks would effectively precompile, compile, and run each time. I forget the PR that changed this or I would link it, but IIRC somewhere around v1.9-RC2 this was addressed so the v1.9 release should be much nicer to use for Pluto notebooks. I need to double check this myself though since I was last testing Pluto in some of the betas and reported this behavior.
cookieperson 3y ago
Probably a decent one. Compilation of even small size projects easily took hours as of a year ago. All while people were screaming about how good it was. The equivalent code in other languages would be statically compiled in milliseconds. The size of the binaries was also thousands of times bigger...
- Sukera 3y ago
  It would be great if you could share such an example work load - such big examples are often few and far between, even though they are VERY good for debugging compilation performance.
  - cookieperson 3y ago
    Grab any project you have with more than 2k lines, multiple dependencies and run package compiler on it. Wait an hour, hopefully it didn't barf irreconcilably and check the file size.
    - hpcjoe 3y ago
      My packages at work are about 1.5k lines. Precompilation is about 10-15 seconds. After precomp, about 1-2 seconds to load with "import" or "using". The longest I've seen on precomp have been DifferentialEquations.jl. I've not used it in a while, though I have plans to for personal (non-work related) projects. My packages have 5-10 dependencies in them, I tend to keep my packages/tooling streamlined, and I performance optimize them (everything from loading time, through data structure/algorithm implementation) quite thoroughly. Other users at my firm are adopting Julia as well. It isn't displacing python, though that is a possible future. Its similar enough that many grasp it right away. Its fast enough that its a viable alternative to Python + C++. This said, Julia is not a silver bullet[1], though it is an excellent programming language. It has been able to do all of the same tasks as my python code. Faster (often multiple orders of magnitude) working with 10s to 100s of GB of data, in parallel. Its a better solution for my workflow, and an increasing number of others. If its not to your liking, that's fine. No need to try to push it down with, from my vantage point, what seems like specious claims (very long precompilation times close to an hour, or so). If you have such actual examples, please post them. I'd love to see them. Chances are we could optimize this fairly easily. [1] https://www.merriam-webster.com/dictionary/silver%20bullet https://www.merriam-webster.com/dictionary/silver%20bullet

Most likely a single or dual core CPU. I have similar compilation times with Rust on an old Asus 1215B, where 8GB and SSD hardly help the compile the world from scratch cargo model, when starting a new project.

kgwgk 3y ago
reasonably simple notebook != compile the world from scratch
- pjmlp 3y ago
  It is, when the libraries aren't shipped as native libraries, and one depends on the slow LLVM to compile them.
- eklavya 3y ago
  But that needs to happen everytime rustc is updated.
- currymj 3y ago
  if a small piece of code depends on a few big, complex packages then depending on how things work out, due to the JIT model, you might have needed to essentially recompile all these dependencies at runtime every time. now there are increasingly better precompilation tools to avoid this.
- deleted 3y ago
  [deleted]
- ChrisRackauckas 3y ago
  IIRC there was a bug triggering precompilation each time you re-instantiated manifests, and Pluto notebooks keep a manifest of all packages that they used (in order to have full reproducibility, so it doesn't necessarily match your global environment), so Pluto notebooks would effectively precompile, compile, and run each time. I forget the PR that changed this or I would link it, but IIRC somewhere around v1.9-RC2 this was addressed so the v1.9 release should be much nicer to use for Pluto notebooks. I need to double check this myself though since I was last testing Pluto in some of the betas and reported this behavior.

cookieperson 3y ago

Probably a decent one. Compilation of even small size projects easily took hours as of a year ago. All while people were screaming about how good it was. The equivalent code in other languages would be statically compiled in milliseconds. The size of the binaries was also thousands of times bigger...

Sukera 3y ago
It would be great if you could share such an example work load - such big examples are often few and far between, even though they are VERY good for debugging compilation performance.
- cookieperson 3y ago
  Grab any project you have with more than 2k lines, multiple dependencies and run package compiler on it. Wait an hour, hopefully it didn't barf irreconcilably and check the file size.
  - hpcjoe 3y ago
    My packages at work are about 1.5k lines. Precompilation is about 10-15 seconds. After precomp, about 1-2 seconds to load with "import" or "using". The longest I've seen on precomp have been DifferentialEquations.jl. I've not used it in a while, though I have plans to for personal (non-work related) projects. My packages have 5-10 dependencies in them, I tend to keep my packages/tooling streamlined, and I performance optimize them (everything from loading time, through data structure/algorithm implementation) quite thoroughly. Other users at my firm are adopting Julia as well. It isn't displacing python, though that is a possible future. Its similar enough that many grasp it right away. Its fast enough that its a viable alternative to Python + C++. This said, Julia is not a silver bullet[1], though it is an excellent programming language. It has been able to do all of the same tasks as my python code. Faster (often multiple orders of magnitude) working with 10s to 100s of GB of data, in parallel. Its a better solution for my workflow, and an increasing number of others. If its not to your liking, that's fine. No need to try to push it down with, from my vantage point, what seems like specious claims (very long precompilation times close to an hour, or so). If you have such actual examples, please post them. I'd love to see them. Chances are we could optimize this fairly easily. [1] https://www.merriam-webster.com/dictionary/silver%20bullet https://www.merriam-webster.com/dictionary/silver%20bullet