4 ms·

I think there is a pretty big base of people who do big data work using Numpy and Pandas (Fintech, etc). They want to squeeze every bit of computing power out o

I think there is a pretty big base of people who do big data work using Numpy and Pandas (Fintech, etc). They want to squeeze every bit of computing power out of the specific Intel chipset, GPUs, etc and Intel's distro really helps them out. A 10% speed improvement on 1000's of jobs could in theory save you a nice chunk of time. This becomes very important in the financial market where you need batch jobs to be finished before markets open, or you just want to save 10% on your EC2 bill.

gnufx 5y ago
10% is around the noise level for HPC, especially for throughput depending on scheduling. I rather doubt you couldn't do the same as free software.
- Sanguinaire 5y ago
  You are correct, nothing Intel provides in their Python distro cannot be obtained elsewhere - this is just a nice wrapper.
- pjmlp 5y ago
  Yet plenty of HPC installations rather use IBM's xl or NVidia's PGI compiler suits. So they definitly don't agree you can do the same with free software.
  - gnufx 5y ago
    They may or may not disagree on the basis of measured performance in different cases. (The US labs are investing heavily in free software LLVM.) Often not with ifort, at least. However, I was talking about the Intel stuff, and partly from experience with R versus the Microsoft version. However, I recently ran the Polyhedron Fortran benchmarks with the compilers to hand (~2020 vintage). XL (on POWER) was the only one that gave a significantly better bottom line; obviously IBM know how to compile Fortran well by now (or by Fortran-H). As far as I remember, that was essentially due to the treatment (vectorization) of maths intrinsics, which probably aren't so dominant in typical HPC code. One bad case -- fmod inlining -- has since been fixed in GCC. Without GCC's unfortunate longstanding failure to vectorize sincos (or equivalent), gfortran should have beaten ifort significantly in the bottom line, and at least got close to XLF. PGI was distinctly worse than GCC, but may do better at OpenACC/GPU offload, for instance. XL may win on OpenMP, since some of the current standard was for Sierra's needs. I should find time for the NAS benchmarks.

gnufx 5y ago

10% is around the noise level for HPC, especially for throughput depending on scheduling. I rather doubt you couldn't do the same as free software.

Sanguinaire 5y ago
You are correct, nothing Intel provides in their Python distro cannot be obtained elsewhere - this is just a nice wrapper.
pjmlp 5y ago
Yet plenty of HPC installations rather use IBM's xl or NVidia's PGI compiler suits. So they definitly don't agree you can do the same with free software.
- gnufx 5y ago
  They may or may not disagree on the basis of measured performance in different cases. (The US labs are investing heavily in free software LLVM.) Often not with ifort, at least. However, I was talking about the Intel stuff, and partly from experience with R versus the Microsoft version. However, I recently ran the Polyhedron Fortran benchmarks with the compilers to hand (~2020 vintage). XL (on POWER) was the only one that gave a significantly better bottom line; obviously IBM know how to compile Fortran well by now (or by Fortran-H). As far as I remember, that was essentially due to the treatment (vectorization) of maths intrinsics, which probably aren't so dominant in typical HPC code. One bad case -- fmod inlining -- has since been fixed in GCC. Without GCC's unfortunate longstanding failure to vectorize sincos (or equivalent), gfortran should have beaten ifort significantly in the bottom line, and at least got close to XLF. PGI was distinctly worse than GCC, but may do better at OpenACC/GPU offload, for instance. XL may win on OpenMP, since some of the current standard was for Sierra's needs. I should find time for the NAS benchmarks.