4 ms·

Why are they making their own distro and not putting code back into mainline if it's useful? Do they have some particular IP that makes this impossibe?

by _joel 5y ago

SkipperCat 5y ago
I think there is a pretty big base of people who do big data work using Numpy and Pandas (Fintech, etc). They want to squeeze every bit of computing power out of the specific Intel chipset, GPUs, etc and Intel's distro really helps them out. A 10% speed improvement on 1000's of jobs could in theory save you a nice chunk of time. This becomes very important in the financial market where you need batch jobs to be finished before markets open, or you just want to save 10% on your EC2 bill.
- gnufx 5y ago
  10% is around the noise level for HPC, especially for throughput depending on scheduling. I rather doubt you couldn't do the same as free software.
  - Sanguinaire 5y ago
    You are correct, nothing Intel provides in their Python distro cannot be obtained elsewhere - this is just a nice wrapper.
  - pjmlp 5y ago
    Yet plenty of HPC installations rather use IBM's xl or NVidia's PGI compiler suits. So they definitly don't agree you can do the same with free software.
    - gnufx 5y ago
      They may or may not disagree on the basis of measured performance in different cases. (The US labs are investing heavily in free software LLVM.) Often not with ifort, at least. However, I was talking about the Intel stuff, and partly from experience with R versus the Microsoft version. However, I recently ran the Polyhedron Fortran benchmarks with the compilers to hand (~2020 vintage). XL (on POWER) was the only one that gave a significantly better bottom line; obviously IBM know how to compile Fortran well by now (or by Fortran-H). As far as I remember, that was essentially due to the treatment (vectorization) of maths intrinsics, which probably aren't so dominant in typical HPC code. One bad case -- fmod inlining -- has since been fixed in GCC. Without GCC's unfortunate longstanding failure to vectorize sincos (or equivalent), gfortran should have beaten ifort significantly in the bottom line, and at least got close to XLF. PGI was distinctly worse than GCC, but may do better at OpenACC/GPU offload, for instance. XL may win on OpenMP, since some of the current standard was for Sierra's needs. I should find time for the NAS benchmarks.
LeifCarrotson 5y ago
Here's the list of CPUs which incorporate the AVX2 instructions that enable some of these optimizations: https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPUs_with_AVX2 https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPU... You could write your distro to check for flags that will tell it whether or not you have these using flags from /proc/cpuinfo. Or you could check whether it's in the Intel half of the list or the AMD half of the list. Or you could write your own distro that only runs on the first half of the list. I get that Intel's contributions aren't purely altruistic. There are likely to be subtle tuning problems that require slight changes to optimize on different platforms, and they can't really be expected to do free work for AMD. But it looks to me like they're being unecessarily anticompetitive.
- falcor84 5y ago
  >being unecessarily anticompetitive Isn't setting up barriers to entry generally considered to be a part of healthy competition? I'd hazard to say that as long as a company is playing within the boundaries of what's allowed, there's nothing they could do that's anticompetitive; at the most, you could accuse them of being somewhat unsportsmanlike.
  - DasIch 5y ago
    Competition is desirable because it aligns with society’s goals of innovation and progress which also imply increased productivity and lower prices. Artificial barriers to entry are contrary to that and if they’re not illegal they should be.
    - falcor84 5y ago
      Where do you define this line of barriers becoming 'artificial'?
      - LeifCarrotson 5y ago
        It's artificial when the vendor expends additional time, effort, or funds to construct a barrier, or chooses an equally-priced non-interoperable design that a rational, informed consumer with a choice would reject. If you're expending great effort to write custom DRM or to reinvent open industry standards that you could have installed cheaply, that's artificial. I fully admit that there are natural barriers that occur at times. I don't think that you should be expected to reverse-engineer your competitor's products and bend over backwards to make them work better. Here, for a concrete example, Intel had a clear choice to test whether a processor supported a feature by checking a feature flag - It's in the name, they're literally implemented for that exact purpose - or they could expend extra effort in building their own feature flag database by checking manufacturer and part number. They could have either expended extra effort to launch and distribute their own entire custom Python distribution, or submitted pull requests to the existing main distribution. For another example, Apple could have used industry-standard Phillips or Torx screws in their hardware: Manufacturers had lines to produce them, distributors had inventory of the fasteners, users had tools to turn them. Instead, they went to great expense to build their own incompatible tri-lobe screws, requiring probably millions of dollars in investment in custom tooling and production lines, all for the sake of creating an artificial barrier.

Why are they making their own distro and not putting code back into mainline if it's useful? Do they have some particular IP that makes this impossibe?

by _joel 5y ago

Why are they making their own distro and not putting code back into mainline if it's useful? Do they have some particular IP that makes this impossibe?

SkipperCat 5y ago

I think there is a pretty big base of people who do big data work using Numpy and Pandas (Fintech, etc). They want to squeeze every bit of computing power out of the specific Intel chipset, GPUs, etc and Intel's distro really helps them out. A 10% speed improvement on 1000's of jobs could in theory save you a nice chunk of time. This becomes very important in the financial market where you need batch jobs to be finished before markets open, or you just want to save 10% on your EC2 bill.

gnufx 5y ago
10% is around the noise level for HPC, especially for throughput depending on scheduling. I rather doubt you couldn't do the same as free software.
- Sanguinaire 5y ago
  You are correct, nothing Intel provides in their Python distro cannot be obtained elsewhere - this is just a nice wrapper.
- pjmlp 5y ago
  Yet plenty of HPC installations rather use IBM's xl or NVidia's PGI compiler suits. So they definitly don't agree you can do the same with free software.
  - gnufx 5y ago
    They may or may not disagree on the basis of measured performance in different cases. (The US labs are investing heavily in free software LLVM.) Often not with ifort, at least. However, I was talking about the Intel stuff, and partly from experience with R versus the Microsoft version. However, I recently ran the Polyhedron Fortran benchmarks with the compilers to hand (~2020 vintage). XL (on POWER) was the only one that gave a significantly better bottom line; obviously IBM know how to compile Fortran well by now (or by Fortran-H). As far as I remember, that was essentially due to the treatment (vectorization) of maths intrinsics, which probably aren't so dominant in typical HPC code. One bad case -- fmod inlining -- has since been fixed in GCC. Without GCC's unfortunate longstanding failure to vectorize sincos (or equivalent), gfortran should have beaten ifort significantly in the bottom line, and at least got close to XLF. PGI was distinctly worse than GCC, but may do better at OpenACC/GPU offload, for instance. XL may win on OpenMP, since some of the current standard was for Sierra's needs. I should find time for the NAS benchmarks.

LeifCarrotson 5y ago

Here's the list of CPUs which incorporate the AVX2 instructions that enable some of these optimizations: https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPUs_with_AVX2 https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPU... You could write your distro to check for flags that will tell it whether or not you have these using flags from /proc/cpuinfo. Or you could check whether it's in the Intel half of the list or the AMD half of the list. Or you could write your own distro that only runs on the first half of the list. I get that Intel's contributions aren't purely altruistic. There are likely to be subtle tuning problems that require slight changes to optimize on different platforms, and they can't really be expected to do free work for AMD. But it looks to me like they're being unecessarily anticompetitive.

falcor84 5y ago
>being unecessarily anticompetitive Isn't setting up barriers to entry generally considered to be a part of healthy competition? I'd hazard to say that as long as a company is playing within the boundaries of what's allowed, there's nothing they could do that's anticompetitive; at the most, you could accuse them of being somewhat unsportsmanlike.
- DasIch 5y ago
  Competition is desirable because it aligns with society’s goals of innovation and progress which also imply increased productivity and lower prices. Artificial barriers to entry are contrary to that and if they’re not illegal they should be.
  - falcor84 5y ago
    Where do you define this line of barriers becoming 'artificial'?
    - LeifCarrotson 5y ago
      It's artificial when the vendor expends additional time, effort, or funds to construct a barrier, or chooses an equally-priced non-interoperable design that a rational, informed consumer with a choice would reject. If you're expending great effort to write custom DRM or to reinvent open industry standards that you could have installed cheaply, that's artificial. I fully admit that there are natural barriers that occur at times. I don't think that you should be expected to reverse-engineer your competitor's products and bend over backwards to make them work better. Here, for a concrete example, Intel had a clear choice to test whether a processor supported a feature by checking a feature flag - It's in the name, they're literally implemented for that exact purpose - or they could expend extra effort in building their own feature flag database by checking manufacturer and part number. They could have either expended extra effort to launch and distribute their own entire custom Python distribution, or submitted pull requests to the existing main distribution. For another example, Apple could have used industry-standard Phillips or Torx screws in their hardware: Manufacturers had lines to produce them, distributors had inventory of the fasteners, users had tools to turn them. Instead, they went to great expense to build their own incompatible tri-lobe screws, requiring probably millions of dollars in investment in custom tooling and production lines, all for the sake of creating an artificial barrier.