9 ms·

Is there a more verbose explanation about why they’re moving away from compiling packages from source? All the explanations I’ve seen make it sound like that’s

by ihuman 8y ago

Is there a more verbose explanation about why they’re moving away from compiling packages from source? All the explanations I’ve seen make it sound like that’s the obvious direction it should go, but it’s not obvious to me.

_em6m 8y ago
Maybe ask the question the other way around. Why should they keep compiling everything from source? What are the benefits?
- black-tea 8y ago
  You get native binaries, can choose optional features and dependencies and you can easily apply your own choice of patches. That's my view as a gentoo user anyway; I don't know how much of that homebrew actually supports.
  - saagarjha 8y ago
    Homebrew used to support optional features, but is now moving away from this to become a binary package manager.
  - ninkendo 8y ago
    What does “native binary” mean?
    - michaericalribo 8y ago
      Compiled locally, so (posssibly) optimized for better performance than a generic binary compiled for your architecture.
    - black-tea 8y ago
      Compilers like GCC can produce code specifically for your CPU such that it uses all of the special features you might have like SSE, AVX etc. When binaries are made for general use they have to be compiled with a generic feature set so might not make best use of your CPU.
      - ninkendo 8y ago
        So can a cross compiler, no?
        black-tea 8y ago
        If you know the exact flags for your CPU then you can compile for it on any other CPU using a cross compiler line GCC, yes. But that doesn't help because in practice you'll just find one binary per architecture, not per CPU.
  - h1d 8y ago
    At least it should no longer take a night to compile some GUI app like back in the days.
- bobwaycott 8y ago
  Well, for one, as a longtime brew user, changing to binary packages alters the trust model.
  - devzer0 8y ago
    Binaries are built on CI infra with logs that are publicly available and with all the source code doing the builds publicly available. Effectively the same process as local, from source builds. The bottle DSL ensures that the SHA binaries are not tampered with between when they were built (and the source code sha checked) and when they are delivered. I know this is still a different trust model, but, IMO, not too different. I know reproducible builds are on the radar for Homebrew in the future, and that should ease some of your concerns too. For a good thread on Homebrew security checkout https://twitter.com/c_pellegrino/status/1093195802871246848 https://twitter.com/c_pellegrino/status/1093195802871246848
    - bobwaycott 8y ago
      Hey! Thanks for the reply. I’ll definitely check that link out. It’s really nice to know more about the process behind delivering binaries.
- eslaught 8y ago
  Compare to e.g. conda, where you get binary packages and no option to build from source at all. This basically means you get support for x86 and very limited support for PPC. Need ARM? Nope, no can do. Or any other slightly exotic architecture. It also means, though this is only a problem with third party channels, that it's possible to get into a situation where you can't even build from source at all. I was working with one of our custom packages, where for the life of me I could not figure out how to get the build to work. Turns out it had been hardcoded to run on a very specific machine, and assumed access to a very specific network file system, and I wasted about a week of my life trying to untangle those assumptions before giving up. If that software had been designed from the beginning with the idea that it ought to be possible to build from source from anywhere, we'd presumably be in a different situation now.
  - kalefranz 8y ago
    Conda has support for armv7 via berryconda.
    - eslaught 8y ago
      Interesting, good to know. I'm guessing third party channel support is probably close to nil though, so for my use case it probably wouldn't get me very far.
Athas 8y ago
As a user, you can always ask for the package to be compiled from source. This is also what will happen if you ask for non-standard options. The advantage of using binary packages is time: I have a Homebrew formula that takes 40 minutes to build from source, but maybe ten seconds to install from a binary package.
- saagarjha 8y ago
  Options have been deprecated, and I haven’t been able to find anything that replaces them that the maintainers are willing to discuss.
  - mikemcquaid 8y ago
    Options have been removed in the main Homebrew/homebrew-core tap but are not deprecated for user taps.
saagarjha 8y ago
I got into a minor argument with the maintainers about it, and the response I got was that they thought it was an unreasonable support burden: https://discourse.brew.sh/t/maintaining-a-tap-to-provide-options-for-a-formula/3871/ https://discourse.brew.sh/t/maintaining-a-tap-to-provide-opt.... Not something I could get them to provide more details about, however. Not really a fan of the response I got :/
- mikemcquaid 8y ago
  The actual response: > We’re not willing to ship things we don’t support, sorry. I do not consider it acceptable to ship software that we are unable to test and we know will break. Even if I did consider that acceptable I’d be unwilling to accept the additional support burden that the maintainers have to take on from people submitting issues when they are clearly told not to (something that still happens often). Clarification beyond that: fixing avoidable user issues is not what most of the maintainers want to spend most of their (volunteer, mostly evening and weekend) time on. Even immediately closing issues as WONTFIX takes more time to deal with than not getting an issue at all. You may disagree with this (clearly some people do) but ultimately you cannot expect a volunteer run project to support more than they are willing to do. If we had people stepping up and saying "I will deal with all the support requests for this!" that'd be different but mostly we have people who want us to do the work for them.
mikemcquaid 8y ago
In short: - it's faster for end-users - it's more reliable (which both makes the user experience better for users and reduces the support burden for our volunteer maintainers)
_rs 8y ago
What happens to all of my packages I have installed with options? When I upgrade them next they will suddenly be stripped down? ffmpeg is a huge one, as the default set of options excludes many things that most people using ffmpeg need by default, but homebrew can't ship as a binary for legal reasons.
- saagarjha 8y ago
  Yes, they get reinstalled without the options.

by ihuman 8y ago

_em6m 8y ago

Maybe ask the question the other way around. Why should they keep compiling everything from source? What are the benefits?

black-tea 8y ago
You get native binaries, can choose optional features and dependencies and you can easily apply your own choice of patches. That's my view as a gentoo user anyway; I don't know how much of that homebrew actually supports.
- saagarjha 8y ago
  Homebrew used to support optional features, but is now moving away from this to become a binary package manager.
- ninkendo 8y ago
  What does “native binary” mean?
  - michaericalribo 8y ago
    Compiled locally, so (posssibly) optimized for better performance than a generic binary compiled for your architecture.
  - black-tea 8y ago
    Compilers like GCC can produce code specifically for your CPU such that it uses all of the special features you might have like SSE, AVX etc. When binaries are made for general use they have to be compiled with a generic feature set so might not make best use of your CPU.
    - ninkendo 8y ago
      So can a cross compiler, no?
      - black-tea 8y ago
        If you know the exact flags for your CPU then you can compile for it on any other CPU using a cross compiler line GCC, yes. But that doesn't help because in practice you'll just find one binary per architecture, not per CPU.
- h1d 8y ago
  At least it should no longer take a night to compile some GUI app like back in the days.
bobwaycott 8y ago
Well, for one, as a longtime brew user, changing to binary packages alters the trust model.
- devzer0 8y ago
  Binaries are built on CI infra with logs that are publicly available and with all the source code doing the builds publicly available. Effectively the same process as local, from source builds. The bottle DSL ensures that the SHA binaries are not tampered with between when they were built (and the source code sha checked) and when they are delivered. I know this is still a different trust model, but, IMO, not too different. I know reproducible builds are on the radar for Homebrew in the future, and that should ease some of your concerns too. For a good thread on Homebrew security checkout https://twitter.com/c_pellegrino/status/1093195802871246848 https://twitter.com/c_pellegrino/status/1093195802871246848
  - bobwaycott 8y ago
    Hey! Thanks for the reply. I’ll definitely check that link out. It’s really nice to know more about the process behind delivering binaries.
eslaught 8y ago
Compare to e.g. conda, where you get binary packages and no option to build from source at all. This basically means you get support for x86 and very limited support for PPC. Need ARM? Nope, no can do. Or any other slightly exotic architecture. It also means, though this is only a problem with third party channels, that it's possible to get into a situation where you can't even build from source at all. I was working with one of our custom packages, where for the life of me I could not figure out how to get the build to work. Turns out it had been hardcoded to run on a very specific machine, and assumed access to a very specific network file system, and I wasted about a week of my life trying to untangle those assumptions before giving up. If that software had been designed from the beginning with the idea that it ought to be possible to build from source from anywhere, we'd presumably be in a different situation now.
- kalefranz 8y ago
  Conda has support for armv7 via berryconda.
  - eslaught 8y ago
    Interesting, good to know. I'm guessing third party channel support is probably close to nil though, so for my use case it probably wouldn't get me very far.

Athas 8y ago

As a user, you can always ask for the package to be compiled from source. This is also what will happen if you ask for non-standard options. The advantage of using binary packages is time: I have a Homebrew formula that takes 40 minutes to build from source, but maybe ten seconds to install from a binary package.

saagarjha 8y ago
Options have been deprecated, and I haven’t been able to find anything that replaces them that the maintainers are willing to discuss.
- mikemcquaid 8y ago
  Options have been removed in the main Homebrew/homebrew-core tap but are not deprecated for user taps.

saagarjha 8y ago

I got into a minor argument with the maintainers about it, and the response I got was that they thought it was an unreasonable support burden: https://discourse.brew.sh/t/maintaining-a-tap-to-provide-options-for-a-formula/3871/ https://discourse.brew.sh/t/maintaining-a-tap-to-provide-opt.... Not something I could get them to provide more details about, however. Not really a fan of the response I got :/

mikemcquaid 8y ago
The actual response: > We’re not willing to ship things we don’t support, sorry. I do not consider it acceptable to ship software that we are unable to test and we know will break. Even if I did consider that acceptable I’d be unwilling to accept the additional support burden that the maintainers have to take on from people submitting issues when they are clearly told not to (something that still happens often). Clarification beyond that: fixing avoidable user issues is not what most of the maintainers want to spend most of their (volunteer, mostly evening and weekend) time on. Even immediately closing issues as WONTFIX takes more time to deal with than not getting an issue at all. You may disagree with this (clearly some people do) but ultimately you cannot expect a volunteer run project to support more than they are willing to do. If we had people stepping up and saying "I will deal with all the support requests for this!" that'd be different but mostly we have people who want us to do the work for them.

mikemcquaid 8y ago

In short: - it's faster for end-users - it's more reliable (which both makes the user experience better for users and reduces the support burden for our volunteer maintainers)

_rs 8y ago

What happens to all of my packages I have installed with options? When I upgrade them next they will suddenly be stripped down? ffmpeg is a huge one, as the default set of options excludes many things that most people using ffmpeg need by default, but homebrew can't ship as a binary for legal reasons.

saagarjha 8y ago
Yes, they get reinstalled without the options.