9 ms·

> Buck2 is an extensible and performant build system written in Rust I really appreciate tooling that is written in Rust or Go that produce single binaries wit

> Buck2 is an extensible and performant build system written in Rust I really appreciate tooling that is written in Rust or Go that produce single binaries with minimal runtime dependencies. Getting tooling written in for example Python to run reliably can be an exercise in frustration due to runtime environmental dependencies.

TikolaNesla 4y ago
Yes, just what I thought when I installed the Shopify CLI (https://github.com/Shopify/cli https://github.com/Shopify/cli) a few days ago because they force you to install Ruby and Node
rektide 4y ago
Personally it seems like a huge waste of memory to me. It's the electron of the backend. It's absolutely done for convenience & simplicity, with good cause after the pain we have endured. But every single binary bringing the whole universe of libraries with it offends. Why have an OS at all if every program is just going to package everything it needs? It feels like we cheapened out. Rather than get good & figure out how to manage things well, rather than driver harder, we're bunting the problem. It sucks & it's lo-fi & a huge waste of resources.
- thangngoc89 4y ago
  > with minimal runtime dependencies You’re probably thinking of static binary. I believe that OP is comparing a single binary vs installing the whole toolchain of Python/Ruby/Node and fetching the dependencies over the wire.
  - crabbone 4y ago
    If it's not a statically linked binary, then the problem is just as bad as it is with Python dependencies: instead, now you need to find the shared libraries that it linked with.
- crabbone 4y ago
  Absolutely. As soon as it started to seem like even couple hundreds of JARs won't put a significant strain on the filesystem having to house them, the typical deployment switched to Docker images, and, on top of the hundred of JARs started to bundle in the whole OS userspace. Which also, conveniently, makes memory explode because shared libraries are no longer shared. This would definitely sound like a conspiracy theory, but I'm quite sure that hardware vendors see this technological development as, at least, a fortunate turn of events...
- maccard 4y ago
  We've had decades to figure this out, and none of the "solutions" work. Meanwhile, the CRT for visual studio id 15MB. If every app I installed grew by 15MB I don't think I would notice.
  - cozzyd 4y ago
    Imagine if every QT program included all of the QT shared libraries.
    - maccard 4y ago
      On windows they do.
- bogwog 4y ago
  I don't think that matters so much. For building a system, you definitely need dynamic linking, but end user apps being as self contained as possible is good for developers, users, and system maintainers (who don't have to worry about breaking apps). As long as it doesn't get out of hand, a few dozen MBs even is a small price to pay IMO for the compatibility benefits. As a long time Linux desktop user, I appreciate any efforts to improve compatibility between distros. Since Linux isn't actually an operating system, successfully running software built for Ubuntu on a Fedora box, for example, is entirely based on luck.
  - rektide 4y ago
    There's also the issue that if a library has a vulnerability, you are now reliant on every static binary updating with the fix & releasing a new version. Where-as with the conventional dynamic library world one would just update openssl or whomever & keep going. Or if someone wanted to shim in an alternate but compatible library, one could. I personally never saw the binary compatibility issue as very big, and generally felt like there was a while where folks were getting good at packaging apps for each OS, making extra repos, that we've lost. So it seems predominantly to me like downsides, that we sell ourselves on, based off of outsized/overrepresented fear & negativity.
    - preseinger 4y ago
      the optimization you describe here is not valuable enough to offset the value provided by statically linked applications the computational model of a fleet of long-lived servers, which receive host/OS updates at one cadence, and serve applications that are deployed at a different cadence, is at this point a niche use case, basically anachronistic, and going away applications are the things that matter, they provide the value, the OS and even shared libraries are really optimizations, details, that don't really make sense any more the unit of maintenance is not a host, or a specific library, it's an application vulnerabilities affect applications, if there is a vulnerability in some library that's used by a bunch of my applications then it's expected that i will need to re-deploy updated versions of those applications, this is not difficult, i am re-deploying updated versions of my applications all the time, because that is my deployment model
- stu2b50 4y ago
  Sometimes things just don’t have good solutions in one space. We solved in another space, as SSD and ram manufacturers made memory exponentially cheaper and more available over the last few decades. So we make the trade off of software complexity for hardware complexity. Such is how life goes sometimes.
- preseinger 4y ago
  when someone writes a program and offers it for other people to execute, it should generally be expected to work the size of a program binary is a distant secondary concern to this main goal static compilation more or less solves this primary requirement, at the cost of an increase to binary size that is statistically zero in the context of any modern computer, outside of maybe embedded (read: niche) use cases there is no meaningful difference between a 1MB binary or a 10MB binary or a 100MB binary, disks are big and memory is cheap the optimization of dynamic linking was based on costs of computation, and a security model of system administration, which are no longer valid there's no reason to be offended by this, just update your models of reality and move on
  - rektide 4y ago
    I never had a problem before. The people saying we need this for convenience felt detached & wrong from the start. It's popular to be cynical & conservative, to disbelieve. That has won the day. It doesn't do anything to convince me it was a good choice or actually helpful, that we were right to just give up.
    - preseinger 4y ago
      "wrong" or "a good choice" or "actually helpful" are not objective measures, they are judged by a specific observer, what's wrong for you can be right for someone else i won't try to refute your personal experience, but i'll observe it's relevant in this discussion only to the extent that your individual context is representative of consumers of this kind of software in general that static linking provides a more reliable end-user experience vs. dynamic linking is hopefully not controversial, the point about security updates is true and important but very infrequent compared to new installations
  - cozzyd 4y ago
    Wait until you use a single board computer with a 4GB emmc OS disk. And don't forget about bandwidth...
    - preseinger 4y ago
- howinteresting 4y ago
  Dynamic linking is an artifact of C, not some sort of universal programming truth.
  - wahern 4y ago
    Dynamic linking originated with Multics (https://en.wikipedia.org/wiki/Multics https://en.wikipedia.org/wiki/Multics) and MTS (https://en.wikipedia.org/wiki/MTS_system_architecture https://en.wikipedia.org/wiki/MTS_system_architecture), years before C even existed. Unix didn't get dynamic linking until the 1980s (https://www.cs.cornell.edu/courses/cs414/2001FA/sharedlib.pdf https://www.cs.cornell.edu/courses/cs414/2001FA/sharedlib.pd...). The impetus for dynamic linking on Multics and MTS was the ability to upgrade libraries without having to recompile software, and to reuse code not originally designed or intended for (e.g. different compilers or languages), let alone compiled with, the primary program. Both of these reasons still pertain, notwithstanding that some alternatives are more viable (e.g. open source code means less reliance on binary distribution).
    - preseinger 4y ago
      neither of those reasons still pertain, really "the primary program" is the atomic unit of change, it is expected that each program behaves in a way that is independent of whatever other files may exist on a host system
crabbone 4y ago
Your problem is that Python sucks, especially it's dependency management. It sucks not because it ought to suck, but because of the incompetence of PyPA (the people responsible for packaging). There are multiple problems with Python packaging which ought not exist, but are there and make lives of Python users worse: * Python doesn't have a package manager. pip can install packages, but installing packages iteratively will break dependencies of packages installed in previous iterations. So, if you call pip install twice or more, you are likely to end up with a broken system. * Python cannot deal with different programs wanting different versions of the same dependency. * Python version iterates very fast. It's even worse for most of the Python packages. To stand still you need to update all the time, because everything goes stale very fast. In addition, this creates too many versions of packages for dependency solvers to process leading to insanely long installation times, which, in turn, prompts the package maintainers to specify very precise version requirements (to reduce the time one has to wait for the solver to figure out what to install), but this, in turn, creates a situation where there are lots of allegedly incompatible packages. * Python package maintainers have too many elements in support matrix. This leads to quick abandonment of old versions, fragmented support across platforms and versions. * Python packages are low quality. Many Python programmers don't understand what needs to go into a package, they either put too little or too much or just the wrong stuff altogether. All of the above could've been solved by better moderation of community-generated packages, stricter rules on package submission process, longer version release cycles, formalizing package requirements across different platforms, creating tools s.a. package manager to aid in this process... PyPA simply doesn't care. That's why it sucks.
- zdw 4y ago
  s/Python/NodeJS/ and everything in this statement is multiplied by 10x
  - IshKebab 4y ago
    Some of it is also true for Node (e.g. poor package quality), but I think it would be hard to argue that the actual package management of Node is anywhere near as bad as Python. Node basically works fine. You get a huge node_modules folder, sure. But it works. Python is a complete mess.

TikolaNesla 4y ago

Yes, just what I thought when I installed the Shopify CLI (https://github.com/Shopify/cli https://github.com/Shopify/cli) a few days ago because they force you to install Ruby and Node

rektide 4y ago

Personally it seems like a huge waste of memory to me. It's the electron of the backend. It's absolutely done for convenience & simplicity, with good cause after the pain we have endured. But every single binary bringing the whole universe of libraries with it offends. Why have an OS at all if every program is just going to package everything it needs? It feels like we cheapened out. Rather than get good & figure out how to manage things well, rather than driver harder, we're bunting the problem. It sucks & it's lo-fi & a huge waste of resources.

thangngoc89 4y ago
> with minimal runtime dependencies You’re probably thinking of static binary. I believe that OP is comparing a single binary vs installing the whole toolchain of Python/Ruby/Node and fetching the dependencies over the wire.
- crabbone 4y ago
  If it's not a statically linked binary, then the problem is just as bad as it is with Python dependencies: instead, now you need to find the shared libraries that it linked with.
crabbone 4y ago
Absolutely. As soon as it started to seem like even couple hundreds of JARs won't put a significant strain on the filesystem having to house them, the typical deployment switched to Docker images, and, on top of the hundred of JARs started to bundle in the whole OS userspace. Which also, conveniently, makes memory explode because shared libraries are no longer shared. This would definitely sound like a conspiracy theory, but I'm quite sure that hardware vendors see this technological development as, at least, a fortunate turn of events...
maccard 4y ago
We've had decades to figure this out, and none of the "solutions" work. Meanwhile, the CRT for visual studio id 15MB. If every app I installed grew by 15MB I don't think I would notice.
- cozzyd 4y ago
  Imagine if every QT program included all of the QT shared libraries.
  - maccard 4y ago
    On windows they do.
bogwog 4y ago
I don't think that matters so much. For building a system, you definitely need dynamic linking, but end user apps being as self contained as possible is good for developers, users, and system maintainers (who don't have to worry about breaking apps). As long as it doesn't get out of hand, a few dozen MBs even is a small price to pay IMO for the compatibility benefits. As a long time Linux desktop user, I appreciate any efforts to improve compatibility between distros. Since Linux isn't actually an operating system, successfully running software built for Ubuntu on a Fedora box, for example, is entirely based on luck.
- rektide 4y ago
  There's also the issue that if a library has a vulnerability, you are now reliant on every static binary updating with the fix & releasing a new version. Where-as with the conventional dynamic library world one would just update openssl or whomever & keep going. Or if someone wanted to shim in an alternate but compatible library, one could. I personally never saw the binary compatibility issue as very big, and generally felt like there was a while where folks were getting good at packaging apps for each OS, making extra repos, that we've lost. So it seems predominantly to me like downsides, that we sell ourselves on, based off of outsized/overrepresented fear & negativity.
  - preseinger 4y ago
    the optimization you describe here is not valuable enough to offset the value provided by statically linked applications the computational model of a fleet of long-lived servers, which receive host/OS updates at one cadence, and serve applications that are deployed at a different cadence, is at this point a niche use case, basically anachronistic, and going away applications are the things that matter, they provide the value, the OS and even shared libraries are really optimizations, details, that don't really make sense any more the unit of maintenance is not a host, or a specific library, it's an application vulnerabilities affect applications, if there is a vulnerability in some library that's used by a bunch of my applications then it's expected that i will need to re-deploy updated versions of those applications, this is not difficult, i am re-deploying updated versions of my applications all the time, because that is my deployment model
stu2b50 4y ago
Sometimes things just don’t have good solutions in one space. We solved in another space, as SSD and ram manufacturers made memory exponentially cheaper and more available over the last few decades. So we make the trade off of software complexity for hardware complexity. Such is how life goes sometimes.
preseinger 4y ago
when someone writes a program and offers it for other people to execute, it should generally be expected to work the size of a program binary is a distant secondary concern to this main goal static compilation more or less solves this primary requirement, at the cost of an increase to binary size that is statistically zero in the context of any modern computer, outside of maybe embedded (read: niche) use cases there is no meaningful difference between a 1MB binary or a 10MB binary or a 100MB binary, disks are big and memory is cheap the optimization of dynamic linking was based on costs of computation, and a security model of system administration, which are no longer valid there's no reason to be offended by this, just update your models of reality and move on
- rektide 4y ago
  I never had a problem before. The people saying we need this for convenience felt detached & wrong from the start. It's popular to be cynical & conservative, to disbelieve. That has won the day. It doesn't do anything to convince me it was a good choice or actually helpful, that we were right to just give up.
  - preseinger 4y ago
    "wrong" or "a good choice" or "actually helpful" are not objective measures, they are judged by a specific observer, what's wrong for you can be right for someone else i won't try to refute your personal experience, but i'll observe it's relevant in this discussion only to the extent that your individual context is representative of consumers of this kind of software in general that static linking provides a more reliable end-user experience vs. dynamic linking is hopefully not controversial, the point about security updates is true and important but very infrequent compared to new installations
- cozzyd 4y ago
  Wait until you use a single board computer with a 4GB emmc OS disk. And don't forget about bandwidth...
  - preseinger 4y ago
howinteresting 4y ago
Dynamic linking is an artifact of C, not some sort of universal programming truth.
- wahern 4y ago
  Dynamic linking originated with Multics (https://en.wikipedia.org/wiki/Multics https://en.wikipedia.org/wiki/Multics) and MTS (https://en.wikipedia.org/wiki/MTS_system_architecture https://en.wikipedia.org/wiki/MTS_system_architecture), years before C even existed. Unix didn't get dynamic linking until the 1980s (https://www.cs.cornell.edu/courses/cs414/2001FA/sharedlib.pdf https://www.cs.cornell.edu/courses/cs414/2001FA/sharedlib.pd...). The impetus for dynamic linking on Multics and MTS was the ability to upgrade libraries without having to recompile software, and to reuse code not originally designed or intended for (e.g. different compilers or languages), let alone compiled with, the primary program. Both of these reasons still pertain, notwithstanding that some alternatives are more viable (e.g. open source code means less reliance on binary distribution).
  - preseinger 4y ago
    neither of those reasons still pertain, really "the primary program" is the atomic unit of change, it is expected that each program behaves in a way that is independent of whatever other files may exist on a host system

crabbone 4y ago

Your problem is that Python sucks, especially it's dependency management. It sucks not because it ought to suck, but because of the incompetence of PyPA (the people responsible for packaging). There are multiple problems with Python packaging which ought not exist, but are there and make lives of Python users worse: * Python doesn't have a package manager. pip can install packages, but installing packages iteratively will break dependencies of packages installed in previous iterations. So, if you call pip install twice or more, you are likely to end up with a broken system. * Python cannot deal with different programs wanting different versions of the same dependency. * Python version iterates very fast. It's even worse for most of the Python packages. To stand still you need to update all the time, because everything goes stale very fast. In addition, this creates too many versions of packages for dependency solvers to process leading to insanely long installation times, which, in turn, prompts the package maintainers to specify very precise version requirements (to reduce the time one has to wait for the solver to figure out what to install), but this, in turn, creates a situation where there are lots of allegedly incompatible packages. * Python package maintainers have too many elements in support matrix. This leads to quick abandonment of old versions, fragmented support across platforms and versions. * Python packages are low quality. Many Python programmers don't understand what needs to go into a package, they either put too little or too much or just the wrong stuff altogether. All of the above could've been solved by better moderation of community-generated packages, stricter rules on package submission process, longer version release cycles, formalizing package requirements across different platforms, creating tools s.a. package manager to aid in this process... PyPA simply doesn't care. That's why it sucks.

zdw 4y ago
s/Python/NodeJS/ and everything in this statement is multiplied by 10x
- IshKebab 4y ago
  Some of it is also true for Node (e.g. poor package quality), but I think it would be hard to argue that the actual package management of Node is anywhere near as bad as Python. Node basically works fine. You get a huge node_modules folder, sure. But it works. Python is a complete mess.