5 ms·

Yeah. The part that gives me goosebumps with that proposal is: as an author of some not very popular packages, what do I do? Is it my job to package and promote

Yeah. The part that gives me goosebumps with that proposal is: as an author of some not very popular packages, what do I do? Is it my job to package and promote my packages on Debian, redhat, homebrew, and all the others? Do I need usage before they’ll pick it up? Sounds like a chicken and egg problem. And doing all that packaging work sounds like a boring, time consuming job that I don’t want. It’s also quite common for me to pull out bits of code I’ve written that I want to share as a dependency between different programs I write. I have dozens of packages in npm along these lines - and it’s quite elegant. I factor out the code, package it independently, write a readme and some tests and publish it. My other packages can then just depend on the brand new package directly. In this brave new world the author is proposing - what? I make a .deb, argue for it on some mailing lists I’m not on, wait who knows how long, eventually gain permission to depend on my own code, then realise I can’t install it on my Mac laptop and do the same dance in homebrew? Oh - there’s a bug. Better manually update all the package repositories? Sounds like a complete nightmare to me. I’m sympathetic to the problem. Supply chain attacks are way too easy. But maintaining up to date packages on every single Linux distribution, homebrew, ports, and whatever windows does sounds like a full time job. A boring full time job. And it would still not work as well for my users, because when a new feature is released in one of my upstream dependencies I can’t use it until all of those package repositories ship the new version of my upstream dependency too. I’d rather eat sand. “It works for C/C++” by making developers working in those languages pay in blood for every dependency they pull in. You can tell by looking at just how few dependencies the average C program uses. It’s like that because of how painful it is. Anything under a certain size tends to get either reinvented every time, badly (hash table libraries in C) or vendored (eg json parsers). Common libraries like rust’s rand don’t exist in C because it’s just too much effort to pull in dependencies. So everyone rolls their own crappy version instead. Ask GTA5 how well that went for them. And that’s not the developer ecosystem I want to work in. It’s frankly a terrible proposal.

galangalalgol 3y ago
C++ apps don't pull in that few dependencies. Take something modern that needs to talk over flatbuffs or protobuffs and maybe has a web ui based on emscripten? Then run ldd on it and recoil in horror. C++ dependency trees now often have a hundred or more items. Many of those are packaged with the OS, but that really shouldn't comfort you.
- pclmulqdq 3y ago
  It kind of does comfort me that they are packaged with the OS, because it's an OS problem (and is relatively major) if they are broken.
  - galangalalgol 3y ago
    It often takes months for distros as large as rhel to fix med. Or even high severity CVE. That is assuming your OS is getting updated properly. Maybe it is a container and you are doing that as part of your pipeline. But all that it means having the dependency in an rpm or deb file vs in a github repo pulled by conan or similar, is that there is someone you can sue that is supposed to notice the cve and roll out a new rpm or deb file. They aren't doing massive pipelines full of fuzzers and custom unit tests for all those packages, they are just... packaging... them. Otherwise we wouldn't be seeing things like the curl and openssl vulns. Yes you might have some recourse to bug rhel to fix it, where a crate author might not care, but that really isn't the experience I am having. If anything I end up pulling dependencies into my system to build if I can so I don't have to wait on the OS company. Between corporate IT and the OS companies delay as middle-men, my OS based dependencies are always the most aged and vulnerable of all my dependencies.
mid-kid 3y ago
When you make a new library for C/C++, you generally just install it locally at first (pkg-config and PKG_CONFIG_PATH make this trivial). It doesn't (need to) get packaged by a distribution until a program using it is packaged. You don't need to do this yourself, unless you yourself use a distribution and want it packaged there. As for the amount of dependencies, you have to wonder if it's better to have tonnes of small dependencies or whether it's better to have a few bigger ones. When I need something in a library, I don't make a new library for it, I first try contributing to it, and failing that I vendor and patch it. I feel this is less common in the brave new world of everyone just doing whatever and pushing it to a largely unmoderated repository...
- josephg 3y ago
  I suppose I could manually traverse my whole dependency tree myself and install all those packages myself. Set up cmake and pkg-config for all of them. If anyone else wants to try out my software, they’d have to manually look up all the dependencies in my github (?) and install those too. And hope the versions all line up nicely - which is a big if during development. Maybe I could script all of that dependency tracing - if only there were some standard software to do that. Guess not. If C/C++ is all you know, you’re used to how painful all of that is. But what you’re proposing is a significantly worse experience in every way compared to what I can do with cargo or npm today. None of the claimed benefits seem remotely worth the cost of wading back into C dependency management hell. And cargo and npm show just how unnecessary all that pain is in C! Just make good tooling. We can have nice things. I stopped using C not because I decided to write assembler. I stopped when better languages came along. You want me to stop using cargo? Nobody wants to go back to manual dependency management. Make something better.
  - mid-kid 3y ago
    If you're smart about it you're not gonna have a very deep or wide dependency tree for your own deps. If you need more than maybe two of your own deps, maybe you need to reconsider how you structure your libraries, or you can join multiple libraries into the same project/build system umbrella (e.g. glib/gobject or the whole of QtBase)
zozbot234 3y ago
> In this brave new world the author is proposing - what? In practice, you just vendor your code as part of the source package. It's the distro maintainers' job to factor it out as a separate dependency.
- mid-kid 3y ago
  If you do this, remember to provide an option to factor it out. Meson (mesonbuild.org) makes this completely trivial and pitfall-free, and you should use it if you can't get away with just installing your libraries locally.
  - varl 3y ago
    I went to mesonbuild.org and it doesn't match the description (some sort of betting site? I didn't stick around ...), and a search turned up: https://mesonbuild.com/ https://mesonbuild.com/
    - mid-kid 3y ago
      Thanks, I mistyped it.
no_wizard 3y ago
>I’m sympathetic to the problem. Supply chain attacks are way too easy This is the crux as to why crate.io is great: you have one official place for packages that can be continuously audited and monitored. This is the way to do security, in my opinion, because you are always able to look at incoming and outgoing packages. Now, you can argue if we are doing enough with this, and that is definitely up for debate, however, I contend that the premise of having one canonical place for package origination and hosting makes it easier to monitor for security purposes
- samtheprogram 3y ago
  That’s like saying it’s best to run everything on AWS, because there’s only one place to monitor for outages. That’s the exact opposite of how supply chain mitigation works.
  - iudqnolq 3y ago
    It's not at all like saying that. If you want to mitigate downtime in your supply chain then diversity is good. If you want to mitigate security vulnerability then it's bad. Imagine you have 100 api servers that all have database credentials. If one server goes down you have 99 working servers (ideally). If one is compromised then the attacker has your database credentials, game over.
- deleted 3y ago
  [deleted]
- josephg 3y ago
  I - and many others - don’t think this is enough to mitigate supply chain attacks. It’s too easy for a malicious developer to add cryptolocker ransomware in a point release. But I also don’t think having a package in Debian will somehow magically solve this problem either. Debian package maintainers do not - and never have - done security audits on the packages they add. It’s magical thinking to assume using packages in Debian will somehow protect you from supply chain attacks. Asking developers to keep their dependency trees small so we can minimise the risk is a losing battle. Using lots of small, popular dependencies is just too convenient. There is a solution out there that I wish got more traction. And that is, capability based security. The idea is simple: we partition programs into application code and library code. Library code can never access the filesystem or network (or other protected data) directly. And this is enforced by the compiler or language runtime. Application code can. If you want a library to interact with a file, you first open the file in application code and pass the file handle to the library. Similar wrappers can be made for directories, network sockets and so on. If I want express to listen on a specific port, I pass it that port in a capability object. Not just an integer. (Eg app.listen(net.listenPort(8000))). The amount of code that would need to be changed is tiny since well behaved libraries don’t usually need many capabilities to do their job. It is ridiculous that my 3rd party json library can invisibly access all the sensitive files on my computer. With capabilities, we take that privilege away. I have no problem trusting a random json library if it literally only has access to the json string I pass it. If we had a capability based security model, I would have no qualms about pulling in hundreds of transitive dependencies. My libraries can still only access exactly what I pass them. Unfortunately this problem requires language level buy-in. It would be very hard to retrofit rust to work this way and that’ll probably never happen. Raw pointers and inline assembly also make a bit of a mess of things too. (Though pointer provenance might help). But a man can dream… Maybe in rust’s successor.
paulddraper 3y ago
You create apt, rpm repos. And users install from there. Not saying that's a nice answer, but that is the answer.
- takluyver 3y ago
  Users install from there, so long as they're using Linux, a Debian or Red Hat based distro, and have sudo access. As you say, not such a nice answer. For certain kinds of software, this approach is getting more viable, thanks to containers. In some contexts, you can tell everyone to run your software in a container with a specific Linux distro inside. But that software is often distributed as container images directly, rather than distro packages to be used in containers.
- josephg 3y ago
  You create apt, rpm repos. And arch. And gentoo. And FreeBSD ports. And homebrew and macports. And whatever windows does. And you need that to be true, on all those systems, for all your dependencies too. And at a recent, usable version. Or, I just use cargo. Up to date version. One command install that works on every operating system. Project scoped dependencies - so there’s no need for sudo, and different projects can depend on different versions of their dependencies. And I can publish new versions of my packages with 1 command and they’re available everywhere. Just in my house I have a Linux mint workstation, a windows gaming pc, a FreeBSD server and a Mac laptop. Cargo lets me publish code that all of them can consume with 1 command. So does npm. So long as apt is worse than cargo I won’t use it for this stuff. Why does everything have to get installed system wide? Nix doesn’t need that. Why doesn’t it have a programmatic mirror of cargo so we don’t need to mirror rust projects in it by hand? Why do major versions have different package names? And why don’t packages support feature flags? Last time I looked there were about 30 vim packages with random combinations of feature flags set. Why can’t evergreen packages like nodejs or cargo stay up to date in Debian? Old nodejs is useless because it can’t run modern JavaScript. I really want apt to solve all these problems because I hate docker. If apt and friends did their job well, docker wouldn’t be necessary or popular. Cargo isn’t pretty either. But it has all the features I need to get my job done. Make apt a compelling tool and I’ll switch to it. The feature list developers want is hardly a mystery.
zajio1am 3y ago
> as an author of some not very popular packages, what do I do? Depends. Do you want your software to be used by users? For me as a user, software that is not in distribution repository almost does not exist. It is too much hassle to install, manage and uninstall it compared to installing software from Linux distribution. If i really, really need it, i will make exception, but that are individual cases.
mattpallissard 3y ago
> “It works for C/C++” by making developers working in those languages pay in blood for every dependency they pull in. I think the industry may have thrown the baby out with the bath water. Now they're is a complete aversion to RYO anything, no matter how simple. Personally, I'm waiting for the pendulum to swing back towards the middle a bit more.
- josephg 3y ago
  Eh. I love rolling my own lots of things. At some point I really want to make a little OS kernel. How hard can it be? But then there’s things like rust’s rand crate which implements about 10 different random number generators, all with different properties around speed, whether they’re cryptographically secure and so on. And fed from a variety of entropy sources to make it work everywhere from Linux to windows to wasm. Look at this documentation. It’s an absolute master class. https://rust-random.github.io/book/guide-rngs.html https://rust-random.github.io/book/guide-rngs.html Do you need this? Probably not. You can probably get by with some wrapper around ar4random or whatever the recommended entropy source is today on your particular operating system. But I’ll be honest, until the day I decide to care about how rngs work, I’m going to just import the rand package. And serde & friends for json support. Criterion for benchmarking. And so on. Cargo is full of great libraries that are much higher quality than anything I’d ever throw together in a side project. For most of this stuff, you’d be mad to roll your own. This thread has reminded me that lots of people don’t know how nice it is to use a package manager like cargo. Here’s my favorite crate browsing website. You can see the most popular packages in every category and search. Look how much great stuff is in here. It’s humbling. https://lib.rs/ https://lib.rs/
IAmNotACellist 3y ago
>Yeah. The part that gives me goosebumps with that proposal is: as an author of some not very popular packages, what do I do? Is it my job to package and promote my packages on Debian, redhat, homebrew, and all the others? Do I need usage before they’ll pick it up? Sounds like a chicken and egg problem. And doing all that packaging work sounds like a boring, time consuming job that I don’t want. Github premium (or similar) will have an army of AI builder bots that spin up VMs and modify your codebase until it works, then send you PRs to approve. People would pay a ton for that service. All your projects, automatically maintained and packaged.

galangalalgol 3y ago

C++ apps don't pull in that few dependencies. Take something modern that needs to talk over flatbuffs or protobuffs and maybe has a web ui based on emscripten? Then run ldd on it and recoil in horror. C++ dependency trees now often have a hundred or more items. Many of those are packaged with the OS, but that really shouldn't comfort you.

pclmulqdq 3y ago
It kind of does comfort me that they are packaged with the OS, because it's an OS problem (and is relatively major) if they are broken.
- galangalalgol 3y ago
  It often takes months for distros as large as rhel to fix med. Or even high severity CVE. That is assuming your OS is getting updated properly. Maybe it is a container and you are doing that as part of your pipeline. But all that it means having the dependency in an rpm or deb file vs in a github repo pulled by conan or similar, is that there is someone you can sue that is supposed to notice the cve and roll out a new rpm or deb file. They aren't doing massive pipelines full of fuzzers and custom unit tests for all those packages, they are just... packaging... them. Otherwise we wouldn't be seeing things like the curl and openssl vulns. Yes you might have some recourse to bug rhel to fix it, where a crate author might not care, but that really isn't the experience I am having. If anything I end up pulling dependencies into my system to build if I can so I don't have to wait on the OS company. Between corporate IT and the OS companies delay as middle-men, my OS based dependencies are always the most aged and vulnerable of all my dependencies.

mid-kid 3y ago

When you make a new library for C/C++, you generally just install it locally at first (pkg-config and PKG_CONFIG_PATH make this trivial). It doesn't (need to) get packaged by a distribution until a program using it is packaged. You don't need to do this yourself, unless you yourself use a distribution and want it packaged there. As for the amount of dependencies, you have to wonder if it's better to have tonnes of small dependencies or whether it's better to have a few bigger ones. When I need something in a library, I don't make a new library for it, I first try contributing to it, and failing that I vendor and patch it. I feel this is less common in the brave new world of everyone just doing whatever and pushing it to a largely unmoderated repository...

josephg 3y ago
I suppose I could manually traverse my whole dependency tree myself and install all those packages myself. Set up cmake and pkg-config for all of them. If anyone else wants to try out my software, they’d have to manually look up all the dependencies in my github (?) and install those too. And hope the versions all line up nicely - which is a big if during development. Maybe I could script all of that dependency tracing - if only there were some standard software to do that. Guess not. If C/C++ is all you know, you’re used to how painful all of that is. But what you’re proposing is a significantly worse experience in every way compared to what I can do with cargo or npm today. None of the claimed benefits seem remotely worth the cost of wading back into C dependency management hell. And cargo and npm show just how unnecessary all that pain is in C! Just make good tooling. We can have nice things. I stopped using C not because I decided to write assembler. I stopped when better languages came along. You want me to stop using cargo? Nobody wants to go back to manual dependency management. Make something better.
- mid-kid 3y ago
  If you're smart about it you're not gonna have a very deep or wide dependency tree for your own deps. If you need more than maybe two of your own deps, maybe you need to reconsider how you structure your libraries, or you can join multiple libraries into the same project/build system umbrella (e.g. glib/gobject or the whole of QtBase)

zozbot234 3y ago

> In this brave new world the author is proposing - what? In practice, you just vendor your code as part of the source package. It's the distro maintainers' job to factor it out as a separate dependency.

mid-kid 3y ago
If you do this, remember to provide an option to factor it out. Meson (mesonbuild.org) makes this completely trivial and pitfall-free, and you should use it if you can't get away with just installing your libraries locally.
- varl 3y ago
  I went to mesonbuild.org and it doesn't match the description (some sort of betting site? I didn't stick around ...), and a search turned up: https://mesonbuild.com/ https://mesonbuild.com/
  - mid-kid 3y ago
    Thanks, I mistyped it.

no_wizard 3y ago

>I’m sympathetic to the problem. Supply chain attacks are way too easy This is the crux as to why crate.io is great: you have one official place for packages that can be continuously audited and monitored. This is the way to do security, in my opinion, because you are always able to look at incoming and outgoing packages. Now, you can argue if we are doing enough with this, and that is definitely up for debate, however, I contend that the premise of having one canonical place for package origination and hosting makes it easier to monitor for security purposes

samtheprogram 3y ago
That’s like saying it’s best to run everything on AWS, because there’s only one place to monitor for outages. That’s the exact opposite of how supply chain mitigation works.
- iudqnolq 3y ago
  It's not at all like saying that. If you want to mitigate downtime in your supply chain then diversity is good. If you want to mitigate security vulnerability then it's bad. Imagine you have 100 api servers that all have database credentials. If one server goes down you have 99 working servers (ideally). If one is compromised then the attacker has your database credentials, game over.
deleted 3y ago
[deleted]
josephg 3y ago
I - and many others - don’t think this is enough to mitigate supply chain attacks. It’s too easy for a malicious developer to add cryptolocker ransomware in a point release. But I also don’t think having a package in Debian will somehow magically solve this problem either. Debian package maintainers do not - and never have - done security audits on the packages they add. It’s magical thinking to assume using packages in Debian will somehow protect you from supply chain attacks. Asking developers to keep their dependency trees small so we can minimise the risk is a losing battle. Using lots of small, popular dependencies is just too convenient. There is a solution out there that I wish got more traction. And that is, capability based security. The idea is simple: we partition programs into application code and library code. Library code can never access the filesystem or network (or other protected data) directly. And this is enforced by the compiler or language runtime. Application code can. If you want a library to interact with a file, you first open the file in application code and pass the file handle to the library. Similar wrappers can be made for directories, network sockets and so on. If I want express to listen on a specific port, I pass it that port in a capability object. Not just an integer. (Eg app.listen(net.listenPort(8000))). The amount of code that would need to be changed is tiny since well behaved libraries don’t usually need many capabilities to do their job. It is ridiculous that my 3rd party json library can invisibly access all the sensitive files on my computer. With capabilities, we take that privilege away. I have no problem trusting a random json library if it literally only has access to the json string I pass it. If we had a capability based security model, I would have no qualms about pulling in hundreds of transitive dependencies. My libraries can still only access exactly what I pass them. Unfortunately this problem requires language level buy-in. It would be very hard to retrofit rust to work this way and that’ll probably never happen. Raw pointers and inline assembly also make a bit of a mess of things too. (Though pointer provenance might help). But a man can dream… Maybe in rust’s successor.

paulddraper 3y ago

You create apt, rpm repos. And users install from there. Not saying that's a nice answer, but that is the answer.

takluyver 3y ago
Users install from there, so long as they're using Linux, a Debian or Red Hat based distro, and have sudo access. As you say, not such a nice answer. For certain kinds of software, this approach is getting more viable, thanks to containers. In some contexts, you can tell everyone to run your software in a container with a specific Linux distro inside. But that software is often distributed as container images directly, rather than distro packages to be used in containers.
josephg 3y ago
You create apt, rpm repos. And arch. And gentoo. And FreeBSD ports. And homebrew and macports. And whatever windows does. And you need that to be true, on all those systems, for all your dependencies too. And at a recent, usable version. Or, I just use cargo. Up to date version. One command install that works on every operating system. Project scoped dependencies - so there’s no need for sudo, and different projects can depend on different versions of their dependencies. And I can publish new versions of my packages with 1 command and they’re available everywhere. Just in my house I have a Linux mint workstation, a windows gaming pc, a FreeBSD server and a Mac laptop. Cargo lets me publish code that all of them can consume with 1 command. So does npm. So long as apt is worse than cargo I won’t use it for this stuff. Why does everything have to get installed system wide? Nix doesn’t need that. Why doesn’t it have a programmatic mirror of cargo so we don’t need to mirror rust projects in it by hand? Why do major versions have different package names? And why don’t packages support feature flags? Last time I looked there were about 30 vim packages with random combinations of feature flags set. Why can’t evergreen packages like nodejs or cargo stay up to date in Debian? Old nodejs is useless because it can’t run modern JavaScript. I really want apt to solve all these problems because I hate docker. If apt and friends did their job well, docker wouldn’t be necessary or popular. Cargo isn’t pretty either. But it has all the features I need to get my job done. Make apt a compelling tool and I’ll switch to it. The feature list developers want is hardly a mystery.

zajio1am 3y ago

> as an author of some not very popular packages, what do I do? Depends. Do you want your software to be used by users? For me as a user, software that is not in distribution repository almost does not exist. It is too much hassle to install, manage and uninstall it compared to installing software from Linux distribution. If i really, really need it, i will make exception, but that are individual cases.

mattpallissard 3y ago

> “It works for C/C++” by making developers working in those languages pay in blood for every dependency they pull in. I think the industry may have thrown the baby out with the bath water. Now they're is a complete aversion to RYO anything, no matter how simple. Personally, I'm waiting for the pendulum to swing back towards the middle a bit more.

josephg 3y ago
Eh. I love rolling my own lots of things. At some point I really want to make a little OS kernel. How hard can it be? But then there’s things like rust’s rand crate which implements about 10 different random number generators, all with different properties around speed, whether they’re cryptographically secure and so on. And fed from a variety of entropy sources to make it work everywhere from Linux to windows to wasm. Look at this documentation. It’s an absolute master class. https://rust-random.github.io/book/guide-rngs.html https://rust-random.github.io/book/guide-rngs.html Do you need this? Probably not. You can probably get by with some wrapper around ar4random or whatever the recommended entropy source is today on your particular operating system. But I’ll be honest, until the day I decide to care about how rngs work, I’m going to just import the rand package. And serde & friends for json support. Criterion for benchmarking. And so on. Cargo is full of great libraries that are much higher quality than anything I’d ever throw together in a side project. For most of this stuff, you’d be mad to roll your own. This thread has reminded me that lots of people don’t know how nice it is to use a package manager like cargo. Here’s my favorite crate browsing website. You can see the most popular packages in every category and search. Look how much great stuff is in here. It’s humbling. https://lib.rs/ https://lib.rs/

IAmNotACellist 3y ago

>Yeah. The part that gives me goosebumps with that proposal is: as an author of some not very popular packages, what do I do? Is it my job to package and promote my packages on Debian, redhat, homebrew, and all the others? Do I need usage before they’ll pick it up? Sounds like a chicken and egg problem. And doing all that packaging work sounds like a boring, time consuming job that I don’t want. Github premium (or similar) will have an army of AI builder bots that spin up VMs and modify your codebase until it works, then send you PRs to approve. People would pay a ton for that service. All your projects, automatically maintained and packaged.