20 ms·

Maven doesn't have these problems, but not because of namespacing itself. The adjacent thread has the details on that.

lolinder 3y ago
Usually with something like this is the whole system that makes it work—namespacing alone may not be enough to solve all our package manager woes, but can a package manager solve them without namespacing?
- mike_hearn 3y ago
  The Java ecosystem doesn't have these problems for several reasons, not just namespacing: 1. The Java ecosystem has no way to distribute executable programs. Maven is only for libraries. There is literally no equivalent of "npm install" or "pip install". This is not a strength! IMHO it's one of the things that pushes people away and towards scripting languages. Being able to distribute programs as well as libraries is a very useful feature. It does, however, mean that there's very little value to be had in pushing malware to Maven Central because the instructions for how to run anything from it would be more complicated than just distributing the JARs yourself. State actors might want to compromise widely used libraries, but that's a different kettle of fish. 2. Not only can you not distribute programs but Maven resolvers don't execute any code. Dependencies are statically declared and the "installation" process just involves walking the dependency tree, downloading the JARs to a file system cache and then computing a list of those JARs for the build system. There's no equivalent of setup.py. You might be wondering, how then do you distribute libraries that require native code? Well, the Java ecosystem has no direct support for that either. You'll have to pre-compile your native libs for each OS and CPU arch that users might want, distribute the libraries as data resources in your JAR, extract them to somewhere in the user's home directory at runtime and load them from there. (There's also no standard mechanism for this so every library rolls its own). 3. Maven Central (but not the clients) requires that all uploads are PGP signed. So, to be able to upload bad releases of existing libraries requires you steal a signing key. 4. The JVM ecosystem is oriented around larger but fewer libraries. Although individuals do create and upload libraries of course, the huge standard library means you don't need to rely on them so often. There is no leftpad library for Java. Many dependencies will come from Google, Apache, JetBrains, etc. When you do rely on individuals they tend to have been around a long time, their libraries are relatively well known, have a lot of entries in the issue tracker, long lived mailing lists and other unforgeable evidence of real-ness. 5. All operating systems come with Python built in these days, even Windows. In contrast no OS comes with Java out of the box. So if you just want people to run malware Python is a better bet because it's just one command, users won't have to install Python itself. I don't think having a central server is that big a difference in the grand scheme of things, though it does mean someone owns the problem of clearing bad packages were one to get in.
  - lolinder 3y ago
    Points 1-4 all sound like substantial strengths to me. 1. Who decided that the same package manager needed to be in charge of downloading libraries for CI pipelines and finished executables? Combining the two muddies the water and makes it unclear if a given package is going to provide a library or execute code or both. Bad actors thrive in confused spaces. 2. This comes down to the same point: I don't see why the package manager for coordinating dependencies in a software project ought to be the same thing as the package manager for installing applications in an OS. They're different use cases, usually different sets of users, and, as you note, different sets of requirements. Nothing good comes from a library being able to execute code as it is downloaded, so Maven lacks that feature because that's not the use case it's targeting. 3. Sounds fabulous. When is this feature coming to PyPI? 4. This doesn't seem like an incidental difference, it's the natural result of the design of the system, and a big part of the reason for the big-package culture is Maven Central. If the Python and JavaScript ecosystems made package ownership more explicit, the culture of thousands of code dependencies would be more obviously a culture of thousands of human dependencies, and I think people would be much more freaked out with the status quo. 5. See points 1 and 2 above: the Python and JS ecosystems make a muddled mess of the distribution between development and distribution, and the result is a dangerous hybrid package manager that is frustrating for either use case.
    - mike_hearn 3y ago
      1. Agree that having a clear separation at the UI level is of value, but a program is a set of modules that depend on each other, no different to a library. Where you start needing very different things is when distributing software to end users who aren't developers, but there's a lot of small utility programs, demos etc that would benefit from being distributed the same way as libraries. There's a reason that so many dev frameworks these days start by telling you to install a framework-specific CLI tool to help you with things. 2. I'm thinking here of cases where the users aren't really different. Apps for developers, for example. Should have made that clear, sorry! 3. PyPI already tried to use PGP signatures and gave up. It doesn't add much security over just having a good authentication system on a central server. Note that nothing checks the signatures except, I think, Maven Central itself, so it doesn't help in case of server compromise. We're in agreement that the situation with Python/JS (and to a lesser extent Ruby?) is a real mess, but I don't see it as due to any fundamental design choices. More like a set of fuzzy accidents of history. If someone added a new command to mvn or gradle tomorrow, for example, (1) might stop being true.

Usually with something like this is the whole system that makes it work—namespacing alone may not be enough to solve all our package manager woes, but can a package manager solve them without namespacing?

mike_hearn 3y ago
The Java ecosystem doesn't have these problems for several reasons, not just namespacing: 1. The Java ecosystem has no way to distribute executable programs. Maven is only for libraries. There is literally no equivalent of "npm install" or "pip install". This is not a strength! IMHO it's one of the things that pushes people away and towards scripting languages. Being able to distribute programs as well as libraries is a very useful feature. It does, however, mean that there's very little value to be had in pushing malware to Maven Central because the instructions for how to run anything from it would be more complicated than just distributing the JARs yourself. State actors might want to compromise widely used libraries, but that's a different kettle of fish. 2. Not only can you not distribute programs but Maven resolvers don't execute any code. Dependencies are statically declared and the "installation" process just involves walking the dependency tree, downloading the JARs to a file system cache and then computing a list of those JARs for the build system. There's no equivalent of setup.py. You might be wondering, how then do you distribute libraries that require native code? Well, the Java ecosystem has no direct support for that either. You'll have to pre-compile your native libs for each OS and CPU arch that users might want, distribute the libraries as data resources in your JAR, extract them to somewhere in the user's home directory at runtime and load them from there. (There's also no standard mechanism for this so every library rolls its own). 3. Maven Central (but not the clients) requires that all uploads are PGP signed. So, to be able to upload bad releases of existing libraries requires you steal a signing key. 4. The JVM ecosystem is oriented around larger but fewer libraries. Although individuals do create and upload libraries of course, the huge standard library means you don't need to rely on them so often. There is no leftpad library for Java. Many dependencies will come from Google, Apache, JetBrains, etc. When you do rely on individuals they tend to have been around a long time, their libraries are relatively well known, have a lot of entries in the issue tracker, long lived mailing lists and other unforgeable evidence of real-ness. 5. All operating systems come with Python built in these days, even Windows. In contrast no OS comes with Java out of the box. So if you just want people to run malware Python is a better bet because it's just one command, users won't have to install Python itself. I don't think having a central server is that big a difference in the grand scheme of things, though it does mean someone owns the problem of clearing bad packages were one to get in.
- lolinder 3y ago
  Points 1-4 all sound like substantial strengths to me. 1. Who decided that the same package manager needed to be in charge of downloading libraries for CI pipelines and finished executables? Combining the two muddies the water and makes it unclear if a given package is going to provide a library or execute code or both. Bad actors thrive in confused spaces. 2. This comes down to the same point: I don't see why the package manager for coordinating dependencies in a software project ought to be the same thing as the package manager for installing applications in an OS. They're different use cases, usually different sets of users, and, as you note, different sets of requirements. Nothing good comes from a library being able to execute code as it is downloaded, so Maven lacks that feature because that's not the use case it's targeting. 3. Sounds fabulous. When is this feature coming to PyPI? 4. This doesn't seem like an incidental difference, it's the natural result of the design of the system, and a big part of the reason for the big-package culture is Maven Central. If the Python and JavaScript ecosystems made package ownership more explicit, the culture of thousands of code dependencies would be more obviously a culture of thousands of human dependencies, and I think people would be much more freaked out with the status quo. 5. See points 1 and 2 above: the Python and JS ecosystems make a muddled mess of the distribution between development and distribution, and the result is a dangerous hybrid package manager that is frustrating for either use case.
  - mike_hearn 3y ago
    1. Agree that having a clear separation at the UI level is of value, but a program is a set of modules that depend on each other, no different to a library. Where you start needing very different things is when distributing software to end users who aren't developers, but there's a lot of small utility programs, demos etc that would benefit from being distributed the same way as libraries. There's a reason that so many dev frameworks these days start by telling you to install a framework-specific CLI tool to help you with things. 2. I'm thinking here of cases where the users aren't really different. Apps for developers, for example. Should have made that clear, sorry! 3. PyPI already tried to use PGP signatures and gave up. It doesn't add much security over just having a good authentication system on a central server. Note that nothing checks the signatures except, I think, Maven Central itself, so it doesn't help in case of server compromise. We're in agreement that the situation with Python/JS (and to a lesser extent Ruby?) is a real mess, but I don't see it as due to any fundamental design choices. More like a set of fuzzy accidents of history. If someone added a new command to mvn or gradle tomorrow, for example, (1) might stop being true.