5 ms·

I think the need for a Magic Solver suggests weaknesses in how we package and load dependencies. Every piece of running code should have its own space for depen

I think the need for a Magic Solver suggests weaknesses in how we package and load dependencies. Every piece of running code should have its own space for dependencies. If most of them happen to coincide, then good: we can save a lot of RAM. But if not, also good.

epage 3y ago
The problem is when you are dealing with dependencies that are "vocabulary terms". These are required to be common. In Rust, one example would be serde and I'm assuming pydantic would similary apply. For Rust, these can also impact build times so people actively work to keep down what dependencies can be duplicated which are semver incompatible versions.
- droelf 3y ago
  Yeah, and what happens if you pass a NumPy array around and the different NumPy versions cannot agree on internal functions or memory layout ...
  - menaerus 3y ago
    I don't understand the purpose of package managers in general during the development where you have the total ownership of your code, 3rd party dependencies and whole build process. I had a privilege to use conan on semi-large codebase - what a clusterfuck that is. I can't imagine a single scenarios why I would be using it.
- 3cats-in-a-coat 3y ago
  Indeed but this comes back to opting to write spaghetti apps by default. For example we're talking in English right here, and it's a rich language we can express a whole lot in. It has no version although it's slowly updated over time. But it's mostly the same version around the world, with some local differences. We still understand each other. Despite our minds may be vastly different, we can express each other in this language. Same goes even for language models now. Imagine we had something as common as JSON, but more in-depth so it can be used as direct representation in memory, and this was the lingua franca not only between servers, but between libraries, apps, and even objects within an app. Then our dependency entanglement issue would also not exist. What we did was a choice. It was a lazy choice, ignorant choice. A choice that introduces instant tech debt. And dependency conflicts are one of the results of this choice.
  - eternityforest 3y ago
    It doesn't matter what the common language is, if library developers enjoy breaking changes like they seem to, they'll make them. They'll just remove features, change the way stuff works, or make trivial changes like renaming stuff... Android has API levels where they don't do that, but that's because Google can afford to provide almost everything themselves.... random npm and pypi contributors don't seem interested in keeping compatibility. So some kind of smart version system or container or something seems to be needed. Although, I think in a perfect world our standard libraries would be rich enough that we would not need many libraries, and the ones we did need would mostly only be a few hundred lines at most, and we'd basically never have libraries depending on other libraries or any tangled webs like that.
    - nerdponx 3y ago
      Maybe I'm just spoiled in Python, but library devs in the Python ecosystem seem highly disciplined in this respect. Packages tend to be forward-compatible for years, even with very active development cycles, and breaking changes tend to be treated carefully and announced with plenty of advance warning. That said, I don't understand this mentality that libraries can never change. Eventually things need to change. It's impossible to predict all future requirements.
  - nerdponx 3y ago
    > Imagine we had something as common as JSON, but more in-depth so it can be used as direct representation in memory, and this was the lingua franca not only between servers, but between libraries, apps, and even objects within an app. There are and have been several attempts to do things like this. One of them is the Arrow library for "arrays" as used in scientific computing, machine learning, etc. It's a huge amount of work.
  - antonvs 3y ago
    > What we did was a choice. It was a lazy choice, ignorant choice. This is utter nonsense. If you think you know how to do this better, there are billions or trillions of dollars waiting to reward you. Intelligent creatures like humans can fairly easily compensate for communication mismatches. That's enormously more difficult for us to program computers to do. We've only just stumbled on technology that might help with that - the latest generation of language models - but it'll probably take decades of work by everyone involved, on the entire planet, to be able to get to a point that resembles what you're imagining as the non-lazy, non-ignorant choice. Which proves that laziness and ignorance are not the issue.
antonvs 3y ago
That’s what containers do - package the dependencies with a piece of running code. That’s also one of the main reasons for their quick, wide adoption, even by conservative enterprises. But this gets more challenging as the granularity of what’s being packaged reduces. You can see this even in the container case: consider a container that depends on a particular version range of another container. For any possible solution to this, there are disadvantages which make it untenable in some situations. There’s no good general solution to this other than being explicit about the version dependencies and solving the dependency issues when deploying, whether manually or automatically. A automated solver helps with that. Btw, you can see something like what you described in Java programs built with Maven (for example). Run Maven’s dependency:tree function on any non-trivial program and you’ll see different versions of the same package appearing over and over, because each package does have its own (logical) space for dependencies. But Maven still has a version solver built in, and even with your proposed solution plus a solver, you can find plenty of articles on the web saying things like “Maven dependency conflicts can be really hard to solve.” The reason for that is when it fails, it’s because it’s encountered a difficult or impossible situation that (currently) requires human intervention.
- 3cats-in-a-coat 3y ago
  Containers are a work around for this issue, a Band-Aid.
  - antonvs 3y ago
    Not at all. They do exactly what you were suggesting. If you think they’re a bandaid, then you’d need to explain why your own suggestion isn’t a bandaid.
    - eternityforest 3y ago
      How many container systems can share dependencies in both RAM and disk, and do all the same kinds of IPC, while also sharing system daemons and all that? The way Android and more recent Windows do things is my favorite solution so far. All dependencies included in static packages, not actually containerized, and with all the heavy lifting done by things built right into the OS, always there, and specifically designed to be used that way, without much emphasis on lots of separate daemons, at least not on desktop. Even just saying "The OS always has this daemon" isn't enough, because so many services are designed to be configured specifically for a deployment and there isn't any one size fits all solution. Even something as simple as MQTT has a ton of ways to configure it, and there's no way for a client to even query what permissions it has and what other clients are on what available servers, it all has to be manually set up.

epage 3y ago

The problem is when you are dealing with dependencies that are "vocabulary terms". These are required to be common. In Rust, one example would be serde and I'm assuming pydantic would similary apply. For Rust, these can also impact build times so people actively work to keep down what dependencies can be duplicated which are semver incompatible versions.

droelf 3y ago
Yeah, and what happens if you pass a NumPy array around and the different NumPy versions cannot agree on internal functions or memory layout ...
- menaerus 3y ago
  I don't understand the purpose of package managers in general during the development where you have the total ownership of your code, 3rd party dependencies and whole build process. I had a privilege to use conan on semi-large codebase - what a clusterfuck that is. I can't imagine a single scenarios why I would be using it.
3cats-in-a-coat 3y ago
Indeed but this comes back to opting to write spaghetti apps by default. For example we're talking in English right here, and it's a rich language we can express a whole lot in. It has no version although it's slowly updated over time. But it's mostly the same version around the world, with some local differences. We still understand each other. Despite our minds may be vastly different, we can express each other in this language. Same goes even for language models now. Imagine we had something as common as JSON, but more in-depth so it can be used as direct representation in memory, and this was the lingua franca not only between servers, but between libraries, apps, and even objects within an app. Then our dependency entanglement issue would also not exist. What we did was a choice. It was a lazy choice, ignorant choice. A choice that introduces instant tech debt. And dependency conflicts are one of the results of this choice.
- eternityforest 3y ago
  It doesn't matter what the common language is, if library developers enjoy breaking changes like they seem to, they'll make them. They'll just remove features, change the way stuff works, or make trivial changes like renaming stuff... Android has API levels where they don't do that, but that's because Google can afford to provide almost everything themselves.... random npm and pypi contributors don't seem interested in keeping compatibility. So some kind of smart version system or container or something seems to be needed. Although, I think in a perfect world our standard libraries would be rich enough that we would not need many libraries, and the ones we did need would mostly only be a few hundred lines at most, and we'd basically never have libraries depending on other libraries or any tangled webs like that.
  - nerdponx 3y ago
    Maybe I'm just spoiled in Python, but library devs in the Python ecosystem seem highly disciplined in this respect. Packages tend to be forward-compatible for years, even with very active development cycles, and breaking changes tend to be treated carefully and announced with plenty of advance warning. That said, I don't understand this mentality that libraries can never change. Eventually things need to change. It's impossible to predict all future requirements.
- nerdponx 3y ago
  > Imagine we had something as common as JSON, but more in-depth so it can be used as direct representation in memory, and this was the lingua franca not only between servers, but between libraries, apps, and even objects within an app. There are and have been several attempts to do things like this. One of them is the Arrow library for "arrays" as used in scientific computing, machine learning, etc. It's a huge amount of work.
- antonvs 3y ago
  > What we did was a choice. It was a lazy choice, ignorant choice. This is utter nonsense. If you think you know how to do this better, there are billions or trillions of dollars waiting to reward you. Intelligent creatures like humans can fairly easily compensate for communication mismatches. That's enormously more difficult for us to program computers to do. We've only just stumbled on technology that might help with that - the latest generation of language models - but it'll probably take decades of work by everyone involved, on the entire planet, to be able to get to a point that resembles what you're imagining as the non-lazy, non-ignorant choice. Which proves that laziness and ignorance are not the issue.

antonvs 3y ago

That’s what containers do - package the dependencies with a piece of running code. That’s also one of the main reasons for their quick, wide adoption, even by conservative enterprises. But this gets more challenging as the granularity of what’s being packaged reduces. You can see this even in the container case: consider a container that depends on a particular version range of another container. For any possible solution to this, there are disadvantages which make it untenable in some situations. There’s no good general solution to this other than being explicit about the version dependencies and solving the dependency issues when deploying, whether manually or automatically. A automated solver helps with that. Btw, you can see something like what you described in Java programs built with Maven (for example). Run Maven’s dependency:tree function on any non-trivial program and you’ll see different versions of the same package appearing over and over, because each package does have its own (logical) space for dependencies. But Maven still has a version solver built in, and even with your proposed solution plus a solver, you can find plenty of articles on the web saying things like “Maven dependency conflicts can be really hard to solve.” The reason for that is when it fails, it’s because it’s encountered a difficult or impossible situation that (currently) requires human intervention.

3cats-in-a-coat 3y ago
Containers are a work around for this issue, a Band-Aid.
- antonvs 3y ago
  Not at all. They do exactly what you were suggesting. If you think they’re a bandaid, then you’d need to explain why your own suggestion isn’t a bandaid.
  - eternityforest 3y ago
    How many container systems can share dependencies in both RAM and disk, and do all the same kinds of IPC, while also sharing system daemons and all that? The way Android and more recent Windows do things is my favorite solution so far. All dependencies included in static packages, not actually containerized, and with all the heavy lifting done by things built right into the OS, always there, and specifically designed to be used that way, without much emphasis on lots of separate daemons, at least not on desktop. Even just saying "The OS always has this daemon" isn't enough, because so many services are designed to be configured specifically for a deployment and there isn't any one size fits all solution. Even something as simple as MQTT has a ton of ways to configure it, and there's no way for a client to even query what permissions it has and what other clients are on what available servers, it all has to be manually set up.