3 ms·

The problem is when you are dealing with dependencies that are "vocabulary terms". These are required to be common. In Rust, one example would be serde and I'm

by epage 3y ago

The problem is when you are dealing with dependencies that are "vocabulary terms". These are required to be common. In Rust, one example would be serde and I'm assuming pydantic would similary apply. For Rust, these can also impact build times so people actively work to keep down what dependencies can be duplicated which are semver incompatible versions.

droelf 3y ago
Yeah, and what happens if you pass a NumPy array around and the different NumPy versions cannot agree on internal functions or memory layout ...
- menaerus 3y ago
  I don't understand the purpose of package managers in general during the development where you have the total ownership of your code, 3rd party dependencies and whole build process. I had a privilege to use conan on semi-large codebase - what a clusterfuck that is. I can't imagine a single scenarios why I would be using it.
3cats-in-a-coat 3y ago
Indeed but this comes back to opting to write spaghetti apps by default. For example we're talking in English right here, and it's a rich language we can express a whole lot in. It has no version although it's slowly updated over time. But it's mostly the same version around the world, with some local differences. We still understand each other. Despite our minds may be vastly different, we can express each other in this language. Same goes even for language models now. Imagine we had something as common as JSON, but more in-depth so it can be used as direct representation in memory, and this was the lingua franca not only between servers, but between libraries, apps, and even objects within an app. Then our dependency entanglement issue would also not exist. What we did was a choice. It was a lazy choice, ignorant choice. A choice that introduces instant tech debt. And dependency conflicts are one of the results of this choice.
- eternityforest 3y ago
  It doesn't matter what the common language is, if library developers enjoy breaking changes like they seem to, they'll make them. They'll just remove features, change the way stuff works, or make trivial changes like renaming stuff... Android has API levels where they don't do that, but that's because Google can afford to provide almost everything themselves.... random npm and pypi contributors don't seem interested in keeping compatibility. So some kind of smart version system or container or something seems to be needed. Although, I think in a perfect world our standard libraries would be rich enough that we would not need many libraries, and the ones we did need would mostly only be a few hundred lines at most, and we'd basically never have libraries depending on other libraries or any tangled webs like that.
  - nerdponx 3y ago
    Maybe I'm just spoiled in Python, but library devs in the Python ecosystem seem highly disciplined in this respect. Packages tend to be forward-compatible for years, even with very active development cycles, and breaking changes tend to be treated carefully and announced with plenty of advance warning. That said, I don't understand this mentality that libraries can never change. Eventually things need to change. It's impossible to predict all future requirements.
- nerdponx 3y ago
  > Imagine we had something as common as JSON, but more in-depth so it can be used as direct representation in memory, and this was the lingua franca not only between servers, but between libraries, apps, and even objects within an app. There are and have been several attempts to do things like this. One of them is the Arrow library for "arrays" as used in scientific computing, machine learning, etc. It's a huge amount of work.
- antonvs 3y ago
  > What we did was a choice. It was a lazy choice, ignorant choice. This is utter nonsense. If you think you know how to do this better, there are billions or trillions of dollars waiting to reward you. Intelligent creatures like humans can fairly easily compensate for communication mismatches. That's enormously more difficult for us to program computers to do. We've only just stumbled on technology that might help with that - the latest generation of language models - but it'll probably take decades of work by everyone involved, on the entire planet, to be able to get to a point that resembles what you're imagining as the non-lazy, non-ignorant choice. Which proves that laziness and ignorance are not the issue.

by epage 3y ago

droelf 3y ago

Yeah, and what happens if you pass a NumPy array around and the different NumPy versions cannot agree on internal functions or memory layout ...

menaerus 3y ago
I don't understand the purpose of package managers in general during the development where you have the total ownership of your code, 3rd party dependencies and whole build process. I had a privilege to use conan on semi-large codebase - what a clusterfuck that is. I can't imagine a single scenarios why I would be using it.

3cats-in-a-coat 3y ago

Indeed but this comes back to opting to write spaghetti apps by default. For example we're talking in English right here, and it's a rich language we can express a whole lot in. It has no version although it's slowly updated over time. But it's mostly the same version around the world, with some local differences. We still understand each other. Despite our minds may be vastly different, we can express each other in this language. Same goes even for language models now. Imagine we had something as common as JSON, but more in-depth so it can be used as direct representation in memory, and this was the lingua franca not only between servers, but between libraries, apps, and even objects within an app. Then our dependency entanglement issue would also not exist. What we did was a choice. It was a lazy choice, ignorant choice. A choice that introduces instant tech debt. And dependency conflicts are one of the results of this choice.

eternityforest 3y ago
It doesn't matter what the common language is, if library developers enjoy breaking changes like they seem to, they'll make them. They'll just remove features, change the way stuff works, or make trivial changes like renaming stuff... Android has API levels where they don't do that, but that's because Google can afford to provide almost everything themselves.... random npm and pypi contributors don't seem interested in keeping compatibility. So some kind of smart version system or container or something seems to be needed. Although, I think in a perfect world our standard libraries would be rich enough that we would not need many libraries, and the ones we did need would mostly only be a few hundred lines at most, and we'd basically never have libraries depending on other libraries or any tangled webs like that.
- nerdponx 3y ago
  Maybe I'm just spoiled in Python, but library devs in the Python ecosystem seem highly disciplined in this respect. Packages tend to be forward-compatible for years, even with very active development cycles, and breaking changes tend to be treated carefully and announced with plenty of advance warning. That said, I don't understand this mentality that libraries can never change. Eventually things need to change. It's impossible to predict all future requirements.
nerdponx 3y ago
> Imagine we had something as common as JSON, but more in-depth so it can be used as direct representation in memory, and this was the lingua franca not only between servers, but between libraries, apps, and even objects within an app. There are and have been several attempts to do things like this. One of them is the Arrow library for "arrays" as used in scientific computing, machine learning, etc. It's a huge amount of work.
antonvs 3y ago
> What we did was a choice. It was a lazy choice, ignorant choice. This is utter nonsense. If you think you know how to do this better, there are billions or trillions of dollars waiting to reward you. Intelligent creatures like humans can fairly easily compensate for communication mismatches. That's enormously more difficult for us to program computers to do. We've only just stumbled on technology that might help with that - the latest generation of language models - but it'll probably take decades of work by everyone involved, on the entire planet, to be able to get to a point that resembles what you're imagining as the non-lazy, non-ignorant choice. Which proves that laziness and ignorance are not the issue.