5 ms·

I don't think the exact URL is the problem, it is the fact that it is so easy to include dependencies from external repository that is the problem. In Rust eve

by dmz73 5y ago

I don't think the exact URL is the problem, it is the fact that it is so easy to include dependencies from external repository that is the problem. In Rust every non-trivial library pulls in 10s or even 100s of dependencies. I don't think anyone can expect that all of these libraries are of good quality but how would one even try to verify that? And you have to verify it every time you update your project. Then there is the issue of licencing - how to verify that I am not using some library in violation of its licence and what happens if the licence changes down the road and I don't notice it because I am implicitly using 500 dependencies due to my 3 main libraries? Rust and Go have solved memory safety compared to C and C++ but have introduced dependency hell of yet unknown proportions. Python and other dynamically typed languages are in a league of their own in that on top of the dependency hell they also do not provide compiler checks that would allow user to see the problem before the exact conditions occur at runtime. They are good for scripting but people keep pumping out full applications and to be honest there is not much difference between giant Python application, giant maze of Excel VBA and giant Node.js heap of code. Of those, Excel VBA is most likely to work for 5 years and across 5 versions of the product yet it is also the most likely one to receive the most negative comments.

throwaway894345 5y ago
> I don't think the exact URL is the problem, it is the fact that it is so easy to include dependencies from external repository that is the problem. In Rust every non-trivial library pulls in 10s or even 100s of dependencies. But it's also quite a lot easier to audit those dependencies, even automatically (incidentally, GitHub provides dependency scanning for free for many languages). > Then there is the issue of licencing - how to verify that I am not using some library in violation of its licence and what happens if the licence changes down the road and I don't notice it because I am implicitly using 500 dependencies due to my 3 main libraries? This is also an automated task. For example, https://github.com/google/go-licenses https://github.com/google/go-licenses: "go-licenses analyzes the dependency tree of a Go package/binary. It can output a report on the libraries used and under what license they can be used. It can also collect all of the license documents, copyright notices and source code into a directory in order to comply with license terms on redistribution." > Rust and Go have solved memory safety compared to C and C++ but have introduced dependency hell of yet unknown proportions. I mean, it's been a decade and things seem to be going pretty well. Also, I don't think anyone who has actually used these languages seriously has ever characterized their dependency management as "dependency hell"; however, lots of people talk about the "dependency hell" of managing C and C++ dependencies. > Python and other dynamically typed languages are in a league of their own in that on top of the dependency hell they also do not provide compiler checks that would allow user to see the problem before the exact conditions occur at runtime. I won't argue with you there.
jcranmer 5y ago
> In Rust every non-trivial library pulls in 10s or even 100s of dependencies. You're exaggerating here. The most recent project I've been working on pulls in 6 dependencies. The anyhow crate has no dependencies, regex 3 (recursively!), clap and csv each 8. Only handlebars and palette pull in 10s of dependencies, and I can trim a fair few dependencies of palette by opting out of named color support (dropping the phf crate for perfect hash functions).
- avar 5y ago
  It's typical for very large C projects to have perhaps 2-5 dependencies that aren't libc, often something very basic such as zlib, curl or openssl. A rust CSV parser has 8 dependencies?
  - throwaway894345 5y ago
    A Go program wouldn't even need curl, zlib, or openssl as it has equivalent implementations in its stdlib.
  - steveklabnik 5y ago
    The CSV crate has five of them: https://crates.io/crates/csv/1.1.6/dependencies https://crates.io/crates/csv/1.1.6/dependencies
  - jcranmer 5y ago
    Three of those dependencies are not entirely unreasonable: one is the actual "core" CSV implementation (with one recursive dependency), one is the dependency for the standard "this interface allows you to serialize/deserialize arbitrary data to arbitrary formats", and one crate that adds traits to make &[u8]/Vec<u8> work much more like &str/String (which has 4 recursive dependencies, although two of those are already mentioned). The last two dependencies are crates that provide accelerated routines for serializing integers and floating points, which are IMHO somewhat excessive, but if it's easy to pull in such dependencies, why not? It would be helpful in places where CSV serialization might actually be a bottleneck. There are two key differences between large projects in ecosystems that have working package managers and those in C/C++. For starters, the basic working 'package' tends to be smaller. Consider something like LLVM (just LLVM itself, none of its subprojects like Clang). This you'd count as "one project" without any dependencies (at least, no required dependencies that aren't vendored, liked zlib and googletest). But if you were to make it in something like Rust, it would likely consist of a dozen or more packages: the core IR, analysis passes, command-line tools, transformations, codegen (with each target likely being its own package!). With such fragmentation, you'd have dozens of dependencies that are essentially nothing more than the 'project' itself. But more importantly is that, because dependencies are hard to add, code duplication is the norm; it's only when what you want to do is too impossible to duplicate that you resort to a dependency. For any large C++ project, I can virtually guarantee that there will exist some form of 'standard library++' dependency in the code that will contain at least an intrusive linked list, better hashtable implementation, std::vector-lookalike that avoids heap allocation for small lists, replacement for iostream, helpers for process creation, custom string formatting routines, and something to make dealing with segfaults sane. In a package ecosystem, each of those kinds of things would be a separate dependency. Is it really necessary for everybody who wants to write something dealing with graphs to have write their own graph-based data structure, as opposed to just using the ecosystem-standard graph library (e.g., networkx in Python, or petgraph in Rust)? Or to have everybody that wants to use a hashtable to have to write their own implementation of a notoriously tricky data structure?

by dmz73 5y ago

throwaway894345 5y ago

> I don't think the exact URL is the problem, it is the fact that it is so easy to include dependencies from external repository that is the problem. In Rust every non-trivial library pulls in 10s or even 100s of dependencies. But it's also quite a lot easier to audit those dependencies, even automatically (incidentally, GitHub provides dependency scanning for free for many languages). > Then there is the issue of licencing - how to verify that I am not using some library in violation of its licence and what happens if the licence changes down the road and I don't notice it because I am implicitly using 500 dependencies due to my 3 main libraries? This is also an automated task. For example, https://github.com/google/go-licenses https://github.com/google/go-licenses: "go-licenses analyzes the dependency tree of a Go package/binary. It can output a report on the libraries used and under what license they can be used. It can also collect all of the license documents, copyright notices and source code into a directory in order to comply with license terms on redistribution." > Rust and Go have solved memory safety compared to C and C++ but have introduced dependency hell of yet unknown proportions. I mean, it's been a decade and things seem to be going pretty well. Also, I don't think anyone who has actually used these languages seriously has ever characterized their dependency management as "dependency hell"; however, lots of people talk about the "dependency hell" of managing C and C++ dependencies. > Python and other dynamically typed languages are in a league of their own in that on top of the dependency hell they also do not provide compiler checks that would allow user to see the problem before the exact conditions occur at runtime. I won't argue with you there.

jcranmer 5y ago

> In Rust every non-trivial library pulls in 10s or even 100s of dependencies. You're exaggerating here. The most recent project I've been working on pulls in 6 dependencies. The anyhow crate has no dependencies, regex 3 (recursively!), clap and csv each 8. Only handlebars and palette pull in 10s of dependencies, and I can trim a fair few dependencies of palette by opting out of named color support (dropping the phf crate for perfect hash functions).

avar 5y ago
It's typical for very large C projects to have perhaps 2-5 dependencies that aren't libc, often something very basic such as zlib, curl or openssl. A rust CSV parser has 8 dependencies?
- throwaway894345 5y ago
  A Go program wouldn't even need curl, zlib, or openssl as it has equivalent implementations in its stdlib.
- steveklabnik 5y ago
  The CSV crate has five of them: https://crates.io/crates/csv/1.1.6/dependencies https://crates.io/crates/csv/1.1.6/dependencies
- jcranmer 5y ago
  Three of those dependencies are not entirely unreasonable: one is the actual "core" CSV implementation (with one recursive dependency), one is the dependency for the standard "this interface allows you to serialize/deserialize arbitrary data to arbitrary formats", and one crate that adds traits to make &[u8]/Vec<u8> work much more like &str/String (which has 4 recursive dependencies, although two of those are already mentioned). The last two dependencies are crates that provide accelerated routines for serializing integers and floating points, which are IMHO somewhat excessive, but if it's easy to pull in such dependencies, why not? It would be helpful in places where CSV serialization might actually be a bottleneck. There are two key differences between large projects in ecosystems that have working package managers and those in C/C++. For starters, the basic working 'package' tends to be smaller. Consider something like LLVM (just LLVM itself, none of its subprojects like Clang). This you'd count as "one project" without any dependencies (at least, no required dependencies that aren't vendored, liked zlib and googletest). But if you were to make it in something like Rust, it would likely consist of a dozen or more packages: the core IR, analysis passes, command-line tools, transformations, codegen (with each target likely being its own package!). With such fragmentation, you'd have dozens of dependencies that are essentially nothing more than the 'project' itself. But more importantly is that, because dependencies are hard to add, code duplication is the norm; it's only when what you want to do is too impossible to duplicate that you resort to a dependency. For any large C++ project, I can virtually guarantee that there will exist some form of 'standard library++' dependency in the code that will contain at least an intrusive linked list, better hashtable implementation, std::vector-lookalike that avoids heap allocation for small lists, replacement for iostream, helpers for process creation, custom string formatting routines, and something to make dealing with segfaults sane. In a package ecosystem, each of those kinds of things would be a separate dependency. Is it really necessary for everybody who wants to write something dealing with graphs to have write their own graph-based data structure, as opposed to just using the ecosystem-standard graph library (e.g., networkx in Python, or petgraph in Rust)? Or to have everybody that wants to use a hashtable to have to write their own implementation of a notoriously tricky data structure?