5 ms·

It's typical for very large C projects to have perhaps 2-5 dependencies that aren't libc, often something very basic such as zlib, curl or openssl. A rust CSV p

by avar 5y ago

It's typical for very large C projects to have perhaps 2-5 dependencies that aren't libc, often something very basic such as zlib, curl or openssl. A rust CSV parser has 8 dependencies?

throwaway894345 5y ago
A Go program wouldn't even need curl, zlib, or openssl as it has equivalent implementations in its stdlib.
steveklabnik 5y ago
The CSV crate has five of them: https://crates.io/crates/csv/1.1.6/dependencies https://crates.io/crates/csv/1.1.6/dependencies
jcranmer 5y ago
Three of those dependencies are not entirely unreasonable: one is the actual "core" CSV implementation (with one recursive dependency), one is the dependency for the standard "this interface allows you to serialize/deserialize arbitrary data to arbitrary formats", and one crate that adds traits to make &[u8]/Vec<u8> work much more like &str/String (which has 4 recursive dependencies, although two of those are already mentioned). The last two dependencies are crates that provide accelerated routines for serializing integers and floating points, which are IMHO somewhat excessive, but if it's easy to pull in such dependencies, why not? It would be helpful in places where CSV serialization might actually be a bottleneck. There are two key differences between large projects in ecosystems that have working package managers and those in C/C++. For starters, the basic working 'package' tends to be smaller. Consider something like LLVM (just LLVM itself, none of its subprojects like Clang). This you'd count as "one project" without any dependencies (at least, no required dependencies that aren't vendored, liked zlib and googletest). But if you were to make it in something like Rust, it would likely consist of a dozen or more packages: the core IR, analysis passes, command-line tools, transformations, codegen (with each target likely being its own package!). With such fragmentation, you'd have dozens of dependencies that are essentially nothing more than the 'project' itself. But more importantly is that, because dependencies are hard to add, code duplication is the norm; it's only when what you want to do is too impossible to duplicate that you resort to a dependency. For any large C++ project, I can virtually guarantee that there will exist some form of 'standard library++' dependency in the code that will contain at least an intrusive linked list, better hashtable implementation, std::vector-lookalike that avoids heap allocation for small lists, replacement for iostream, helpers for process creation, custom string formatting routines, and something to make dealing with segfaults sane. In a package ecosystem, each of those kinds of things would be a separate dependency. Is it really necessary for everybody who wants to write something dealing with graphs to have write their own graph-based data structure, as opposed to just using the ecosystem-standard graph library (e.g., networkx in Python, or petgraph in Rust)? Or to have everybody that wants to use a hashtable to have to write their own implementation of a notoriously tricky data structure?
- avar 5y ago
  I wasn't attempting to make a value-judgement about whether something /should/ have these dependencies, but only to point out that if you think this is comparable to C-based projects you seemed to be pretty out of touch with common practices in that domain. I take your point that NIH is a large thing driving this sort of divergence in many languages, with some (such as Common Lisp, and C) being more notorious than others. But I'd say it's just as common in other languages for programmers to pile on needless dependencies. E.g. how many Rust/Node.js/Java etc. projects are using some "standard library++" to e.g. get better memory behavior for strings, but for whom string allocation overhead is never ever going to be an issue? It would be an interesting metric to check what % of the code in a given library is commonly used by its consumers across languages. Even if you only used 1% you're implicitly assuming some of the maintenance overhead for 100% of it. E.g. if it won't build on an older toolchain that's now a hassle for someone packaging your program, even if it's in the 99% you don't use.

by avar 5y ago

It's typical for very large C projects to have perhaps 2-5 dependencies that aren't libc, often something very basic such as zlib, curl or openssl. A rust CSV parser has 8 dependencies?

throwaway894345 5y ago

A Go program wouldn't even need curl, zlib, or openssl as it has equivalent implementations in its stdlib.

steveklabnik 5y ago

The CSV crate has five of them: https://crates.io/crates/csv/1.1.6/dependencies https://crates.io/crates/csv/1.1.6/dependencies

jcranmer 5y ago

Three of those dependencies are not entirely unreasonable: one is the actual "core" CSV implementation (with one recursive dependency), one is the dependency for the standard "this interface allows you to serialize/deserialize arbitrary data to arbitrary formats", and one crate that adds traits to make &[u8]/Vec<u8> work much more like &str/String (which has 4 recursive dependencies, although two of those are already mentioned). The last two dependencies are crates that provide accelerated routines for serializing integers and floating points, which are IMHO somewhat excessive, but if it's easy to pull in such dependencies, why not? It would be helpful in places where CSV serialization might actually be a bottleneck. There are two key differences between large projects in ecosystems that have working package managers and those in C/C++. For starters, the basic working 'package' tends to be smaller. Consider something like LLVM (just LLVM itself, none of its subprojects like Clang). This you'd count as "one project" without any dependencies (at least, no required dependencies that aren't vendored, liked zlib and googletest). But if you were to make it in something like Rust, it would likely consist of a dozen or more packages: the core IR, analysis passes, command-line tools, transformations, codegen (with each target likely being its own package!). With such fragmentation, you'd have dozens of dependencies that are essentially nothing more than the 'project' itself. But more importantly is that, because dependencies are hard to add, code duplication is the norm; it's only when what you want to do is too impossible to duplicate that you resort to a dependency. For any large C++ project, I can virtually guarantee that there will exist some form of 'standard library++' dependency in the code that will contain at least an intrusive linked list, better hashtable implementation, std::vector-lookalike that avoids heap allocation for small lists, replacement for iostream, helpers for process creation, custom string formatting routines, and something to make dealing with segfaults sane. In a package ecosystem, each of those kinds of things would be a separate dependency. Is it really necessary for everybody who wants to write something dealing with graphs to have write their own graph-based data structure, as opposed to just using the ecosystem-standard graph library (e.g., networkx in Python, or petgraph in Rust)? Or to have everybody that wants to use a hashtable to have to write their own implementation of a notoriously tricky data structure?

avar 5y ago
I wasn't attempting to make a value-judgement about whether something /should/ have these dependencies, but only to point out that if you think this is comparable to C-based projects you seemed to be pretty out of touch with common practices in that domain. I take your point that NIH is a large thing driving this sort of divergence in many languages, with some (such as Common Lisp, and C) being more notorious than others. But I'd say it's just as common in other languages for programmers to pile on needless dependencies. E.g. how many Rust/Node.js/Java etc. projects are using some "standard library++" to e.g. get better memory behavior for strings, but for whom string allocation overhead is never ever going to be an issue? It would be an interesting metric to check what % of the code in a given library is commonly used by its consumers across languages. Even if you only used 1% you're implicitly assuming some of the maintenance overhead for 100% of it. E.g. if it won't build on an older toolchain that's now a hassle for someone packaging your program, even if it's in the 99% you don't use.