13 ms·

Build a Database in Four Months with Rust and 647 Open-Source Dependencies

130 points by tison 2y ago

henning 2y ago
I automatically don't want to use this database because the number of third party dependencies are an unfixable, never-ending source of security vulnerabilities.
- bityard 2y ago
  Sometimes I'm pretty sure people upvote stories just to see what happens in the comments.
  - Idiot211 2y ago
    Guilty as charged. To steal a phrase from Reddit, "the true LPT is in the comments" The true insightful discussion comes in the comments.
    - callamdelaney 2y ago
      LPT?
      - orion138 2y ago
        I believe it means Life Pro Tip.
  - airstrike 2y ago
    Guilty as charged.
- arccy 2y ago
  yeah, rust copied the dumpster fire that was npm, i shudder to think of the future of supply chain security when people say rewrite it in rust.
  - norman784 2y ago
    What would a better model to manage dependencies in your opinion? I do like that is easy to add dependencies, but also don't like that a simple hello world Axum app IIRC is around 150 dependencies.
    - reaperducer 2y ago
      You don't have to have a solution to recognize that there is a problem.
      - kibwen 2y ago
        This is both right and wrong in a pernicious way. When pointing out a problem, you don't necessarily need to provide a better solution. However, if you refrain from providing a better solution, you are still implicitly asserting that there exists some better solution. So then it's possible to counter that with: a better solution may not exist. If you think a better solution does exist, then the burden of proof is on you to point out an existing solution that does better, or to otherwise establish that some better solution must exist. Rust could very well be at a global optimum for the problems it's trying to solve. Sometimes tradeoffs are just inevitable.
        arccy 2y ago
        it may be that rust tried to solve for the wrong problems, so while it may be at the global optimum, the foundation is just broken. that said, design choices like a flat package namespace are inexcusable. even npm started to move away from it.
        jpc0 2y ago
        I would say, evaluate how much work it would take to build it yourself, and if that is larger than your scope allows ask yourself some serious questions about whether the solution you came up with is overly complex. Some problems are hard to solve. But not all of them are. An example, and this is an observation. Where can I grab a library that just parses parses HTTP 1.0, HTTP 1.1 and HTTP 2.0 messages. Not a HTTP framework, something along the lines of httpparse. I pass it a buffer of bytes and out the other end pops a Result<error, HTTPResponse>. Sure there are hard problems to solve there but they are API design problems. I don't need tokio or some sort of web abstraction. If I want to use HTTP as a transport over carrier pidgeon I want to be able to do that with said library. Doesn't exactly exist though and because everyone just pulls in Tokio( I do mean the entire Tower stack or whatever it's called these says) nobody even notices the issue. And every single HTTP server rewrites that functionality with slightly different edge cases and bugs. That's basic internet infrastructure right there and we can't get a conical library for that yet you are arguing that 3 different implementations of the aame hash function pulled into the same project is viable?
- wslh 2y ago
  Nowadays this applies to everything that depends on modules that depend on more modules (e.g. NodeJS).
- rectang 2y ago
  Yes, the amount of effort it takes to audit dependencies scales roughly linearly, so unless you're going to blindly install them, choosing to use a project with so many dependencies means taking on a tremendous amount of ongoing work.
  - estebank 2y ago
    > the amount of effort it takes to audit dependencies scales roughly linearly With the lines of code, not the number of dependencies. 10 dependencies of 100 lines of code are arguably easier, but certainly not harder than a single dependency of 1000 lines of code.
    - rectang 2y ago
      I should clarify that I mean auditing dependency-publisher authentication, rather than full code review. This returns us to status quo ante, back before supply chain attacks were something we worried about. Bugs and such from dependencies are an annoyance but a manageable problem. Supply chain attacks after publisher account compromise are catastrophic and are not manageable.
      - estebank 2y ago
        I see, I have a different mental model for what auditing a dependency means. Auditing is "review the code and release processes of my dependency". In my mind what you describe would be "validating my Software Bill of Materials". It doesn't mean that either of us is wrong on what we call auditing, it just explains why sometimes we end up talking past each other in these conversations.
      - marcosdumay 2y ago
        > auditing dependency-publisher authentication What does this mean? It means you'll trust the random people pushing code to cargo if you can prove they indeed are the random people they claim to be?
        2y ago
dboreham 2y ago
Poster boy for all that's wrong with modern software modularity.
Deukhoofd 2y ago
I really chuckled about how the blog post opens with how great Rusts open-source ecosystem is, and ends with an "anyway, we made our software private and proprietary"
- bdcravens 2y ago
  Isn't that pretty much the modern stack? Open source language, framework, and libraries, and proprietary end product?
  - goodpoint 2y ago
    And then tech companies fire engineers while making record profits.
- bbkane 2y ago
  That's technically correct, but they listed several ways they contribute back to the OSS ecosystem: PRs, issues, creating new libraries... This comment makes it seem like all this company does is take, which feels unfair to me
  - PittleyDunkin 2y ago
    >This comment makes it seem like all this company does is take, which feels unfair to me Profit isn't far removed from theft, so maybe this shouldn't feel so unfair.
    - bbkane 2y ago
      > Profit isn't far removed from theft I definitely think there are unethical ways to profit - capitalism needs to be regulated for the good of the consumer/ecosystem/society. However, I don't believe that a blanket comparison of any type of profit to theft can be useful or correct. > so maybe this shouldn't feel so unfair Do you think this company is unethical for writing closed source software and trying to sell it?
etaioinshrdlu 2y ago
My main question is why observability data needs (or benefits from) a tailor-made database instead of a general purpose one. In 2025, anyone working on observability who told me they have to build their own database, I would be very suspicious!
- tison 2y ago
  Datadog always builds their own event store: https://www.datadoghq.com/blog/engineering/introducing-husky/ https://www.datadoghq.com/blog/engineering/introducing-husky... It may not be named "database" but actually take the place of a database. Observability vendors will try to store logs with ElasticSearch and later find it over expensive and has weak support for archiving cold data. Data Warehouse solution requires a complex ETL pipeline and can be awkward when handling log data (semi-structured data). That said, if you're building an observability solution for a single company, I'd totally agree to start with single node PG with backup, and only consider other solution when data and query workload grow.
  - jcgrillo 2y ago
    In 2025 I'd consider starting with clickhouse instead, if you're going the DIY route
- Jolter 2y ago
  Not even limited to general purpose ones, there are existing tailor made databases for observability. Maybe somewhere on that page, they explain why this one is better.
binaryturtle 2y ago
Isn't that something that should be posted April 1? I'm really not sure if the author is proud about the fact that his project has so many dependencies. Is that something modern coders aim for these days? I usually try to achieve the exact opposite in my projects.
- ramon156 2y ago
  Its just really tongue-in-cheek about everything which makes this article more fun to read imo
- griomnib 2y ago
  April 20th as you’d have to be high as hell to think this was a good idea.
- bdcravens 2y ago
  Even developers with "few" dependencies often lean on projects (languages, frameworks, etc) where there are hundreds of dependencies.
- joquarky 2y ago
  I also prefer to minimize dependencies, and it feels like this is why I can't find work.
  - johnisgood 2y ago
    So do I. I am writing a Perl script right now, and I could either use a non-core dependency, or implement my own. I went with my own. It is only a few lines of code. It works without the need to cpan i the module.
- hulitu 2y ago
  > Is that something modern coders aim for these days? Yes. No dependencies is so 80's. Just run an ldd on your commonly used programs.
eknkc 2y ago
Is the dependency count supposed to be impressive?
- jjtheblunt 2y ago
  i think the implication is that it's precarious...how does one know all are bug free, for example?
  - thinkharderdev 2y ago
    Is it? You know for a fact that there are bugs in some of your dependencies. But how many bugs would the code you wrote from scratch instead of adding a dependency have?
    - jjtheblunt 2y ago
      Are you asking if it is the implication, or if it is the implication that which is implied is true?
      - thinkharderdev 2y ago
        Asking if the implication is true that having more dependencies is on net bad for security for a complex system. The alternative being reimplementing whatever you would otherwise pull in a third-party dependency for. On the one hand, you reduce the attack surface in your supply chain. On the other hand you run the risk of introducing security bugs in the code you write that is outside your domain of expertise. It's not at all clear to me which one would be more important.
- speed_spread 2y ago
  Past a number of dependencies, actually getting anything to build deterministically, run reliably and then not get 0wnd to bits becomes an actual challenge, which many enthusiastic developers have a masochistic kink for. The thrill of complexity is real.
ergonaught 2y ago
While acknowledging one does not "have to" have so many dependencies, the prevalence of this npm-esque type of practice is one of the two things that destroyed all of my interest in Rust.
- kibwen 2y ago
  How many transitive dependencies is the right number for a database?
  - jandrewrogers 2y ago
    Honestly, current best practice puts that number right around zero, which you see for ambitious implementations. A non-obvious issue is that database engines have peculiar requirements for how libraries are designed and implemented which almost no conventional library satisfies. To make matters worse, two different database implementations may have different requirements in this regard, so you can't even share libraries between databases. There are no black boxes in good database engines.
    - almostdeadguy 2y ago
      Compression libraries, OpenSSL, ICU, etc. are all common dependencies for databases. Looking at the dependencies list (https://gist.github.com/tisonkun/06550d2dcd9cf6551887ee6305e92edc https://gist.github.com/tisonkun/06550d2dcd9cf6551887ee6305e...) I see plenty of reasonable things like: * Base64/checksum/compression encoding libraries * Encryption/hash libraries * Platform-specific bindings (likely conditional dependencies) * Bit hacking/casting/zero-copy libraries like bytemuck, zerocopy, zero-vec, etc. * "Small"/stack allocated data structure libraries (smallvec, tinystr, etc.) * Unicode libraries There are certainly things that would add bloat too, but I think it's silly to pretend like everything here is something a database engine would need custom implementations of.
      - jandrewrogers 2y ago
        I think you'd be surprised how many of these things are custom implementations in databases. The main motivation is performance. Databases tend to have detailed and well-specified constraints on each use case for data structures and algorithms that can be used to codegen narrowly optimized implementations. You can do significantly better than generic library codecs or data structures in most cases, those implementations lack the context and metaprogramming hooks to make it feasible. Combine this with the challenge of implementations being async, non-allocating, compatible with explicitly paged memory, etc and it generally becomes worth the effort. You'll find more libraries used at the periphery for integration and compatibility where it matters less but not in the core.
estebank 2y ago
Yet another thread where people go "Dependency number too big! Rust bad!" with the level of nuance of my dogs discussing dinner. The full list is linked in the article https://gist.github.com/tisonkun/06550d2dcd9cf6551887ee6305e92edc https://gist.github.com/tisonkun/06550d2dcd9cf6551887ee6305e... There isn't a single thing there that seems iffy to me. Rust projects split themselves into as small of a crate as possible to 1) ease their own development, 2) improve compile times to make their compilation trivially parallelizable, and 3) allow for reuse. Because of this, you can easily end up with a dozen crates all written by the same group of people, meant to be used together. If a project is a single big crate, or a dozen small crates, you're on the exact same situation. If you wouldn't audit the small crates because they are a lot, you wouldn't audit the big crate thoroughly either. But what about transitive dependencies? Similar thing: if you have a crate to check for the terminal width, I prefer to take the existing small crate than copy paste its code. I can do the latter, but then you end up with effectively a vendored library in your code that no tool can know about to warn you when a security vulnerability has happened.
- pessimizer 2y ago
  Agreed, the dependency list looks extremely boring and completely auditable to me. The dependencies are modular, not diffuse. I think people saw the title, and got triggered into hate. When actually, this seems author-submitted, and they were probably just trying to be humble about their accomplishment. It's not even the title of the article.
  - tison 2y ago
    > they were probably just trying to be humble about their accomplishment Thanks for your reply. To be honest, I simply recognize that depending on open-source software a trivial choice. Any non-trivial Rust project can pull in hundreds of dependencies and even when you audit distributed system written in C++/Java, it's a common case. For example, Cloudflare's pingora has more than 400 dependencies. Other databases written in Rust, e.g., Databend and Materialize, have more than 1000 dependencies in the lockfile. TiKV has more than 700 dependencies. People seem to jump in the debt of the number of dependencies or blame why you close the source code, ignoring the purpose that I'd like to show how you can organically contribute to the open-source ecosystem during your DAYJOB, and this is a way to write open-source code sustainable.
moi2388 2y ago
“ With a team of three experienced developers, we have implemented ScopeDB from scratch” “ with 100 direct dependencies and 647 dependencies in total” Next up: watch me build numpy from scratch with only 150 dependencies, one of which numpy.
- remram 2y ago
  You're not wrong, they depend on an external SQL database, which they access with sqlx.
  - tison 2y ago
    In the linked article below, we talked about "If RDS has already been used, why is another database needed?" and "Why RDS?" Briefly, you need to manage metadata for the database. You can write your own raft based solution or leverage existing software like etcd or zookeeper that may not "a relational database". Now you need to deploy them with EBS and reimplement data replication + multi AZ fault tolerance, and it's likely still worse performance than RDS because first-class RDS can typically use internal storage API and advanced hardware. Such a scenario is not software driven. https://flex-ninja.medium.com/from-shared-nothing-to-shared-disk-build-a-fully-flexible-data-system-on-cloud-services-31538a356db2 https://flex-ninja.medium.com/from-shared-nothing-to-shared-...
EVa5I7bHFq9mnYK 2y ago
57 of which written by DPRK Koding Forces, waiting for the right moment to push a glorious update, striking at the heart of The Biggest Enemy.
carlos-menezes 2y ago
100 direct dependencies is insane.
flufluflufluffy 2y ago
I read the title thinking it was a joke, and after reading the article, I still can’t tell if it is or not.
thadt 2y ago
"An absolutely outrageous number of dependencies! What a bunch of wankers." I comment, in a Chromium[1] tab, running on my Ubuntu[2] box. [1] https://github.com/chromium/chromium/blob/main/.gitmodules https://github.com/chromium/chromium/blob/main/.gitmodules [2] https://releases.ubuntu.com/24.04/ubuntu-24.04.1-desktop-amd64.manifest https://releases.ubuntu.com/24.04/ubuntu-24.04.1-desktop-amd...
synergy20 2y ago
so,npm hell,or pip hell again? to be fair, python pkg dependency are fine to me,there might be a lot of pip pkgs still,but not a few hundreds like npm and cargo normally pulls in. golang also has a reasonable amount of dependencies. npm and cargo dependencies are just scary due to the huge number.
- eximius 2y ago
  NPM and pip hell come about for several reasons, one of the biggest being that package versions are global. In rust, you can project A can use dependencies B and C which can both depend on different versions of D. Cargo/crates generally also solve some of the other metadata problems Python has. This means the developer experience is _significantly_ improved, at a potential cost of larger binaries. In practice, projects seem to have sufficiently liberal bounds that duplication isn't an issue.
kpcyrd 2y ago
The title of the submission is somewhat bait, unfortunately the Cargo.lock doesn't seem to be public. Since my current Rust side-project also has some kind of database (along with, well, a p2p system) and also totals 454 dependencies, I've decided to do a breakdown of my dependency graph (also because I was curious myself): - 85 are related to gix (a Rust reimplementation of git, 53 of those are gix itself, that project is unfortunately infamous for splitting things into crates that probably should've been modules) - 91 are related to pgp and all the complexity it involves (aes with various cipher modes, des, dsa, ecdsa, ed25519, p256, p384, p521, rsa, sha3, sha2, sha1, md5, blowfish, camellia, cast5, ripemd, pkcs8, pkcs1, pem, sec1, ...) - 71 are related to http/irc/tokio (this includes a memory-safe tls implementation, an http stack like percent-encoding, mime, chunked encoding, ...) - 26 are related to the winapi (which I don't use myself, but are still part of the resolved dependency graph) - 8 are related to web assembly (unused when compiling for Linux) - 2 are relatd to android (also unused when compiling for Linux) In some ways this is a reminder of how much complexity we're building on top of for the sake of compatibility. Also keep in mind "reviewing 100 lines of code in 1 library" and "reviewing 100 lines of code split into 2 libraries" is still pretty much the same amount of code (if any of us actually reviewed all their dependencies). You might even have a better time reviewing the sha2 crate vs the entirety of libcrypto.so, if that's all you needed. My project has been around for (almost) two years, I scanned every commit for vulnerable dependencies using this command: for commit in $(git log --all --pretty='%H'); do git show "$commit":Cargo.lock > Cargo.lock && cargo audit -n --json | jq -r '.vulnerabilities.list[] | (.advisory.id + " - " + .package.name)'; done | sort | uniq I got a total of 25 advisories (basically what you would be exposed to if you ran all binaries from every single commit simultaneously today). Here's the list: RUSTSEC-2020-0071 - time RUSTSEC-2023-0018 - remove_dir_all RUSTSEC-2023-0034 - h2 RUSTSEC-2023-0038 - sequoia-openpgp RUSTSEC-2023-0039 - buffered-reader RUSTSEC-2023-0052 - webpki RUSTSEC-2023-0053 - rustls-webpki RUSTSEC-2023-0071 - rsa RUSTSEC-2024-0003 - h2 RUSTSEC-2024-0006 - shlex RUSTSEC-2024-0019 - mio RUSTSEC-2024-0332 - h2 RUSTSEC-2024-0336 - rustls RUSTSEC-2024-0345 - sequoia-openpgp RUSTSEC-2024-0348 - gix-index RUSTSEC-2024-0349 - gix-worktree RUSTSEC-2024-0350 - gix-fs RUSTSEC-2024-0351 - gix-ref RUSTSEC-2024-0352 - gix-index RUSTSEC-2024-0353 - gix-worktree RUSTSEC-2024-0355 - gix-path RUSTSEC-2024-0367 - gix-path RUSTSEC-2024-0371 - gix-path RUSTSEC-2024-0373 - quinn-proto RUSTSEC-2024-0421 - idna I guess I'm doing fine. Keep in mind, the binary is fully self-contained, there is no "look, my program has zero dependencies, but I need to ship an entire implementation of the gnu operating system along with it".
dabinat 2y ago
I was hoping this would be a discussion of Rust build times and how they optimized them with that number of dependencies. But I think it’s easy for people to criticize dependencies from afar without understanding what they’re used for. I’m sure the dependencies in my projects would look strange to others - for example, I use three HTTP libraries: one for 95% of cases and the others for very specific use-cases where I need control at a low level. But without that context it might seem excessive.
robertclaus 2y ago
I'm having a real crisis trying to decide whether this system should be called a database or not. It's a system for managing data, so obviously it is.. but by that loose interpretation any CRUD webserver would count too.
1vuio0pswjnm7 2y ago
At first I thought this was sarcasm. 647 points of failure.
stuhood 2y ago
When it comes to understanding the risks involved with having this many dependencies, one thing that folks might not understand is that Rust's support for dependency resolution and lock files is fantastic. Tools like `cargo audit` can tell you statically based on the lockfile which dependencies have security vulnerabilities reported against them (but you have to run it!). And Github's https://github.com/dependabot/ https://github.com/dependabot/ will do that same thing automatically, just based on the existence of the lockfile in your repo (and will also open PRs to bump deps for you). And as mentioned elsewhere: Cargo's dependency resolver supports providing multiple versions of a dep in different dependency subgraphs, which all but eliminates the "dependency hell" that folks expect from ecosystems like Python or the JVM. Two copies of a dep at different versions? Totally fine.
- Threadbare 2y ago
  Doesn't node npm also do similar?
  - stuhood 2y ago
    Yes. AFAIK, it evolved over time across 3+ package managers (`npm`, `yarn`, `pnpm`, etc), but the current state of that ecosystem is similar (including the behavior of dependabot).
  - robertlagrant 2y ago
    Python's Poetry has poetry audit as well, and there are third-party tools such as Safety (Python), Nancy (Golang), etc. Lots of languages have something like this.
    - stuhood 2y ago
      They support lockfiles and tools like `audit`, yes. But they do not support having multiple versions of a dependency. Tools based on loading libraries from a *PATH (Go, Python, JVM) usually do so by grabbing the first one that they encounter that contains the appropriate symbols. That is incompatible with having multiple versions of a package. On the other hand, Rust and node.js support this -- each in their own way. In Rust, artifact names are transparently suffixed with a hash to prevent collisions. And in node.js, almost all symbol lookups are accomplished with relative filesystem paths.
mlok 2y ago
Related : "Build a Database in 3000 Lines with 0 Dependencies" https://news.ycombinator.com/item?id=42725163 https://news.ycombinator.com/item?id=42725163

130 points by tison 2y ago

I automatically don't want to use this database because the number of third party dependencies are an unfixable, never-ending source of security vulnerabilities.

bityard 2y ago
Sometimes I'm pretty sure people upvote stories just to see what happens in the comments.
- Idiot211 2y ago
  Guilty as charged. To steal a phrase from Reddit, "the true LPT is in the comments" The true insightful discussion comes in the comments.
  - callamdelaney 2y ago
    LPT?
    - orion138 2y ago
      I believe it means Life Pro Tip.
- airstrike 2y ago
  Guilty as charged.
arccy 2y ago
yeah, rust copied the dumpster fire that was npm, i shudder to think of the future of supply chain security when people say rewrite it in rust.
- norman784 2y ago
  What would a better model to manage dependencies in your opinion? I do like that is easy to add dependencies, but also don't like that a simple hello world Axum app IIRC is around 150 dependencies.
  - reaperducer 2y ago
    You don't have to have a solution to recognize that there is a problem.
    - kibwen 2y ago
      This is both right and wrong in a pernicious way. When pointing out a problem, you don't necessarily need to provide a better solution. However, if you refrain from providing a better solution, you are still implicitly asserting that there exists some better solution. So then it's possible to counter that with: a better solution may not exist. If you think a better solution does exist, then the burden of proof is on you to point out an existing solution that does better, or to otherwise establish that some better solution must exist. Rust could very well be at a global optimum for the problems it's trying to solve. Sometimes tradeoffs are just inevitable.
      - arccy 2y ago
        it may be that rust tried to solve for the wrong problems, so while it may be at the global optimum, the foundation is just broken. that said, design choices like a flat package namespace are inexcusable. even npm started to move away from it.
      - jpc0 2y ago
        I would say, evaluate how much work it would take to build it yourself, and if that is larger than your scope allows ask yourself some serious questions about whether the solution you came up with is overly complex. Some problems are hard to solve. But not all of them are. An example, and this is an observation. Where can I grab a library that just parses parses HTTP 1.0, HTTP 1.1 and HTTP 2.0 messages. Not a HTTP framework, something along the lines of httpparse. I pass it a buffer of bytes and out the other end pops a Result<error, HTTPResponse>. Sure there are hard problems to solve there but they are API design problems. I don't need tokio or some sort of web abstraction. If I want to use HTTP as a transport over carrier pidgeon I want to be able to do that with said library. Doesn't exactly exist though and because everyone just pulls in Tokio( I do mean the entire Tower stack or whatever it's called these says) nobody even notices the issue. And every single HTTP server rewrites that functionality with slightly different edge cases and bugs. That's basic internet infrastructure right there and we can't get a conical library for that yet you are arguing that 3 different implementations of the aame hash function pulled into the same project is viable?
wslh 2y ago
Nowadays this applies to everything that depends on modules that depend on more modules (e.g. NodeJS).
rectang 2y ago
Yes, the amount of effort it takes to audit dependencies scales roughly linearly, so unless you're going to blindly install them, choosing to use a project with so many dependencies means taking on a tremendous amount of ongoing work.
- estebank 2y ago
  > the amount of effort it takes to audit dependencies scales roughly linearly With the lines of code, not the number of dependencies. 10 dependencies of 100 lines of code are arguably easier, but certainly not harder than a single dependency of 1000 lines of code.
  - rectang 2y ago
    I should clarify that I mean auditing dependency-publisher authentication, rather than full code review. This returns us to status quo ante, back before supply chain attacks were something we worried about. Bugs and such from dependencies are an annoyance but a manageable problem. Supply chain attacks after publisher account compromise are catastrophic and are not manageable.
    - estebank 2y ago
      I see, I have a different mental model for what auditing a dependency means. Auditing is "review the code and release processes of my dependency". In my mind what you describe would be "validating my Software Bill of Materials". It doesn't mean that either of us is wrong on what we call auditing, it just explains why sometimes we end up talking past each other in these conversations.
    - marcosdumay 2y ago
      > auditing dependency-publisher authentication What does this mean? It means you'll trust the random people pushing code to cargo if you can prove they indeed are the random people they claim to be?
      - 2y ago

dboreham 2y ago

Poster boy for all that's wrong with modern software modularity.

Deukhoofd 2y ago

I really chuckled about how the blog post opens with how great Rusts open-source ecosystem is, and ends with an "anyway, we made our software private and proprietary"

bdcravens 2y ago
Isn't that pretty much the modern stack? Open source language, framework, and libraries, and proprietary end product?
- goodpoint 2y ago
  And then tech companies fire engineers while making record profits.
bbkane 2y ago
That's technically correct, but they listed several ways they contribute back to the OSS ecosystem: PRs, issues, creating new libraries... This comment makes it seem like all this company does is take, which feels unfair to me
- PittleyDunkin 2y ago
  >This comment makes it seem like all this company does is take, which feels unfair to me Profit isn't far removed from theft, so maybe this shouldn't feel so unfair.
  - bbkane 2y ago
    > Profit isn't far removed from theft I definitely think there are unethical ways to profit - capitalism needs to be regulated for the good of the consumer/ecosystem/society. However, I don't believe that a blanket comparison of any type of profit to theft can be useful or correct. > so maybe this shouldn't feel so unfair Do you think this company is unethical for writing closed source software and trying to sell it?

etaioinshrdlu 2y ago

My main question is why observability data needs (or benefits from) a tailor-made database instead of a general purpose one. In 2025, anyone working on observability who told me they have to build their own database, I would be very suspicious!

tison 2y ago
Datadog always builds their own event store: https://www.datadoghq.com/blog/engineering/introducing-husky/ https://www.datadoghq.com/blog/engineering/introducing-husky... It may not be named "database" but actually take the place of a database. Observability vendors will try to store logs with ElasticSearch and later find it over expensive and has weak support for archiving cold data. Data Warehouse solution requires a complex ETL pipeline and can be awkward when handling log data (semi-structured data). That said, if you're building an observability solution for a single company, I'd totally agree to start with single node PG with backup, and only consider other solution when data and query workload grow.
- jcgrillo 2y ago
  In 2025 I'd consider starting with clickhouse instead, if you're going the DIY route
Jolter 2y ago
Not even limited to general purpose ones, there are existing tailor made databases for observability. Maybe somewhere on that page, they explain why this one is better.

binaryturtle 2y ago

Isn't that something that should be posted April 1? I'm really not sure if the author is proud about the fact that his project has so many dependencies. Is that something modern coders aim for these days? I usually try to achieve the exact opposite in my projects.

ramon156 2y ago
Its just really tongue-in-cheek about everything which makes this article more fun to read imo
griomnib 2y ago
April 20th as you’d have to be high as hell to think this was a good idea.
bdcravens 2y ago
Even developers with "few" dependencies often lean on projects (languages, frameworks, etc) where there are hundreds of dependencies.
joquarky 2y ago
I also prefer to minimize dependencies, and it feels like this is why I can't find work.
- johnisgood 2y ago
  So do I. I am writing a Perl script right now, and I could either use a non-core dependency, or implement my own. I went with my own. It is only a few lines of code. It works without the need to cpan i the module.
hulitu 2y ago
> Is that something modern coders aim for these days? Yes. No dependencies is so 80's. Just run an ldd on your commonly used programs.

eknkc 2y ago

Is the dependency count supposed to be impressive?

jjtheblunt 2y ago
i think the implication is that it's precarious...how does one know all are bug free, for example?
- thinkharderdev 2y ago
  Is it? You know for a fact that there are bugs in some of your dependencies. But how many bugs would the code you wrote from scratch instead of adding a dependency have?
  - jjtheblunt 2y ago
    Are you asking if it is the implication, or if it is the implication that which is implied is true?
    - thinkharderdev 2y ago
      Asking if the implication is true that having more dependencies is on net bad for security for a complex system. The alternative being reimplementing whatever you would otherwise pull in a third-party dependency for. On the one hand, you reduce the attack surface in your supply chain. On the other hand you run the risk of introducing security bugs in the code you write that is outside your domain of expertise. It's not at all clear to me which one would be more important.
speed_spread 2y ago
Past a number of dependencies, actually getting anything to build deterministically, run reliably and then not get 0wnd to bits becomes an actual challenge, which many enthusiastic developers have a masochistic kink for. The thrill of complexity is real.

ergonaught 2y ago

While acknowledging one does not "have to" have so many dependencies, the prevalence of this npm-esque type of practice is one of the two things that destroyed all of my interest in Rust.

kibwen 2y ago
How many transitive dependencies is the right number for a database?
- jandrewrogers 2y ago
  Honestly, current best practice puts that number right around zero, which you see for ambitious implementations. A non-obvious issue is that database engines have peculiar requirements for how libraries are designed and implemented which almost no conventional library satisfies. To make matters worse, two different database implementations may have different requirements in this regard, so you can't even share libraries between databases. There are no black boxes in good database engines.
  - almostdeadguy 2y ago
    Compression libraries, OpenSSL, ICU, etc. are all common dependencies for databases. Looking at the dependencies list (https://gist.github.com/tisonkun/06550d2dcd9cf6551887ee6305e92edc https://gist.github.com/tisonkun/06550d2dcd9cf6551887ee6305e...) I see plenty of reasonable things like: * Base64/checksum/compression encoding libraries * Encryption/hash libraries * Platform-specific bindings (likely conditional dependencies) * Bit hacking/casting/zero-copy libraries like bytemuck, zerocopy, zero-vec, etc. * "Small"/stack allocated data structure libraries (smallvec, tinystr, etc.) * Unicode libraries There are certainly things that would add bloat too, but I think it's silly to pretend like everything here is something a database engine would need custom implementations of.
    - jandrewrogers 2y ago
      I think you'd be surprised how many of these things are custom implementations in databases. The main motivation is performance. Databases tend to have detailed and well-specified constraints on each use case for data structures and algorithms that can be used to codegen narrowly optimized implementations. You can do significantly better than generic library codecs or data structures in most cases, those implementations lack the context and metaprogramming hooks to make it feasible. Combine this with the challenge of implementations being async, non-allocating, compatible with explicitly paged memory, etc and it generally becomes worth the effort. You'll find more libraries used at the periphery for integration and compatibility where it matters less but not in the core.

estebank 2y ago

Yet another thread where people go "Dependency number too big! Rust bad!" with the level of nuance of my dogs discussing dinner. The full list is linked in the article https://gist.github.com/tisonkun/06550d2dcd9cf6551887ee6305e92edc https://gist.github.com/tisonkun/06550d2dcd9cf6551887ee6305e... There isn't a single thing there that seems iffy to me. Rust projects split themselves into as small of a crate as possible to 1) ease their own development, 2) improve compile times to make their compilation trivially parallelizable, and 3) allow for reuse. Because of this, you can easily end up with a dozen crates all written by the same group of people, meant to be used together. If a project is a single big crate, or a dozen small crates, you're on the exact same situation. If you wouldn't audit the small crates because they are a lot, you wouldn't audit the big crate thoroughly either. But what about transitive dependencies? Similar thing: if you have a crate to check for the terminal width, I prefer to take the existing small crate than copy paste its code. I can do the latter, but then you end up with effectively a vendored library in your code that no tool can know about to warn you when a security vulnerability has happened.

pessimizer 2y ago
Agreed, the dependency list looks extremely boring and completely auditable to me. The dependencies are modular, not diffuse. I think people saw the title, and got triggered into hate. When actually, this seems author-submitted, and they were probably just trying to be humble about their accomplishment. It's not even the title of the article.
- tison 2y ago
  > they were probably just trying to be humble about their accomplishment Thanks for your reply. To be honest, I simply recognize that depending on open-source software a trivial choice. Any non-trivial Rust project can pull in hundreds of dependencies and even when you audit distributed system written in C++/Java, it's a common case. For example, Cloudflare's pingora has more than 400 dependencies. Other databases written in Rust, e.g., Databend and Materialize, have more than 1000 dependencies in the lockfile. TiKV has more than 700 dependencies. People seem to jump in the debt of the number of dependencies or blame why you close the source code, ignoring the purpose that I'd like to show how you can organically contribute to the open-source ecosystem during your DAYJOB, and this is a way to write open-source code sustainable.

moi2388 2y ago

“ With a team of three experienced developers, we have implemented ScopeDB from scratch” “ with 100 direct dependencies and 647 dependencies in total” Next up: watch me build numpy from scratch with only 150 dependencies, one of which numpy.

remram 2y ago
You're not wrong, they depend on an external SQL database, which they access with sqlx.
- tison 2y ago
  In the linked article below, we talked about "If RDS has already been used, why is another database needed?" and "Why RDS?" Briefly, you need to manage metadata for the database. You can write your own raft based solution or leverage existing software like etcd or zookeeper that may not "a relational database". Now you need to deploy them with EBS and reimplement data replication + multi AZ fault tolerance, and it's likely still worse performance than RDS because first-class RDS can typically use internal storage API and advanced hardware. Such a scenario is not software driven. https://flex-ninja.medium.com/from-shared-nothing-to-shared-disk-build-a-fully-flexible-data-system-on-cloud-services-31538a356db2 https://flex-ninja.medium.com/from-shared-nothing-to-shared-...

EVa5I7bHFq9mnYK 2y ago

57 of which written by DPRK Koding Forces, waiting for the right moment to push a glorious update, striking at the heart of The Biggest Enemy.

carlos-menezes 2y ago

100 direct dependencies is insane.

flufluflufluffy 2y ago

I read the title thinking it was a joke, and after reading the article, I still can’t tell if it is or not.

thadt 2y ago

"An absolutely outrageous number of dependencies! What a bunch of wankers." I comment, in a Chromium[1] tab, running on my Ubuntu[2] box. [1] https://github.com/chromium/chromium/blob/main/.gitmodules https://github.com/chromium/chromium/blob/main/.gitmodules [2] https://releases.ubuntu.com/24.04/ubuntu-24.04.1-desktop-amd64.manifest https://releases.ubuntu.com/24.04/ubuntu-24.04.1-desktop-amd...

synergy20 2y ago

so,npm hell,or pip hell again? to be fair, python pkg dependency are fine to me,there might be a lot of pip pkgs still,but not a few hundreds like npm and cargo normally pulls in. golang also has a reasonable amount of dependencies. npm and cargo dependencies are just scary due to the huge number.

eximius 2y ago
NPM and pip hell come about for several reasons, one of the biggest being that package versions are global. In rust, you can project A can use dependencies B and C which can both depend on different versions of D. Cargo/crates generally also solve some of the other metadata problems Python has. This means the developer experience is _significantly_ improved, at a potential cost of larger binaries. In practice, projects seem to have sufficiently liberal bounds that duplication isn't an issue.

kpcyrd 2y ago

The title of the submission is somewhat bait, unfortunately the Cargo.lock doesn't seem to be public. Since my current Rust side-project also has some kind of database (along with, well, a p2p system) and also totals 454 dependencies, I've decided to do a breakdown of my dependency graph (also because I was curious myself): - 85 are related to gix (a Rust reimplementation of git, 53 of those are gix itself, that project is unfortunately infamous for splitting things into crates that probably should've been modules) - 91 are related to pgp and all the complexity it involves (aes with various cipher modes, des, dsa, ecdsa, ed25519, p256, p384, p521, rsa, sha3, sha2, sha1, md5, blowfish, camellia, cast5, ripemd, pkcs8, pkcs1, pem, sec1, ...) - 71 are related to http/irc/tokio (this includes a memory-safe tls implementation, an http stack like percent-encoding, mime, chunked encoding, ...) - 26 are related to the winapi (which I don't use myself, but are still part of the resolved dependency graph) - 8 are related to web assembly (unused when compiling for Linux) - 2 are relatd to android (also unused when compiling for Linux) In some ways this is a reminder of how much complexity we're building on top of for the sake of compatibility. Also keep in mind "reviewing 100 lines of code in 1 library" and "reviewing 100 lines of code split into 2 libraries" is still pretty much the same amount of code (if any of us actually reviewed all their dependencies). You might even have a better time reviewing the sha2 crate vs the entirety of libcrypto.so, if that's all you needed. My project has been around for (almost) two years, I scanned every commit for vulnerable dependencies using this command: for commit in $(git log --all --pretty='%H'); do git show "$commit":Cargo.lock > Cargo.lock && cargo audit -n --json | jq -r '.vulnerabilities.list[] | (.advisory.id + " - " + .package.name)'; done | sort | uniq I got a total of 25 advisories (basically what you would be exposed to if you ran all binaries from every single commit simultaneously today). Here's the list: RUSTSEC-2020-0071 - time RUSTSEC-2023-0018 - remove_dir_all RUSTSEC-2023-0034 - h2 RUSTSEC-2023-0038 - sequoia-openpgp RUSTSEC-2023-0039 - buffered-reader RUSTSEC-2023-0052 - webpki RUSTSEC-2023-0053 - rustls-webpki RUSTSEC-2023-0071 - rsa RUSTSEC-2024-0003 - h2 RUSTSEC-2024-0006 - shlex RUSTSEC-2024-0019 - mio RUSTSEC-2024-0332 - h2 RUSTSEC-2024-0336 - rustls RUSTSEC-2024-0345 - sequoia-openpgp RUSTSEC-2024-0348 - gix-index RUSTSEC-2024-0349 - gix-worktree RUSTSEC-2024-0350 - gix-fs RUSTSEC-2024-0351 - gix-ref RUSTSEC-2024-0352 - gix-index RUSTSEC-2024-0353 - gix-worktree RUSTSEC-2024-0355 - gix-path RUSTSEC-2024-0367 - gix-path RUSTSEC-2024-0371 - gix-path RUSTSEC-2024-0373 - quinn-proto RUSTSEC-2024-0421 - idna I guess I'm doing fine. Keep in mind, the binary is fully self-contained, there is no "look, my program has zero dependencies, but I need to ship an entire implementation of the gnu operating system along with it".

dabinat 2y ago

I was hoping this would be a discussion of Rust build times and how they optimized them with that number of dependencies. But I think it’s easy for people to criticize dependencies from afar without understanding what they’re used for. I’m sure the dependencies in my projects would look strange to others - for example, I use three HTTP libraries: one for 95% of cases and the others for very specific use-cases where I need control at a low level. But without that context it might seem excessive.

robertclaus 2y ago

I'm having a real crisis trying to decide whether this system should be called a database or not. It's a system for managing data, so obviously it is.. but by that loose interpretation any CRUD webserver would count too.

1vuio0pswjnm7 2y ago

At first I thought this was sarcasm. 647 points of failure.

stuhood 2y ago

When it comes to understanding the risks involved with having this many dependencies, one thing that folks might not understand is that Rust's support for dependency resolution and lock files is fantastic. Tools like `cargo audit` can tell you statically based on the lockfile which dependencies have security vulnerabilities reported against them (but you have to run it!). And Github's https://github.com/dependabot/ https://github.com/dependabot/ will do that same thing automatically, just based on the existence of the lockfile in your repo (and will also open PRs to bump deps for you). And as mentioned elsewhere: Cargo's dependency resolver supports providing multiple versions of a dep in different dependency subgraphs, which all but eliminates the "dependency hell" that folks expect from ecosystems like Python or the JVM. Two copies of a dep at different versions? Totally fine.

Threadbare 2y ago
Doesn't node npm also do similar?
- stuhood 2y ago
  Yes. AFAIK, it evolved over time across 3+ package managers (`npm`, `yarn`, `pnpm`, etc), but the current state of that ecosystem is similar (including the behavior of dependabot).
- robertlagrant 2y ago
  Python's Poetry has poetry audit as well, and there are third-party tools such as Safety (Python), Nancy (Golang), etc. Lots of languages have something like this.
  - stuhood 2y ago
    They support lockfiles and tools like `audit`, yes. But they do not support having multiple versions of a dependency. Tools based on loading libraries from a *PATH (Go, Python, JVM) usually do so by grabbing the first one that they encounter that contains the appropriate symbols. That is incompatible with having multiple versions of a package. On the other hand, Rust and node.js support this -- each in their own way. In Rust, artifact names are transparently suffixed with a hash to prevent collisions. And in node.js, almost all symbol lookups are accomplished with relative filesystem paths.

mlok 2y ago

Related : "Build a Database in 3000 Lines with 0 Dependencies" https://news.ycombinator.com/item?id=42725163 https://news.ycombinator.com/item?id=42725163