10 ms·

The fact is that dependency jungle is the prevalent way to get shit done these days. The best the runtime can do is embrace it, make it as performant and safe a

The fact is that dependency jungle is the prevalent way to get shit done these days. The best the runtime can do is embrace it, make it as performant and safe as possible and try to support minimum-dependency projects by having a broad std library. Also I am no expert, but I think file-watchers are definitely not simple at all, especially if they are multi-platform.

dist1ll 2y ago
That's the usual response I get when I bring this issue up. "file watching is actually very complicated" or "if you avoided deps, you'd just reimplement millions of loc yourself. Forgive me if I'm making a very bold claim, but I think cross-platform file watching should not require this much code. It's 32x larger than the Linux memory management subsystem.
- selfmodruntime 2y ago
  Eh. The standard library is also a gigantic dependency written entirely by volunteers.
  - dist1ll 2y ago
    I don't have a problem with dependencies in principle. There's a good reason for standard libraries to contain a decent amount of code. It is a vector for supply chain attacks, but I also have a lot of trust in the Rust maintainers. The Rust standard library is exceptionally well-written from what I've seen, so I'm not too worried about it. FWIW I checked out the nightly toolchain, and it looks like the stdlib is less than 400k SLoC. So literally 10x smaller.
  - DanielHB 2y ago
    I am not a Rust expert but the thing with the standard libraries is that it only has peer dependencies with itself and they are all synced to the same version. Meaning if you only use the std lib you: 1) Will never include two different versions of the same peer dependency because of incompatible version requirements. 2) Will usually not have two dependencies relying on two different peer-dependencies that do the same thing. This can still happen for deprecated std lib features, but tends to be a much lesser issue. These two issues are usually the ones that cause dependency size explosion in projects.
    - deleted 2y ago
      [deleted]
- DanielHB 2y ago
  I think the issue of file-watching is that the libs usually support multiple implementations (with different tradeoffs and with multiple fallbacks) for file-watching with a lot of them being platform specific.
- joatmon-snoo 2y ago
  Good file watching that provides flexible primitives absolutely requires: - ok, a single ext4 file inode changes, and its filename matches my hardcoded string - oh, you don’t want to match against just changes to “package.json” but you want to match against a regex? voila, now you need a regex engine - what about handling a directory rename? should that trigger matches on all files in the renamed directory? - should the file watcher be triggered once per file, or just every 5ms? turns out this depends on your use case - how do symlinks fit into this story? - let’s say i want to handle once every 5ms- how do i actually wait for 5ms? do i yield the thread? do i allow other async contexts to execute while i’m waiting? how do those contexts know when to execute and when to yield back to me? now you have an async runtime with timers - how does buffering work? are there limits on how many file change events can be buffered? do i dynamically allocate more memory as more file changes get buffered? now you need a vector/arraylist implementation And this is before you look at what this looks like on different platforms, or if you want polling fallbacks. Can you do it with less dependencies? Probably, if you start making hard tradeoffs and adding even more complexity about what features you activate - but that only adds lines of code, it doesn’t remove them. What you describe is ideologically nice, but in practice it’s over-optimizing for a goal that most people don’t really care about.
  - varjag 2y ago
    Are you 100% sure all these cases are handled by cargo-watch?
    - cogman10 2y ago
      Yup. Every user facing case mentioned has a corresponding flag. The non-user facing stuff, like being cross platform, is common sense. https://crates.io/crates/cargo-watch/8.5.2 https://crates.io/crates/cargo-watch/8.5.2
  - lostmsu 2y ago
    Regex one stands out as a negative example here. Why does it have to be built-in instead of exposing a str -> bool filter lambda?
- SkiFire13 2y ago
  It's not just file watching, that would be the watchexec crate, while cargo-watch properly integrate with cargo. Moreover cargo-watch also includes: - proper CLI support, with help messages, subcommands and so on - support for reading cargo's metadata - logging - support for dotenv files - proper shell escaping support - and it seems also support for colored terminal writing. Moreover both watchexec and cargo-watch end up depending on winapi, which includes binding for a lot of windows API, some which might be needed and some which not be. This could also be worse if the offial windows crate by Microsoft was used (or maybe it's already used due to some dependency, I haven't checked), since that's gigantic.
SkiFire13 2y ago
> try to support minimum-dependency projects by having a broad std library. Since everyone depends on the standard library this will just mean everyone will depend on even more lines of code. You are decreasing the number of nominal dependencies but increasing of much code those amount to. Moreover the moment the stdlib's bundled dependency is not enough there are two problems: - it can't be changed because that would be a breaking change, so you're stuck with the old bad implementation; - you will have to use an alternative implementation in another crate, so now you're back at the starting situation except with another dependency bundled in the stdlib. Just look at the dependency situation with the python stdlib, e.g. how many versions of urllib there are.
- DanielHB 2y ago
  You do have good points as well and it depends heavily on how disciplined the std lib makers are. Go for example has a very clean and stable std lib. I posted this in some other thread: I am not a Rust expert but the thing with the standard libraries is that it only has peer dependencies with itself and they are all synced to the same version. Meaning if you only use the std lib you: 1) Will never include two different versions of the same peer dependency because of incompatible version requirements. 2) Will usually not have two dependencies relying on two different peer-dependencies that do the same thing. This can still happen for deprecated std lib features, but tends to be a much lesser issue. These two issues are usually the ones that cause dependency size explosion in projects.
  - SkiFire13 2y ago
    For 1), Cargo already take care of that if you use the same major version. Bundling dependencies in the stdlib "solves" the problem by making new major versions impossible. This means that if a bundled dependency in the stdlib is even found to have some design issue that require breaking changes to fix then you're out of luck. As you said the stdlib could deprecate the old version and add a new one, but then you're just making problem 2) worse by forcing everyone to include the old deprecated dependency too! Or you could use a third-party implementation, especially if the stdlib doesn't have the features you need, but even then you will still be including the stdlib version in your dependency graph! Ultimately IMO bundling dependencies in the stdlib just makes the problem worse over time, though it can raise awareness about how to better handle them.
chillfox 2y ago
"I think file-watchers are definitely not simple at all" I don't really know much about Rust, but I got curious and had a look at the file watching apis for windows/linux/macos and it really didn't seem that complicated. Maybe a bit fiddly, but I have a hard time imagining how it could take more than 500 lines of code. I would love to know where the hard part is if anyone knows of a good blog post or video about it.
kreyenborgi 2y ago
https://github.com/eradman/entr https://github.com/eradman/entr is Language files blank comment code ------------------------------------------------------------------------------- C 4 154 163 880 Bourne Shell 2 74 28 536 C/C++ Header 4 21 66 70 Markdown 1 21 0 37 YAML 1 0 0 14 ------------------------------------------------------------------------------- SUM: 12 270 257 1537 ------------------------------------------------------------------------------- including a well-designed CLI. entr supports BSD, Mac OS, and Linux (even WSL). So that's several platforms in <2k lines of code. By using MATHEMATICS and EXTRAPOLATION we find that non-WSL Windows file-watching must take four million minus two thousand equals calculate calculate 3998000 lines of code. Ahem. Though to be fair, cargo watch probably does more than just file-watching. (Should it? Is it worth the complexity? I guess that depends on where you land on the worse-is-better discussion.)
- lifthrasiir 2y ago
  You are comparing a bicycle and a car; while you might only need a bicycle for your daily life, they are not directly comparable. BSD, Mac OS and Linux share the same interface that approximates POSIX---so it only supports a single platform with different variants. Its CLI is not well-designed, it's just a fixed unconditional terminal sequence that even doesn't look at $TERM and its options have no long counterpart (probably because it couldn't use getopt_long which is a GNU extension). And cargo-watch actually parses the `cargo metadata` JSON output (guess what's required for parsing JSON in C) and deals with ignore patterns which are consistent in syntax (guess what's required for doing that besides from fnmatch). And I'm not even meant to say that the supposed figure of 4M LoC is all required. In fact, while the problem itself does exist, I don't think that figure is accurate at all, given the massive `windows` crate was blindly counted towards. I guess the faithful reproduction of cargo-watch without any external library will take about 20--50K lines of code in Rust and in C. But doing it in C would be much more painful and you will instead cut requirements.
0x000xca0xfe 2y ago
If you lose track of your dependencies you are just asking for supply chain attacks. And since xz we know resourceful and patient attackers are reality and not just "it might happen". Sorry but sprawling transitive micro-dependencies are not sustainable. It's convenient and many modern projects right now utilize it but they require a high-trust environment and we don't have that anymore, unfortunately.

dist1ll 2y ago

That's the usual response I get when I bring this issue up. "file watching is actually very complicated" or "if you avoided deps, you'd just reimplement millions of loc yourself. Forgive me if I'm making a very bold claim, but I think cross-platform file watching should not require this much code. It's 32x larger than the Linux memory management subsystem.

selfmodruntime 2y ago
Eh. The standard library is also a gigantic dependency written entirely by volunteers.
- dist1ll 2y ago
  I don't have a problem with dependencies in principle. There's a good reason for standard libraries to contain a decent amount of code. It is a vector for supply chain attacks, but I also have a lot of trust in the Rust maintainers. The Rust standard library is exceptionally well-written from what I've seen, so I'm not too worried about it. FWIW I checked out the nightly toolchain, and it looks like the stdlib is less than 400k SLoC. So literally 10x smaller.
- DanielHB 2y ago
  I am not a Rust expert but the thing with the standard libraries is that it only has peer dependencies with itself and they are all synced to the same version. Meaning if you only use the std lib you: 1) Will never include two different versions of the same peer dependency because of incompatible version requirements. 2) Will usually not have two dependencies relying on two different peer-dependencies that do the same thing. This can still happen for deprecated std lib features, but tends to be a much lesser issue. These two issues are usually the ones that cause dependency size explosion in projects.
  - deleted 2y ago
    [deleted]
DanielHB 2y ago
I think the issue of file-watching is that the libs usually support multiple implementations (with different tradeoffs and with multiple fallbacks) for file-watching with a lot of them being platform specific.
joatmon-snoo 2y ago
Good file watching that provides flexible primitives absolutely requires: - ok, a single ext4 file inode changes, and its filename matches my hardcoded string - oh, you don’t want to match against just changes to “package.json” but you want to match against a regex? voila, now you need a regex engine - what about handling a directory rename? should that trigger matches on all files in the renamed directory? - should the file watcher be triggered once per file, or just every 5ms? turns out this depends on your use case - how do symlinks fit into this story? - let’s say i want to handle once every 5ms- how do i actually wait for 5ms? do i yield the thread? do i allow other async contexts to execute while i’m waiting? how do those contexts know when to execute and when to yield back to me? now you have an async runtime with timers - how does buffering work? are there limits on how many file change events can be buffered? do i dynamically allocate more memory as more file changes get buffered? now you need a vector/arraylist implementation And this is before you look at what this looks like on different platforms, or if you want polling fallbacks. Can you do it with less dependencies? Probably, if you start making hard tradeoffs and adding even more complexity about what features you activate - but that only adds lines of code, it doesn’t remove them. What you describe is ideologically nice, but in practice it’s over-optimizing for a goal that most people don’t really care about.
- varjag 2y ago
  Are you 100% sure all these cases are handled by cargo-watch?
  - cogman10 2y ago
    Yup. Every user facing case mentioned has a corresponding flag. The non-user facing stuff, like being cross platform, is common sense. https://crates.io/crates/cargo-watch/8.5.2 https://crates.io/crates/cargo-watch/8.5.2
- lostmsu 2y ago
  Regex one stands out as a negative example here. Why does it have to be built-in instead of exposing a str -> bool filter lambda?
SkiFire13 2y ago
It's not just file watching, that would be the watchexec crate, while cargo-watch properly integrate with cargo. Moreover cargo-watch also includes: - proper CLI support, with help messages, subcommands and so on - support for reading cargo's metadata - logging - support for dotenv files - proper shell escaping support - and it seems also support for colored terminal writing. Moreover both watchexec and cargo-watch end up depending on winapi, which includes binding for a lot of windows API, some which might be needed and some which not be. This could also be worse if the offial windows crate by Microsoft was used (or maybe it's already used due to some dependency, I haven't checked), since that's gigantic.

SkiFire13 2y ago

> try to support minimum-dependency projects by having a broad std library. Since everyone depends on the standard library this will just mean everyone will depend on even more lines of code. You are decreasing the number of nominal dependencies but increasing of much code those amount to. Moreover the moment the stdlib's bundled dependency is not enough there are two problems: - it can't be changed because that would be a breaking change, so you're stuck with the old bad implementation; - you will have to use an alternative implementation in another crate, so now you're back at the starting situation except with another dependency bundled in the stdlib. Just look at the dependency situation with the python stdlib, e.g. how many versions of urllib there are.

DanielHB 2y ago
You do have good points as well and it depends heavily on how disciplined the std lib makers are. Go for example has a very clean and stable std lib. I posted this in some other thread: I am not a Rust expert but the thing with the standard libraries is that it only has peer dependencies with itself and they are all synced to the same version. Meaning if you only use the std lib you: 1) Will never include two different versions of the same peer dependency because of incompatible version requirements. 2) Will usually not have two dependencies relying on two different peer-dependencies that do the same thing. This can still happen for deprecated std lib features, but tends to be a much lesser issue. These two issues are usually the ones that cause dependency size explosion in projects.
- SkiFire13 2y ago
  For 1), Cargo already take care of that if you use the same major version. Bundling dependencies in the stdlib "solves" the problem by making new major versions impossible. This means that if a bundled dependency in the stdlib is even found to have some design issue that require breaking changes to fix then you're out of luck. As you said the stdlib could deprecate the old version and add a new one, but then you're just making problem 2) worse by forcing everyone to include the old deprecated dependency too! Or you could use a third-party implementation, especially if the stdlib doesn't have the features you need, but even then you will still be including the stdlib version in your dependency graph! Ultimately IMO bundling dependencies in the stdlib just makes the problem worse over time, though it can raise awareness about how to better handle them.

chillfox 2y ago

"I think file-watchers are definitely not simple at all" I don't really know much about Rust, but I got curious and had a look at the file watching apis for windows/linux/macos and it really didn't seem that complicated. Maybe a bit fiddly, but I have a hard time imagining how it could take more than 500 lines of code. I would love to know where the hard part is if anyone knows of a good blog post or video about it.

kreyenborgi 2y ago

https://github.com/eradman/entr https://github.com/eradman/entr is Language files blank comment code ------------------------------------------------------------------------------- C 4 154 163 880 Bourne Shell 2 74 28 536 C/C++ Header 4 21 66 70 Markdown 1 21 0 37 YAML 1 0 0 14 ------------------------------------------------------------------------------- SUM: 12 270 257 1537 ------------------------------------------------------------------------------- including a well-designed CLI. entr supports BSD, Mac OS, and Linux (even WSL). So that's several platforms in <2k lines of code. By using MATHEMATICS and EXTRAPOLATION we find that non-WSL Windows file-watching must take four million minus two thousand equals calculate calculate 3998000 lines of code. Ahem. Though to be fair, cargo watch probably does more than just file-watching. (Should it? Is it worth the complexity? I guess that depends on where you land on the worse-is-better discussion.)

lifthrasiir 2y ago
You are comparing a bicycle and a car; while you might only need a bicycle for your daily life, they are not directly comparable. BSD, Mac OS and Linux share the same interface that approximates POSIX---so it only supports a single platform with different variants. Its CLI is not well-designed, it's just a fixed unconditional terminal sequence that even doesn't look at $TERM and its options have no long counterpart (probably because it couldn't use getopt_long which is a GNU extension). And cargo-watch actually parses the `cargo metadata` JSON output (guess what's required for parsing JSON in C) and deals with ignore patterns which are consistent in syntax (guess what's required for doing that besides from fnmatch). And I'm not even meant to say that the supposed figure of 4M LoC is all required. In fact, while the problem itself does exist, I don't think that figure is accurate at all, given the massive `windows` crate was blindly counted towards. I guess the faithful reproduction of cargo-watch without any external library will take about 20--50K lines of code in Rust and in C. But doing it in C would be much more painful and you will instead cut requirements.

0x000xca0xfe 2y ago

If you lose track of your dependencies you are just asking for supply chain attacks. And since xz we know resourceful and patient attackers are reality and not just "it might happen". Sorry but sprawling transitive micro-dependencies are not sustainable. It's convenient and many modern projects right now utilize it but they require a high-trust environment and we don't have that anymore, unfortunately.