9 ms·

> And don’t tell me about Bytes crate—it should not be a separate crate I'd be interested to hear the author's reasoning behind this, if it does what they want

> And don’t tell me about Bytes crate—it should not be a separate crate I'd be interested to hear the author's reasoning behind this, if it does what they want then why not use it? It's small and well written, so I don't think vetting it should be a problem. The rest of the article seems quite sensible, that comment just strikes me as a little odd.

zanny 9y ago
There is a real change in mentality you have to go through if you transition from a fairly strict C/C++/ even Java background to trying out Rust. In the former languages, adding dependencies rapidly becomes a painful experience, whereas Rust does much better dependency management and automatic building than even Python (where you need a requirements file or something similar to go pull down all the deps). With Rust, you really should just use crates. The std is meant to be limited to just the most used code and that which should not change for the sake of keeping the ecosystem stable.
- ditonal 9y ago
  > even Python Off topic, but it shouldn't be better than "even Python", because Python has a really, really broken dependency system. Far more so than Java, which has Maven/Gradle which are both infinitely better than the pip/virtualenv disaster. People complain about things like shading in Maven being complicated. What they might not realize is that pip doesn't even try to address conflicting dependencies, it will just silently give you the wrong version! You ask for A==0.2 and it will give you A==0.1 if another dependency asked for A==0.1 first. And it won't even warn you even though it's straight up broken behavior. Virtualenv makes packaging annoying since it's almost vendoring but not quite. To totally understand the packaging system forces you into the world of eggs, wheels, disutils, conflicting versions, condas, etc. Sorry for tangent, just thought it was funny you would hold Python up as a standard of dependency excellence when it's probably the worst overall ecosystem of major languages.
  - nolok 9y ago
    I find it super weird how Python is still in this state while even PHP has managed to get something good going. To me, this is a clear case of trying to fix a broken system whereas starting from scratch would be much better.
  - zanny 9y ago
    My language development went from C -> C++ -> Java -> Python. So when I got there and figured out pip was a thing (or easy_install back in the day) it was a major innovation at the time. Additionally, for anyone coming from almost all compiled native languages, a native environment like Rust with better dependency management than Python (which, as you say, and in retrospect, is pretty broken) is a bit of a mind screw.
    - k__ 9y ago
      The Python devs I worked with always envied me for npm. I asked them if they don't have something similar with pip, but seems like npm is a whole different level. Cargo should even be better than npm. On the other hand I always asked myself if they couldn't simply use Nix?
      - steveklabnik 9y ago
        I can't simply use Nix; it doesn't support Windows.
        k__ 9y ago
        It doesn't run on the Linux subsystem?
        steveklabnik 9y ago
        I hear it sorta works, but nix-shell doesn't. Regardless, while it's cool and useful, it's not really actual Windows support.
- nolok 9y ago
  Haven't used Rust yet, is it kind of similar to Go where any file can simply import and then Go knows how to fetch and build when needed ? And then when you remove the calls (eg during a refactor) the compiler force you to remove the imports too, stopping the infinite bloat caused by "no one really knows if we still need this" that can be common in other language. I really liked that "dependancy as part of the language and build tool". Of course, Go still had its own dependancy issues (versionning, availability, ...) that they now work on, but that was a step in a direction I quite liked.
  - Manishearth 9y ago
    > is it kind of similar to Go where any file can simply import You declare your crate dependencies in a Cargo.toml file. Rust does proper versioning of dependencies so having a separate manifest is desirable. Within your code you declare the existence of a crate via `extern crate` and then you can use it wherever. > the compiler force you to remove the imports too it warns.
    - greenhouse_gas 9y ago
      Do you know if there are plans for rustfmt to auto-import like gofmt does? In the sense that if a crate is available (in the .toml file) and you reference it, rustfmt will automatically insert the required "import" and "use".
      - steveklabnik 9y ago
        There's been talk of it, but it hasn't been built yet. It's more likely to be a part of RLS than rustfmt.
      - killercup 9y ago
        If the compiler gives you an error with a suggestion "You probably need to add `extern crate foo;`" (and it already does in some cases), RLS (i.e., your editor plugin) will be able to automatically add it for you (soon™).
      - 9y ago
- pjmlp 9y ago
  A current problem with " should just use crates" is that cargo only works with source code dependencies, not binary libraries. So each new crate added to your project ends up increasing the overall build time, which gets exponentially bad if you happen to add a dependency to a crate than happens to have a big dependency list. I have a very basic word counting application with a GUI written in Gtk-rs, a fresh build straight out of "git clone" takes a few minutes, mainly thanks to Pango.
  - Sean1708 9y ago
    > So each new crate added to your project ends up increasing the overall build time, That's only a problem the first time you compile though, isn't it?
    - pjmlp 9y ago
      A problem that I usually don't have with C++[0], Java[1] or .NET[2], thanks to the build systems support for binary libraries. [0] - I eschew header only libraries, only if there is no technical way around them (e.g. templates). [1] - I can AOT compile with Excelsior JET, IBM J9, JamaicaVM, PTC, , Oracle Java 9 (Linux x64 for the time being), ... [2] - I can AOT compile with Mono, NGEN, .NET Native, CoreRT, IL2CPP, ...
      - Sean1708 9y ago
        I'm not saying it isn't a problem, I just wanted to know if I was correct in thinking that it is only a problem the first time you compile each crate (or rather each version of each crate, I think).
        pjmlp 9y ago
        I see. Yes, it is a problem when you do a clean build, or when you have common crates across projects, because cargo doesn't have a concept of build cache. You can try to workaround it by setting all target directories to same one via target-dir in your .cargo/config file, but there is no guarantee that the crate won't get rebuild.
- CodesInChaos 9y ago
  Every dependency can run code on every computer your project runs on. That means you have to trust its author to: 1. not be malicious 2. not write a vulnerability by accident 3. not get their computer infected, their email account hijacked, etc. 4. be wise in transferring ownership 5. not add a dependency with a license incompatible with your project All the above concerns apply recursively to the dependencies of the dependency.
mastax 9y ago
Some C & C++ programmers are averse to third-party libraries (moreso than other languages in my experience). It's a valid position, but if you really value that then perhaps Rust is not for you.
valarauca1 9y ago
I agree with the author. Bytes is basically just C++'s `std::string` (kind of don't bite my head off people who've memorized the C++17 standard). Its an ARC backed array [1]. This suggests it should be a fairly fundamental abstraction. Edit: ^^^My C++ is wrong sorry :( Really I disagree with its purpose. In its immutable, non-threaded safe form you can create the same structure by just borrowing a value. This ofc requires making your peace with the borrow checker and `Cow<'a, T>` copy on write types. By in large Bytes advertised purpose is network code. And for networking code its really only _super_ useful if your using a Packet Ring in Linux. As jemalloc will not return regularly re-allocated buffers. Really this is all performance theater. How you manage/architect your socket reads/writes will have an order of magnitude larger effect then what abstraction you _store_ those bytes in once read. [1] https://carllerche.github.io/bytes/bytes/struct.Bytes.htm https://carllerche.github.io/bytes/bytes/struct.Bytes.htm
- tatterdemalion 9y ago
  Bytes is intended for use with the tokio ecoystem where you cna't use references often because borrowing across a yield point in a future would be a lifetime error.
  - valarauca1 9y ago
    Can't `Vec<u8>` serve the same purpose?
    - tatterdemalion 9y ago
      Then you have to do a deep clone every time. Arc<Vec<u8>> is also not sufficient, because Bytes lets you share a reference count among slices to different offset into the buffer.
- dbaupp 9y ago
  Being an ARC backed array isn't like std::string, and I don't see why having that copy-on-write behaviour implies it is fundamental. Could you expand?
binarycrusader 9y ago
A little copying is better than a little dependency. https://go-proverbs.github.io/ https://go-proverbs.github.io/ As a developer at a large software company, every dependency that is not part of the language runtime itself is a pain because legal paperwork and evaluation has to be done for each individual component before I can use it / ship it. NOTE: I am not referring to copying I would be doing, but to the crates thats have little dependencies that should consider including a copy of their little dependency instead via whatever method is appropriate for licensing. This makes languages like Python, Go, Perl, etc. preferable over languages/projects like node.js, rust, etc. because (at least I) don't generally end up with tens/hundreds of dependencies since the standard library is rich enough for most work. Additionally, I personally despise have many, small dependencies because it means I have more to think about and manage. Instead of just being able to think about the version of the compiler/standard library I'm using, I have to consider every individual crate. I'm well aware of why rust chose to make certain tradeoffs with crates, but having numerous pieces of what many of us consider "basic functionality" as a third-party dependency is frustrating. Languages with richer standard libraries have spoiled us all.
- pcwalton 9y ago
  > A little copying is better than a little dependency. https://go-proverbs.github.io/ https://go-proverbs.github.io/ I really disagree with that quote. Copying is how you get bugs sticking around in software for all time (for example, doing binary searches in a way that avoids overflow is surprisingly tricky, and the endless copying of naive binary search code is why this bug is so difficult to eradicate). Honestly, that quote is just an excuse to avoid the hard work of making the language ecosystem handle dependencies properly. > As a developer at a large software company, every dependency that is not part of the language runtime itself is a pain because legal paperwork and evaluation has to be done for each individual component before I can use it / ship it. How is copying better than dependencies in this regard? You presumably need legal signoff either way. > Additionally, I personally despise have many, small dependencies because it means I have more to think about and manage. Instead of just being able to think about the version of the compiler/standard library I'm using, I have to consider every individual crate. This is what the "Rust platform" is designed to address. It is nice to be able to refer to a specific version of the Rust platform, but that doesn't mean you have to give up on the massive ergonomic benefit of the Cargo ecosystem relative to copying and pasting code.
camgunz 9y ago
I can't speak for the author of course, but probably their argument is that a systems language should be able to directly manipulate bits and bytes without outside dependencies. I don't know that I agree. A reasonable counter example is that you need library support to allocate memory in C. The argument is that it's a feature to not require C implementations to include dynamic memory allocation because not all projects allow it. My point being that what "should" be in a language usually depends on what you're using it for, and for a general purpose language keeping that very small is at least a consistent design.
- benlorenzetti 9y ago
  You don't need a library to allocate memory in C. For example on Linux, use sbrk() system call to grow the heap, then start using it. But your larger point still stands.
  - camgunz 9y ago
    Aha yeah, fair point :)
  - Sharlin 9y ago
    I think that the point was more that heap allocation is not a standard language primitive in C. And indeed, it would be ludicruous to require dynamic allocation support from freestanding implementations (no standard library because typically your platform doesn't even have an OS).
    - benlorenzetti 9y ago
      Ah sorry camgunz I interpreted the counter example as claiming malloc() was a dependency. referencing the wrong subject!
    - Sprocklem 9y ago
      For what it's worth, C++ can have freestanding implementations and it provides dynamic allocation support at a syntactic level. Freestanding applications, however, need to provide an "operator new" function (with the appropriate memory allocation code) if they want to use it.

zanny 9y ago

There is a real change in mentality you have to go through if you transition from a fairly strict C/C++/ even Java background to trying out Rust. In the former languages, adding dependencies rapidly becomes a painful experience, whereas Rust does much better dependency management and automatic building than even Python (where you need a requirements file or something similar to go pull down all the deps). With Rust, you really should just use crates. The std is meant to be limited to just the most used code and that which should not change for the sake of keeping the ecosystem stable.

ditonal 9y ago
> even Python Off topic, but it shouldn't be better than "even Python", because Python has a really, really broken dependency system. Far more so than Java, which has Maven/Gradle which are both infinitely better than the pip/virtualenv disaster. People complain about things like shading in Maven being complicated. What they might not realize is that pip doesn't even try to address conflicting dependencies, it will just silently give you the wrong version! You ask for A==0.2 and it will give you A==0.1 if another dependency asked for A==0.1 first. And it won't even warn you even though it's straight up broken behavior. Virtualenv makes packaging annoying since it's almost vendoring but not quite. To totally understand the packaging system forces you into the world of eggs, wheels, disutils, conflicting versions, condas, etc. Sorry for tangent, just thought it was funny you would hold Python up as a standard of dependency excellence when it's probably the worst overall ecosystem of major languages.
- nolok 9y ago
  I find it super weird how Python is still in this state while even PHP has managed to get something good going. To me, this is a clear case of trying to fix a broken system whereas starting from scratch would be much better.
- zanny 9y ago
  My language development went from C -> C++ -> Java -> Python. So when I got there and figured out pip was a thing (or easy_install back in the day) it was a major innovation at the time. Additionally, for anyone coming from almost all compiled native languages, a native environment like Rust with better dependency management than Python (which, as you say, and in retrospect, is pretty broken) is a bit of a mind screw.
  - k__ 9y ago
    The Python devs I worked with always envied me for npm. I asked them if they don't have something similar with pip, but seems like npm is a whole different level. Cargo should even be better than npm. On the other hand I always asked myself if they couldn't simply use Nix?
    - steveklabnik 9y ago
      I can't simply use Nix; it doesn't support Windows.
      - k__ 9y ago
        It doesn't run on the Linux subsystem?
        steveklabnik 9y ago
        I hear it sorta works, but nix-shell doesn't. Regardless, while it's cool and useful, it's not really actual Windows support.
nolok 9y ago
Haven't used Rust yet, is it kind of similar to Go where any file can simply import and then Go knows how to fetch and build when needed ? And then when you remove the calls (eg during a refactor) the compiler force you to remove the imports too, stopping the infinite bloat caused by "no one really knows if we still need this" that can be common in other language. I really liked that "dependancy as part of the language and build tool". Of course, Go still had its own dependancy issues (versionning, availability, ...) that they now work on, but that was a step in a direction I quite liked.
- Manishearth 9y ago
  > is it kind of similar to Go where any file can simply import You declare your crate dependencies in a Cargo.toml file. Rust does proper versioning of dependencies so having a separate manifest is desirable. Within your code you declare the existence of a crate via `extern crate` and then you can use it wherever. > the compiler force you to remove the imports too it warns.
  - greenhouse_gas 9y ago
    Do you know if there are plans for rustfmt to auto-import like gofmt does? In the sense that if a crate is available (in the .toml file) and you reference it, rustfmt will automatically insert the required "import" and "use".
    - steveklabnik 9y ago
      There's been talk of it, but it hasn't been built yet. It's more likely to be a part of RLS than rustfmt.
    - killercup 9y ago
      If the compiler gives you an error with a suggestion "You probably need to add `extern crate foo;`" (and it already does in some cases), RLS (i.e., your editor plugin) will be able to automatically add it for you (soon™).
    - 9y ago
pjmlp 9y ago
A current problem with " should just use crates" is that cargo only works with source code dependencies, not binary libraries. So each new crate added to your project ends up increasing the overall build time, which gets exponentially bad if you happen to add a dependency to a crate than happens to have a big dependency list. I have a very basic word counting application with a GUI written in Gtk-rs, a fresh build straight out of "git clone" takes a few minutes, mainly thanks to Pango.
- Sean1708 9y ago
  > So each new crate added to your project ends up increasing the overall build time, That's only a problem the first time you compile though, isn't it?
  - pjmlp 9y ago
    A problem that I usually don't have with C++[0], Java[1] or .NET[2], thanks to the build systems support for binary libraries. [0] - I eschew header only libraries, only if there is no technical way around them (e.g. templates). [1] - I can AOT compile with Excelsior JET, IBM J9, JamaicaVM, PTC, , Oracle Java 9 (Linux x64 for the time being), ... [2] - I can AOT compile with Mono, NGEN, .NET Native, CoreRT, IL2CPP, ...
    - Sean1708 9y ago
      I'm not saying it isn't a problem, I just wanted to know if I was correct in thinking that it is only a problem the first time you compile each crate (or rather each version of each crate, I think).
      - pjmlp 9y ago
        I see. Yes, it is a problem when you do a clean build, or when you have common crates across projects, because cargo doesn't have a concept of build cache. You can try to workaround it by setting all target directories to same one via target-dir in your .cargo/config file, but there is no guarantee that the crate won't get rebuild.
CodesInChaos 9y ago
Every dependency can run code on every computer your project runs on. That means you have to trust its author to: 1. not be malicious 2. not write a vulnerability by accident 3. not get their computer infected, their email account hijacked, etc. 4. be wise in transferring ownership 5. not add a dependency with a license incompatible with your project All the above concerns apply recursively to the dependencies of the dependency.

mastax 9y ago

Some C & C++ programmers are averse to third-party libraries (moreso than other languages in my experience). It's a valid position, but if you really value that then perhaps Rust is not for you.

valarauca1 9y ago

I agree with the author. Bytes is basically just C++'s `std::string` (kind of don't bite my head off people who've memorized the C++17 standard). Its an ARC backed array [1]. This suggests it should be a fairly fundamental abstraction. Edit: ^^^My C++ is wrong sorry :( Really I disagree with its purpose. In its immutable, non-threaded safe form you can create the same structure by just borrowing a value. This ofc requires making your peace with the borrow checker and `Cow<'a, T>` copy on write types. By in large Bytes advertised purpose is network code. And for networking code its really only _super_ useful if your using a Packet Ring in Linux. As jemalloc will not return regularly re-allocated buffers. Really this is all performance theater. How you manage/architect your socket reads/writes will have an order of magnitude larger effect then what abstraction you _store_ those bytes in once read. [1] https://carllerche.github.io/bytes/bytes/struct.Bytes.htm https://carllerche.github.io/bytes/bytes/struct.Bytes.htm

tatterdemalion 9y ago
Bytes is intended for use with the tokio ecoystem where you cna't use references often because borrowing across a yield point in a future would be a lifetime error.
- valarauca1 9y ago
  Can't `Vec<u8>` serve the same purpose?
  - tatterdemalion 9y ago
    Then you have to do a deep clone every time. Arc<Vec<u8>> is also not sufficient, because Bytes lets you share a reference count among slices to different offset into the buffer.
dbaupp 9y ago
Being an ARC backed array isn't like std::string, and I don't see why having that copy-on-write behaviour implies it is fundamental. Could you expand?

binarycrusader 9y ago

A little copying is better than a little dependency. https://go-proverbs.github.io/ https://go-proverbs.github.io/ As a developer at a large software company, every dependency that is not part of the language runtime itself is a pain because legal paperwork and evaluation has to be done for each individual component before I can use it / ship it. NOTE: I am not referring to copying I would be doing, but to the crates thats have little dependencies that should consider including a copy of their little dependency instead via whatever method is appropriate for licensing. This makes languages like Python, Go, Perl, etc. preferable over languages/projects like node.js, rust, etc. because (at least I) don't generally end up with tens/hundreds of dependencies since the standard library is rich enough for most work. Additionally, I personally despise have many, small dependencies because it means I have more to think about and manage. Instead of just being able to think about the version of the compiler/standard library I'm using, I have to consider every individual crate. I'm well aware of why rust chose to make certain tradeoffs with crates, but having numerous pieces of what many of us consider "basic functionality" as a third-party dependency is frustrating. Languages with richer standard libraries have spoiled us all.

pcwalton 9y ago
> A little copying is better than a little dependency. https://go-proverbs.github.io/ https://go-proverbs.github.io/ I really disagree with that quote. Copying is how you get bugs sticking around in software for all time (for example, doing binary searches in a way that avoids overflow is surprisingly tricky, and the endless copying of naive binary search code is why this bug is so difficult to eradicate). Honestly, that quote is just an excuse to avoid the hard work of making the language ecosystem handle dependencies properly. > As a developer at a large software company, every dependency that is not part of the language runtime itself is a pain because legal paperwork and evaluation has to be done for each individual component before I can use it / ship it. How is copying better than dependencies in this regard? You presumably need legal signoff either way. > Additionally, I personally despise have many, small dependencies because it means I have more to think about and manage. Instead of just being able to think about the version of the compiler/standard library I'm using, I have to consider every individual crate. This is what the "Rust platform" is designed to address. It is nice to be able to refer to a specific version of the Rust platform, but that doesn't mean you have to give up on the massive ergonomic benefit of the Cargo ecosystem relative to copying and pasting code.

camgunz 9y ago

I can't speak for the author of course, but probably their argument is that a systems language should be able to directly manipulate bits and bytes without outside dependencies. I don't know that I agree. A reasonable counter example is that you need library support to allocate memory in C. The argument is that it's a feature to not require C implementations to include dynamic memory allocation because not all projects allow it. My point being that what "should" be in a language usually depends on what you're using it for, and for a general purpose language keeping that very small is at least a consistent design.

benlorenzetti 9y ago
You don't need a library to allocate memory in C. For example on Linux, use sbrk() system call to grow the heap, then start using it. But your larger point still stands.
- camgunz 9y ago
  Aha yeah, fair point :)
- Sharlin 9y ago
  I think that the point was more that heap allocation is not a standard language primitive in C. And indeed, it would be ludicruous to require dynamic allocation support from freestanding implementations (no standard library because typically your platform doesn't even have an OS).
  - benlorenzetti 9y ago
    Ah sorry camgunz I interpreted the counter example as claiming malloc() was a dependency. referencing the wrong subject!
  - Sprocklem 9y ago
    For what it's worth, C++ can have freestanding implementations and it provides dynamic allocation support at a syntactic level. Freestanding applications, however, need to provide an "operator new" function (with the appropriate memory allocation code) if they want to use it.