8 ms·

Porting a Golang and Rust CLI tool to D

117 points by nurettin 6y ago

mhh__ 6y ago
I'm surprised that the rust binary is 3.5 times bigger. D is already templated to the nth degree, what more is rust doing? On top of that D links in a whole GC whereas rust does not?
- nurettin 6y ago
  Rust will generate a lot of guarding code similar to what ada does.
  - kibwen 6y ago
    The only "guarding code" that I can think of Rust generating would be for bounds checking on array access, which isn't going to be any significant contributor to binary size. When it comes to generated code, I can imagine the panic unwinding code being a noticeable contributor to the size of a Rust binary, but since D has exceptions I imagine the D binaries have comparable code for unwinding.
    - nurettin 6y ago
      Yes, it will generate Exception unwinding and bounds checks and godbolt seems to agree. Anything else you can think of? I think D is less strict when optimizing out exception code.
    - mhh__ 6y ago
      Exception unwinding is not particularly cheap on the compiler although I think it's in the sense that it muddles the CFG and the register allocation rather than inserting huge amounts of code. The LLVM codebase has exceptions turned off for those reasons and binary size (it plays a role at their scale) although its quite hard to compare exceptions Vs error codes or monads for obvious reasons (LLVM's coding standards are rightly or wrongly a C++ sleuths dream with their custom dynamic casts etc.)
  - mhh__ 6y ago
    D also generates significant amounts of said code (unless you turn it off, obviously) One feature D stole from Ada that is an absolutely inexcusable ommision in a modern programming is contract programming. Every time I start a new python script I end up writing helpers for function pre and postconditions because otherwise I can't rely on anything in the language's syntax. Invariant clauses in D (e.g. this integer in my struct must be less than 5 when the struct is in a valid state) are an absolute godsend although they do not aid compiler optimisations despite some people saying otherwise (it could have changed but last time I checked the invariant does not make it's way to LLVM or GCC)
    - ponsin 6y ago
      how is that different than in c/c++ simply adding asserts at the start and end of functions?
      - mhh__ 6y ago
        It's the same but it's a different switch to turn it on and off. Having separate syntax is also very useful because it's much easier to see through my usual spaghetti to find what the code is actually supposed to do in the abstract (i.e. if the string should be a valid email, check it in the in {} clause rather than in the middle of the function)
    - noch 6y ago
      > One feature D stole from Ada that is an absolutely inexcusable omission in a modern programming is contract programming. Incidentally, Walter Bright considers contracts to be a low value feature of D that he considered removing. [0][1] [0]: https://youtu.be/p22MM1wc7xQ?t=1h34m30s https://youtu.be/p22MM1wc7xQ?t=1h34m30s [1]: https://digitalmars.com/articles/hits.pdf https://digitalmars.com/articles/hits.pdf
- kibwen 6y ago
  Rust, like Go, statically links everything by default, and only uses dynamic linking if you ask it to. Is D using dynamic linking here?
- PudgePacket 6y ago
  Rust is statically linked by default, not sure how D works. Rust still includes some debug features that can give you a stack trace. Size will also depend on the compile options, if LTO was enabled, how much generics were used, how much string printing was used (print is a macro in Rust that generates code at the callsite), etc. Most, if not all, of these can be resolved. See https://github.com/johnthagen/min-sized-rust https://github.com/johnthagen/min-sized-rust for aiming for small binary size. This article shows a 93% size reduction, 2.7m -> 200kb, with some relatively simple modifications https://www.collabora.com/news-and-blog/blog/2020/04/28/reducing-size-rust-gstreamer-plugin/ https://www.collabora.com/news-and-blog/blog/2020/04/28/redu.... Here is someone using Rust for a 3kb demoscene program https://www.codeslow.com/2019/12/tiny-windows-executable-in-rust.html https://www.codeslow.com/2019/12/tiny-windows-executable-in-....
  - mhh__ 6y ago
    D is usually dynamic although I try to be static for the most part. IIRC The runtime is linked statically (or not at all) but Phobos is dynamic by default Rust compiles seem to blow up pretty easily, e.g. I was cleaning out my SSD the other day an I noticed that a rust GUI framework (Iced?) was sat on about 2Gb of dependencies despite being significantly incomplete (but very promising)
    - destructionator 6y ago
      > Phobos is dynamic by default That depends on your distro. The packages from dlang.org all do the static link config out-of-the-box, but some distro packages dynamic link by default instead. Given the size of that file, it looks static linked. It dynamic links libc, pthread, and a couple other OS-level things, but all D libs including druntime, phobos, etc, are statically linked. Might just be difference in dependencies.
    - PudgePacket 6y ago
      Oh boy I definitely understand you, in fact I built a Rust CLI tool to solve exactly that ;) https://github.com/tbillington/kondo https://github.com/tbillington/kondo
- cerebellum42 6y ago
  Most likely largely because of this: >Rust downloaded a lot of crates, but that's probably because the Rust version of the code has more features than mine has. For example, it fetched rpassword, a tool that hides password characters as you type them into a terminal, much like Python's getpass function. That's one of the many things I don't have in my code.
hu3 6y ago
How can D use so little memory despite sporting a GC? Go Memory usage: 13mb Rust Memory usage: 9mb D Memory usage: 7mb
- romanoderoma 6y ago
  Being around for longer means more optimizations
  - mhh__ 6y ago
    D has been around for longer but Rust and Go both have (I'd guess) on the order of 100 full time staff each. D has something like 3 at most.
    - The_rationalist 6y ago
      What? Rust has less than 5 full time paid devs. And with the fall of Mozilla it's shrinking. But yeah external contributors summed up are a big human resource
      - mhh__ 6y ago
        But how many companies outside Mozilla use rust in a capacity to contribute to the compiler, standard library, and crate ecosystem? e.g. Now rust is in chromium if Google want stuff fixing they can pay for it to be done in the same way that Linux is developed mainly by 9-5ers not working directly for the Linux project specifically. D has no paid developers at all full time IIRC (Walter Bright is living off his empire empire I think). Go has Google's direct backing, which is more what I was pointing towards.
        qzw 6y ago
        According to the recent blog post that was on the front page, Chromium is still exploring the possibility of using Rust. They’ve yet to commit.
    - 6y ago
monadic2 6y ago
...go Lang AND rust? Why would anyone do this to themselves?
- detaro 6y ago
  If only there was a link to a post with more context somewhere, like in the first sentence on the page.
  - monadic2 6y ago
    That doesn't resolve the question. It's a very odd combination of languages to decide to sling together arbitrarily.
    - detaro 6y ago
      Two statically compiled languages that get a lot of attention for writing tools in doesn't seem that odd choice to compare?
      - monadic2 6y ago
        I mean you can statically compile a lot of languages and literally every language is used to write tools. Go doesn't even use a stack compatible with the system abi and has probably the worst FFI of any language I've ever used, it has a very, err, unique type system. I mean there's a shit ton of great stuff too (compile times, great test patterns and pretty good tooling, the http stack and string manipulation and green threading is fantastic), I don't have it out for go! Meanwhile rust has a completely different memory model and type system and concurrency model, has probably the best FFI of any language outside of maybe some lisps and cpython and probably D (never used it), and they've taken forever to role out production ready http client/server stacks (though I think this is finally smoothing out!) I just don't see why they're tied together except a) static compilation and b) they emerged at roughly the same time. Rust and go are very, very different languages aimed at completely different niches.
1f60c 6y ago
> For example, it fetched rpassword, a tool that hides password characters as you type them into a terminal, much like Python's getpass function. I want to note that you can call actual C functions from D without much extra effort. Here’s a program that calls getpass: https://glot.io/snippets/fqf04b6wg2 https://glot.io/snippets/fqf04b6wg2
- Scroph 6y ago
  Thanks! I added it in this PR: https://github.com/azihassan/hashtrack-client/pull/11 https://github.com/azihassan/hashtrack-client/pull/11
  - 1f60c 6y ago
    Oh, oops. I forgot to edit my comment to add that, for some reason, it doesn’t seem work on macOS. But if that’s not a problem for you, then I’m happy I could help. Edit: I just read the code, and it seems like you found out about that before I could warn you. :)
    - Scroph 6y ago
      I didn't know actually, I only did it so that it would also compile on Windows (which I haven't tested yet tbh). Good to know it would still compile on macOS
finder83 6y ago
I've never really dived into D, but why was it never more popular? It seems like it was posed to take off, but never quite did.
- tastyminerals2 6y ago
  D had an unfortunate early history. For a long time being a one man project, surviving through Python2 to Python3 akin transition and slowly maturing during all these years. Many top D features were added along the way but were not available from the start. Also in the beginning D focus was to be a better C++ with GC which is not something that excited C++ crowd who smirked as soon as they heard about GC and never actually approached the language. It is only later that it became clear that it is a separate unique language. I think D just grew and matured naturally into something very solid. D has a small but a very skilled team of seasoned C++ veterans and language designers. I dare to say, I am yet to see better forum answers than on forum.dlang.org both on the account of the language and programming in general.
  - finder83 6y ago
    Interesting, thanks for the synopsis. :) Sounds worth looking into.
- teleforce 6y ago
  For a really good background reading on D language landscape, check this paper by its authors: https://dl.acm.org/doi/abs/10.1145/3386323 https://dl.acm.org/doi/abs/10.1145/3386323

117 points by nurettin 6y ago

I'm surprised that the rust binary is 3.5 times bigger. D is already templated to the nth degree, what more is rust doing? On top of that D links in a whole GC whereas rust does not?

nurettin 6y ago
Rust will generate a lot of guarding code similar to what ada does.
- kibwen 6y ago
  The only "guarding code" that I can think of Rust generating would be for bounds checking on array access, which isn't going to be any significant contributor to binary size. When it comes to generated code, I can imagine the panic unwinding code being a noticeable contributor to the size of a Rust binary, but since D has exceptions I imagine the D binaries have comparable code for unwinding.
  - nurettin 6y ago
    Yes, it will generate Exception unwinding and bounds checks and godbolt seems to agree. Anything else you can think of? I think D is less strict when optimizing out exception code.
  - mhh__ 6y ago
    Exception unwinding is not particularly cheap on the compiler although I think it's in the sense that it muddles the CFG and the register allocation rather than inserting huge amounts of code. The LLVM codebase has exceptions turned off for those reasons and binary size (it plays a role at their scale) although its quite hard to compare exceptions Vs error codes or monads for obvious reasons (LLVM's coding standards are rightly or wrongly a C++ sleuths dream with their custom dynamic casts etc.)
- mhh__ 6y ago
  D also generates significant amounts of said code (unless you turn it off, obviously) One feature D stole from Ada that is an absolutely inexcusable ommision in a modern programming is contract programming. Every time I start a new python script I end up writing helpers for function pre and postconditions because otherwise I can't rely on anything in the language's syntax. Invariant clauses in D (e.g. this integer in my struct must be less than 5 when the struct is in a valid state) are an absolute godsend although they do not aid compiler optimisations despite some people saying otherwise (it could have changed but last time I checked the invariant does not make it's way to LLVM or GCC)
  - ponsin 6y ago
    how is that different than in c/c++ simply adding asserts at the start and end of functions?
    - mhh__ 6y ago
      It's the same but it's a different switch to turn it on and off. Having separate syntax is also very useful because it's much easier to see through my usual spaghetti to find what the code is actually supposed to do in the abstract (i.e. if the string should be a valid email, check it in the in {} clause rather than in the middle of the function)
  - noch 6y ago
    > One feature D stole from Ada that is an absolutely inexcusable omission in a modern programming is contract programming. Incidentally, Walter Bright considers contracts to be a low value feature of D that he considered removing. [0][1] [0]: https://youtu.be/p22MM1wc7xQ?t=1h34m30s https://youtu.be/p22MM1wc7xQ?t=1h34m30s [1]: https://digitalmars.com/articles/hits.pdf https://digitalmars.com/articles/hits.pdf
kibwen 6y ago
Rust, like Go, statically links everything by default, and only uses dynamic linking if you ask it to. Is D using dynamic linking here?
PudgePacket 6y ago
Rust is statically linked by default, not sure how D works. Rust still includes some debug features that can give you a stack trace. Size will also depend on the compile options, if LTO was enabled, how much generics were used, how much string printing was used (print is a macro in Rust that generates code at the callsite), etc. Most, if not all, of these can be resolved. See https://github.com/johnthagen/min-sized-rust https://github.com/johnthagen/min-sized-rust for aiming for small binary size. This article shows a 93% size reduction, 2.7m -> 200kb, with some relatively simple modifications https://www.collabora.com/news-and-blog/blog/2020/04/28/reducing-size-rust-gstreamer-plugin/ https://www.collabora.com/news-and-blog/blog/2020/04/28/redu.... Here is someone using Rust for a 3kb demoscene program https://www.codeslow.com/2019/12/tiny-windows-executable-in-rust.html https://www.codeslow.com/2019/12/tiny-windows-executable-in-....
- mhh__ 6y ago
  D is usually dynamic although I try to be static for the most part. IIRC The runtime is linked statically (or not at all) but Phobos is dynamic by default Rust compiles seem to blow up pretty easily, e.g. I was cleaning out my SSD the other day an I noticed that a rust GUI framework (Iced?) was sat on about 2Gb of dependencies despite being significantly incomplete (but very promising)
  - destructionator 6y ago
    > Phobos is dynamic by default That depends on your distro. The packages from dlang.org all do the static link config out-of-the-box, but some distro packages dynamic link by default instead. Given the size of that file, it looks static linked. It dynamic links libc, pthread, and a couple other OS-level things, but all D libs including druntime, phobos, etc, are statically linked. Might just be difference in dependencies.
  - PudgePacket 6y ago
    Oh boy I definitely understand you, in fact I built a Rust CLI tool to solve exactly that ;) https://github.com/tbillington/kondo https://github.com/tbillington/kondo
cerebellum42 6y ago
Most likely largely because of this: >Rust downloaded a lot of crates, but that's probably because the Rust version of the code has more features than mine has. For example, it fetched rpassword, a tool that hides password characters as you type them into a terminal, much like Python's getpass function. That's one of the many things I don't have in my code.

hu3 6y ago

How can D use so little memory despite sporting a GC? Go Memory usage: 13mb Rust Memory usage: 9mb D Memory usage: 7mb

romanoderoma 6y ago
Being around for longer means more optimizations
- mhh__ 6y ago
  D has been around for longer but Rust and Go both have (I'd guess) on the order of 100 full time staff each. D has something like 3 at most.
  - The_rationalist 6y ago
    What? Rust has less than 5 full time paid devs. And with the fall of Mozilla it's shrinking. But yeah external contributors summed up are a big human resource
    - mhh__ 6y ago
      But how many companies outside Mozilla use rust in a capacity to contribute to the compiler, standard library, and crate ecosystem? e.g. Now rust is in chromium if Google want stuff fixing they can pay for it to be done in the same way that Linux is developed mainly by 9-5ers not working directly for the Linux project specifically. D has no paid developers at all full time IIRC (Walter Bright is living off his empire empire I think). Go has Google's direct backing, which is more what I was pointing towards.
      - qzw 6y ago
        According to the recent blog post that was on the front page, Chromium is still exploring the possibility of using Rust. They’ve yet to commit.
  - 6y ago

monadic2 6y ago

...go Lang AND rust? Why would anyone do this to themselves?

detaro 6y ago
If only there was a link to a post with more context somewhere, like in the first sentence on the page.
- monadic2 6y ago
  That doesn't resolve the question. It's a very odd combination of languages to decide to sling together arbitrarily.
  - detaro 6y ago
    Two statically compiled languages that get a lot of attention for writing tools in doesn't seem that odd choice to compare?
    - monadic2 6y ago
      I mean you can statically compile a lot of languages and literally every language is used to write tools. Go doesn't even use a stack compatible with the system abi and has probably the worst FFI of any language I've ever used, it has a very, err, unique type system. I mean there's a shit ton of great stuff too (compile times, great test patterns and pretty good tooling, the http stack and string manipulation and green threading is fantastic), I don't have it out for go! Meanwhile rust has a completely different memory model and type system and concurrency model, has probably the best FFI of any language outside of maybe some lisps and cpython and probably D (never used it), and they've taken forever to role out production ready http client/server stacks (though I think this is finally smoothing out!) I just don't see why they're tied together except a) static compilation and b) they emerged at roughly the same time. Rust and go are very, very different languages aimed at completely different niches.

1f60c 6y ago

> For example, it fetched rpassword, a tool that hides password characters as you type them into a terminal, much like Python's getpass function. I want to note that you can call actual C functions from D without much extra effort. Here’s a program that calls getpass: https://glot.io/snippets/fqf04b6wg2 https://glot.io/snippets/fqf04b6wg2

Scroph 6y ago
Thanks! I added it in this PR: https://github.com/azihassan/hashtrack-client/pull/11 https://github.com/azihassan/hashtrack-client/pull/11
- 1f60c 6y ago
  Oh, oops. I forgot to edit my comment to add that, for some reason, it doesn’t seem work on macOS. But if that’s not a problem for you, then I’m happy I could help. Edit: I just read the code, and it seems like you found out about that before I could warn you. :)
  - Scroph 6y ago
    I didn't know actually, I only did it so that it would also compile on Windows (which I haven't tested yet tbh). Good to know it would still compile on macOS

finder83 6y ago

I've never really dived into D, but why was it never more popular? It seems like it was posed to take off, but never quite did.

tastyminerals2 6y ago
D had an unfortunate early history. For a long time being a one man project, surviving through Python2 to Python3 akin transition and slowly maturing during all these years. Many top D features were added along the way but were not available from the start. Also in the beginning D focus was to be a better C++ with GC which is not something that excited C++ crowd who smirked as soon as they heard about GC and never actually approached the language. It is only later that it became clear that it is a separate unique language. I think D just grew and matured naturally into something very solid. D has a small but a very skilled team of seasoned C++ veterans and language designers. I dare to say, I am yet to see better forum answers than on forum.dlang.org both on the account of the language and programming in general.
- finder83 6y ago
  Interesting, thanks for the synopsis. :) Sounds worth looking into.
teleforce 6y ago
For a really good background reading on D language landscape, check this paper by its authors: https://dl.acm.org/doi/abs/10.1145/3386323 https://dl.acm.org/doi/abs/10.1145/3386323