10 ms·

The D Garbage Collector

139 points by sconxu 10y ago

tombert 10y ago
Has anyone here done any production-level stuff with D? I'm debating using it for my next project, but I want to see if it's industry-tested.
- mark_element 10y ago
  https://dlang.org/orgs-using-d.html https://dlang.org/orgs-using-d.html
  - Cyph0n 10y ago
    Doesn't Facebook use D internally?
    - gbrown_ 10y ago
      They wrote a C/C++ preprocessor in D, but this now appears to be abandoned. https://github.com/facebookarchive/warp https://github.com/facebookarchive/warp
      - Cyph0n 10y ago
        That's too bad. I guess OCaml is the non-mainstream language Facebook seems to be interested in. One thing D and Nim are lacking is adoption by a large company, and I feel that's partly what's keeping them behind Go and Rust.
    - codemac 10y ago
      Andrei left fb in 2015 - probably due to their lack of commitment to D & his F stocks doing well.
      - bachmeier 10y ago
        Andrei works full time on the D language/D foundation. To my knowledge, his decision to leave Facebook had something to do with his family and his desire to work on D full time, not any problems with his employer.
- swah 10y ago
  Most folks will say that either C++ has improved or use Rust. (D was a really lovely experience a few years ago, though..)
  - p0nce 10y ago
    Most folks haven't tried D.
    - x0x0 10y ago
      Since you use it in prod / sell products written in it, could you (briefly) share why you like it and the advantages you see in it? Thank you!
      - p0nce 10y ago
        Went from Delphi to Ocaml to lots of C++ to D. Basically I'm happier with the products I wrote in D so I keep using it. It's an almost schizophrenic language that always says "yes" when you ask if something is possible. The community regularly "discovers" new idioms within the language. Learning D for real is as involved as any other language, but what people remark first is the low mental overhead and how absurdly practical it is. Advantages I see would be: fast to edit, compile, and run. Non-controversial syntax, focus on power, and the fact you kind of already know it.
        int_19h 10y ago
        How good is the tooling? IDE support, code completion and debugging especially.
        Ace17 10y ago
        There are 3 major compiler implementations (dmd, gdc, ldc). I mostly work with gdc from GNU/Linux, and basically, everything that works with g++ works with gdc. More precisely, I use, on a daily basis: - vim + ctags (editor + symbol lookup) - gdb (debugger) - gcov (code coverage) - valgrind (mostly callgrind (call-graph) and massif (memory profiler)) - oprofile (cpu profiler) Unfortunately there's no ccache equivalent yet, and I'm missing it. For the Windows people, there's VisualD, which basically is the integration in Visual Studio. At this point, you have an IDE and a debugger (I don't know about code completion). There's also a build-tool (with package-managing abilities), called "dub" ; that handles fetching and building external dependencies (in the vein of npm).
- glaze 10y ago
  Quantum Break is a modern Xbox One & PC game and uses D in some subsystems.
- oripel 10y ago
  I do every day. It's pretty awesome, I recommend trying it. D's CTFE and templates are really really nice to use. I can write very low-level high-performance memory sensitive code, controlling everything, but the code can still be beautiful with great compiler checking. It can confuse my C++ brain sometimes, though, and then it's great to have someone to consult with (e.g on the D forums).
- p0nce 10y ago
  You'll be okay.
- vogt 10y ago
  Our data team at Emsi (www.economicmodeling.com) use it in production every day. We are a labor market data viz company with a multitude of different sources of data, most of which are public.
- rdnetto 10y ago
  I used it for a while, ended up switching to Haskell, which had a lot of similar benefits despite being so different. If you're going to use it, I highly recommend the D Programming Language (book). It's extremely well written, and remains once of my favourite technical books.
- Ace17 10y ago
  My company (~50 people) has been using D in production for several years. We rely on gdc (we measured it generates the fastest code for our usage). Nearly once per year, a blocking bug sneaks into the compiler and needs to be worked-around (crashes on invalid code are easy to avoid ; sometimes, it's a little more tricky, but we've never been blocked for more than 1 day). The maintainers are very active, though, and it's generally quickly fixed, provided that I report the bug myself. There's also been some friction when it came to cross-compilation (definitely doable, but not as easy at "apt install mingw-gcc"). In the team, the adoption went well ; coming from C++, switching to D feels natural. The syntax is more concise and consistent than C++14 syntax. The standard library handles so much stuff (md5, sending emails, command line parsing, filesystem, process creation) that most of our projects don't depend on any other external library (nor do they depend on any OS-specific stuff). The fact that all variables are initialized by default makes it hard to have non-deterministic behaviour ; and the fact that the "buffer" abstraction (aka "Span"/"Slice"/"string_view"/"ptr+len") is in the language makes it really easy to write code which is concise and secure. C++ has been slowly catching up since 2011, however, why wait?
rurban 10y ago
This is not really an article about "How the D garbage collector works", rather an article "How to avoid garbage in D". More tips and tricks are at: https://bitbucket.org/infognition/dstuff https://bitbucket.org/infognition/dstuff Here is the better explanation of the GC: https://dlang.org/spec/garbage.html https://dlang.org/spec/garbage.html, which is basically a very slow and conservative stop the world Mark & Sweep, which runs at every allocation. Basically a simple version of BoehmGC. Due to the unfortunate C ABI, a fast copying collector is not really doable. One cannot track all the internal and external pointers. Well, one could, but has to pessimize the locals on every external pointer reference.
- p0nce 10y ago
  A fast copying collector for D would require write barriers: using memory barriers everytime a pointer is written. This has a runtime cost. > which runs at every allocation. The article explains in great depth that this is not the case.
  - dbaupp 10y ago
    > This has a runtime cost. As does using a conservative/"very slow" GC. I'm sceptical, but it may be that D's situation means trading a pervasive slowness (if the parent is true, I don't know) for no write barriers is the right choice, differing to the conventional wisdom from other languages, but that's the argument you need to make, not just that write barriers are slow.
    - atilaneves 10y ago
      The slow GC is only a runtime cost if it ever runs. If no garbage is created in inner loops and stack allocation is preferred there's no reason it _has_ to run.
    - p0nce 10y ago
      The case against write barriers is pretty simple: the slowness of the GC can be workarounded, while memory barriers cannot. If they are needed, no other way to copy pointer values (which are quite pervasive in system programming).
dominotw 10y ago
what are the advantages of running GC only when allocations are happening, instead of running concurrently? Is the idea that you can explicitly control the behavior of GC by tuning your allocations? Is it burdensome for the programmer to constantly think about this while writing code?
- p0nce 10y ago
  > what are the advantages of running GC only when allocations are happening, instead of running concurrently? That way you don't need a dedicated thread for the GC (current one is hijecked). > Is it burdensome for the programmer to constantly think about this while writing code? Not really. I sell real-time software in D, you don't constantly think about this when writing code. It's no more burden that avoiding allocations in real-time software in C++.
  - adrianratnapala 10y ago
    But avoiding allocations in real-time C++ is a burden to some (increasingly common) kinds of programmers. The kids today are so indoctrinated in modern C++ that they don't stop to think that `vector<>::push_back()` potentially allocates behind the scenes. Writing real-time code requires a certain mentality that is at odds with modern frameworks. I expect nice languages like D have similar cultural issues.
    - p0nce 10y ago
      If you need a feel of what the community values (apart from rants), have a look at the forums. https://forum.dlang.org/ https://forum.dlang.org/ D is backed by companies like Sociomantic and Weka.io that operates in high-performance domains.
    - CyberShadow 10y ago
      FWIW, D's @nogc makes it easy to trap code that allocates accidentally.
zyngaro 10y ago
Too bad Dlang never reached mainstream. It's a perfectly suitable language for backend programming and system programming where golang is growing popular.
- jordigh 10y ago
  Golang isn't going mainstream either, and D isn't doomed to never got popular either. Let's not speak in such hyperbole. If you like language $X, use language $X and as long as you can find sufficient resources for it, don't worry about who else is using it (I personally can find sufficient resources for D).
- dpc_pw 10y ago
  If D had in-language-integrated Fibers like Go does it would be just better in every aspect. Native-fast code, fast compile, powerful generic and meta-programming. It was trying to be better C/C++ but should have instead push where Go is going: highly productive and fast. I think Rust is much better C replacement - much higher overhead and mental cost, for software that can't compromise on performance, reliability and safety.
  - WalterBright 10y ago
    I'm not sure that language integrated ones would be better. As D's abstraction abilities have improved, we've been moving features from the core language to the library.
    - dpc_pw 10y ago
      The only thing that I find great about Go is the fact that you can write, normal, imperative, blocking-io-like code dealing with about anything (network, file, any io) and get a service that can handle 10k concurrent connections without thinking much about it. If D could do the same, no matter if through language support, or a library it would be really awesome. Note: it seems to me, currently D's fibers require manual `yield`. The entire point is not to have to do it.
  - Doxin 10y ago
nialv7 10y ago
If you are interested in current D memory management situation, you might want to look at this as well: https://dlang.org/phobos/std_experimental_allocator.html https://dlang.org/phobos/std_experimental_allocator.html

139 points by sconxu 10y ago

Has anyone here done any production-level stuff with D? I'm debating using it for my next project, but I want to see if it's industry-tested.

mark_element 10y ago
https://dlang.org/orgs-using-d.html https://dlang.org/orgs-using-d.html
- Cyph0n 10y ago
  Doesn't Facebook use D internally?
  - gbrown_ 10y ago
    They wrote a C/C++ preprocessor in D, but this now appears to be abandoned. https://github.com/facebookarchive/warp https://github.com/facebookarchive/warp
    - Cyph0n 10y ago
      That's too bad. I guess OCaml is the non-mainstream language Facebook seems to be interested in. One thing D and Nim are lacking is adoption by a large company, and I feel that's partly what's keeping them behind Go and Rust.
  - codemac 10y ago
    Andrei left fb in 2015 - probably due to their lack of commitment to D & his F stocks doing well.
    - bachmeier 10y ago
      Andrei works full time on the D language/D foundation. To my knowledge, his decision to leave Facebook had something to do with his family and his desire to work on D full time, not any problems with his employer.
swah 10y ago
Most folks will say that either C++ has improved or use Rust. (D was a really lovely experience a few years ago, though..)
- p0nce 10y ago
  Most folks haven't tried D.
  - x0x0 10y ago
    Since you use it in prod / sell products written in it, could you (briefly) share why you like it and the advantages you see in it? Thank you!
    - p0nce 10y ago
      Went from Delphi to Ocaml to lots of C++ to D. Basically I'm happier with the products I wrote in D so I keep using it. It's an almost schizophrenic language that always says "yes" when you ask if something is possible. The community regularly "discovers" new idioms within the language. Learning D for real is as involved as any other language, but what people remark first is the low mental overhead and how absurdly practical it is. Advantages I see would be: fast to edit, compile, and run. Non-controversial syntax, focus on power, and the fact you kind of already know it.
      - int_19h 10y ago
        How good is the tooling? IDE support, code completion and debugging especially.
        Ace17 10y ago
        There are 3 major compiler implementations (dmd, gdc, ldc). I mostly work with gdc from GNU/Linux, and basically, everything that works with g++ works with gdc. More precisely, I use, on a daily basis: - vim + ctags (editor + symbol lookup) - gdb (debugger) - gcov (code coverage) - valgrind (mostly callgrind (call-graph) and massif (memory profiler)) - oprofile (cpu profiler) Unfortunately there's no ccache equivalent yet, and I'm missing it. For the Windows people, there's VisualD, which basically is the integration in Visual Studio. At this point, you have an IDE and a debugger (I don't know about code completion). There's also a build-tool (with package-managing abilities), called "dub" ; that handles fetching and building external dependencies (in the vein of npm).
glaze 10y ago
Quantum Break is a modern Xbox One & PC game and uses D in some subsystems.
oripel 10y ago
I do every day. It's pretty awesome, I recommend trying it. D's CTFE and templates are really really nice to use. I can write very low-level high-performance memory sensitive code, controlling everything, but the code can still be beautiful with great compiler checking. It can confuse my C++ brain sometimes, though, and then it's great to have someone to consult with (e.g on the D forums).
p0nce 10y ago
You'll be okay.
vogt 10y ago
Our data team at Emsi (www.economicmodeling.com) use it in production every day. We are a labor market data viz company with a multitude of different sources of data, most of which are public.
rdnetto 10y ago
I used it for a while, ended up switching to Haskell, which had a lot of similar benefits despite being so different. If you're going to use it, I highly recommend the D Programming Language (book). It's extremely well written, and remains once of my favourite technical books.
Ace17 10y ago
My company (~50 people) has been using D in production for several years. We rely on gdc (we measured it generates the fastest code for our usage). Nearly once per year, a blocking bug sneaks into the compiler and needs to be worked-around (crashes on invalid code are easy to avoid ; sometimes, it's a little more tricky, but we've never been blocked for more than 1 day). The maintainers are very active, though, and it's generally quickly fixed, provided that I report the bug myself. There's also been some friction when it came to cross-compilation (definitely doable, but not as easy at "apt install mingw-gcc"). In the team, the adoption went well ; coming from C++, switching to D feels natural. The syntax is more concise and consistent than C++14 syntax. The standard library handles so much stuff (md5, sending emails, command line parsing, filesystem, process creation) that most of our projects don't depend on any other external library (nor do they depend on any OS-specific stuff). The fact that all variables are initialized by default makes it hard to have non-deterministic behaviour ; and the fact that the "buffer" abstraction (aka "Span"/"Slice"/"string_view"/"ptr+len") is in the language makes it really easy to write code which is concise and secure. C++ has been slowly catching up since 2011, however, why wait?

rurban 10y ago

This is not really an article about "How the D garbage collector works", rather an article "How to avoid garbage in D". More tips and tricks are at: https://bitbucket.org/infognition/dstuff https://bitbucket.org/infognition/dstuff Here is the better explanation of the GC: https://dlang.org/spec/garbage.html https://dlang.org/spec/garbage.html, which is basically a very slow and conservative stop the world Mark & Sweep, which runs at every allocation. Basically a simple version of BoehmGC. Due to the unfortunate C ABI, a fast copying collector is not really doable. One cannot track all the internal and external pointers. Well, one could, but has to pessimize the locals on every external pointer reference.

p0nce 10y ago
A fast copying collector for D would require write barriers: using memory barriers everytime a pointer is written. This has a runtime cost. > which runs at every allocation. The article explains in great depth that this is not the case.
- dbaupp 10y ago
  > This has a runtime cost. As does using a conservative/"very slow" GC. I'm sceptical, but it may be that D's situation means trading a pervasive slowness (if the parent is true, I don't know) for no write barriers is the right choice, differing to the conventional wisdom from other languages, but that's the argument you need to make, not just that write barriers are slow.
  - atilaneves 10y ago
    The slow GC is only a runtime cost if it ever runs. If no garbage is created in inner loops and stack allocation is preferred there's no reason it _has_ to run.
  - p0nce 10y ago
    The case against write barriers is pretty simple: the slowness of the GC can be workarounded, while memory barriers cannot. If they are needed, no other way to copy pointer values (which are quite pervasive in system programming).

dominotw 10y ago

what are the advantages of running GC only when allocations are happening, instead of running concurrently? Is the idea that you can explicitly control the behavior of GC by tuning your allocations? Is it burdensome for the programmer to constantly think about this while writing code?

p0nce 10y ago
> what are the advantages of running GC only when allocations are happening, instead of running concurrently? That way you don't need a dedicated thread for the GC (current one is hijecked). > Is it burdensome for the programmer to constantly think about this while writing code? Not really. I sell real-time software in D, you don't constantly think about this when writing code. It's no more burden that avoiding allocations in real-time software in C++.
- adrianratnapala 10y ago
  But avoiding allocations in real-time C++ is a burden to some (increasingly common) kinds of programmers. The kids today are so indoctrinated in modern C++ that they don't stop to think that `vector<>::push_back()` potentially allocates behind the scenes. Writing real-time code requires a certain mentality that is at odds with modern frameworks. I expect nice languages like D have similar cultural issues.
  - p0nce 10y ago
    If you need a feel of what the community values (apart from rants), have a look at the forums. https://forum.dlang.org/ https://forum.dlang.org/ D is backed by companies like Sociomantic and Weka.io that operates in high-performance domains.
  - CyberShadow 10y ago
    FWIW, D's @nogc makes it easy to trap code that allocates accidentally.

zyngaro 10y ago

Too bad Dlang never reached mainstream. It's a perfectly suitable language for backend programming and system programming where golang is growing popular.

jordigh 10y ago
Golang isn't going mainstream either, and D isn't doomed to never got popular either. Let's not speak in such hyperbole. If you like language $X, use language $X and as long as you can find sufficient resources for it, don't worry about who else is using it (I personally can find sufficient resources for D).
dpc_pw 10y ago
If D had in-language-integrated Fibers like Go does it would be just better in every aspect. Native-fast code, fast compile, powerful generic and meta-programming. It was trying to be better C/C++ but should have instead push where Go is going: highly productive and fast. I think Rust is much better C replacement - much higher overhead and mental cost, for software that can't compromise on performance, reliability and safety.
- WalterBright 10y ago
  I'm not sure that language integrated ones would be better. As D's abstraction abilities have improved, we've been moving features from the core language to the library.
  - dpc_pw 10y ago
    The only thing that I find great about Go is the fact that you can write, normal, imperative, blocking-io-like code dealing with about anything (network, file, any io) and get a service that can handle 10k concurrent connections without thinking much about it. If D could do the same, no matter if through language support, or a library it would be really awesome. Note: it seems to me, currently D's fibers require manual `yield`. The entire point is not to have to do it.
- Doxin 10y ago

nialv7 10y ago

If you are interested in current D memory management situation, you might want to look at this as well: https://dlang.org/phobos/std_experimental_allocator.html https://dlang.org/phobos/std_experimental_allocator.html