5 ms·

(2017) More impressive: manipulating strings at compile-time. Even more impressive: using string manipulation at compile-time to build valid D code, then use

(2017) More impressive: manipulating strings at compile-time. Even more impressive: using string manipulation at compile-time to build valid D code, then use string mixins to inject it into the current program, passing it on to the compiler to be compiled as part of the program being built. This is done by std.regex to compile regular expressions to native code during compilation. Even further impressive: using a string import to read a DSL from disk, transpiling it into D code, and passing it on to the compiler. This is used by Vibe.d to compile HTML templates to native code. Granted, all this is fairly old news, and a few other languages have since caught up.

dmead 5y ago
pretty much lisp macros with the constraints of a compile language. haskell templates are similar.
- Kranar 5y ago
  Most people don't compare D to Lisp or other languages with excellent metaprogramming. D is almost exclusively compared to C++ and C++ went through a painful metaprogramming phase of its own in the 2000s. While C++ is not by any means a great meta-language, it's improved considerably since that time. D still has some interesting metaprogramming features, but on the whole I don't think it's enough to justify the rest of the baggage that comes with the language. Buggy implementations, releases that constantly break valid source code and make it a pain to work with dependencies along with a poor/unscalable garbage collector. The ability to write a string literal that contains D code, perform raw string manipulations on it and then compile the result certainly sounds neat on paper and if you're working on a solo project it can be a fun exercise to blog about, but beyond the novelty you'd hardly find a mature or reliable codebase written by a team of professionals using hacks like these.
  - mhh__ 5y ago
    No one does manipulation of D code as strings, and no one is really telling you to IMO. I do however work on a large-ish codebase with many professionals developers which makes extensive use of metaprogramming. We also don't have particularly big issues with moving from version to version so how old is your hearsay? C++ doesn't even have proper modules yet, it's still not as good as D in more regards than just metaprogramming.
    - pjmlp 5y ago
      It certainly has, using them on Visual Studio 2022 C++20 mode, and you don't need D's hack to import them inside functions.
      - mhh__ 5y ago
        Which hack would that be? So visual studio has an implementation not many people use, what of the other compilers? I'm still yet to even compile a project that actually uses them, and they must have spent a significant proportion of my entire life trying to get them in the standard.
        pjmlp 5y ago
        It is on the standard, update yourself. Naturally not every compiler is yet fully C++20 compliant but they will get there, VC++ is there, next release of GCC is looking quite good, clang well lots of issues anyway now that Apple and Google resources went away and not everyone has the same love for upstream. As for the hack, go dive into D forums from like five years ago thereabouts, since import statements in D fully parse the source of the respective modules, the hack to speed compilation time was to move import statements into the function bodies so that they are only processed if that code is actually parsed. If I am not mistaken, Andrei was one of the persons suggesting this.
        mhh__ 5y ago
        I'm aware it's in the standard now, my point is that it's taken them a very long time to get to this state and what they do have isn't all that impressive. Concepts have possibly taken my actual whole life although I don't know when the first proposal was. That is a damp squib, of all the things I view as hacky in D that is not one of them. Local imports are favoured for stylistic reasons anyway, they don't actually make all that much difference for code that isn't dead anyway. What's wrong with importing things where they are used? IIRC Andrei proposed the local imports, but it was only a matter of removing an error message because they were banned at the time.
  - WalterBright 5y ago
    You can use malloc/free and RAII if you prefer it to automatic memory management. It's nice to have a choice. Having a high performance GC requires inserting write gates into the executable code. This is only worthwhile if GC is the only allocation strategy the language uses. As the GC is optional in D, and is not actually used that much it is not worth the general slowdown of the write gates. > you'd hardly find a mature or reliable codebase written by a team of professionals using hacks like these. I wouldn't be so sure, as mature and reliable codebases use the hackish unhygienic preprocessor macros every day :-)
    - spijdar 5y ago
      > I wouldn't be so sure, as mature and reliable codebases use the hackish unhygienic preprocessor macros every day :-) Isn't it funny how the most mature software (and often by extension, "reliable" e.g. have been in use the longest with the most bugs beaten out by years of hacks and patches) seem so far away from "best practices"? I've always wondered about the root of this. Are programmers trudging along using "brute force" more effective, or is it just pure survivor bias on projects/products that people didn't give up on?
      - nicwilson 5y ago
        Probably because best practice and changes at a faster rate than coding standards for old codebases. I suspect this observation exhibits some survivorship bias as well, then again you get things like OpenSSL...
      - tzs 5y ago
        It's probably useful to distinguish between two kinds of "best practices". One kind is things dealing directly with the underlying thing you are trying to accomplish with a program. Say it is a program doing some sort of physics or engineering calculation that at some point needs to add a bunch of floating point numbers. Best practice might include using something like Kahan summation [1] to reduce the error. I'd expect to see that kind of best practice in both mature programs and new programs because it is something that directly relates to what the program actually is trying to do. The other kind of "best practices" are those concerning the environment and practice of programming itself. Many of those seem designed to deal with the case where there is a lot of turnover in who is working on the software. People join a project, work on it for a year or two, and then move off to something else. You adopt these best practices so that new people will be productive faster so you can get useful work out of them before they leave. So these best practices tend to change as fad and fashion come and go. The mature software that has been in use for a very long time I think is more likely to have a much higher percentage of its work done by people who have been working on it long term, and have a lower turnover, and when someone new does come on they stay longer. They don't need to have their best practices match the fad and fashion best practices used by the shorter term projects. [1] https://en.wikipedia.org/wiki/Kahan_summation_algorithm https://en.wikipedia.org/wiki/Kahan_summation_algorithm
  - p0nce 5y ago
    Been parsing JSON at compile time with std.json for years, nothing ever broke. It wasn't even designed for this.
  - nicwilson 5y ago
    As noted elsewhere it seems your experience is somewhat outdated: the releases of the LLVM D Compiler (one of the two compilers worth using for production builds, the other being GDC) are buffered to the bugs introduced in DMD (which is more stable than it used to be although there are still regressions), and there is a fork based GC available for linux, but as the GC will only ever trigger on allocation, don't use it and it won't collect. > While C++ is not by any means a great meta-language, it's improved considerably since that time. C++ has also painted itself into a corner multiple times too, which despite being technically an improvement over the status quo are lacking severely in their utility. C++ screwed up "constexpr if" big time by always introducing a scope (which costs you a pair of {}'s in the rare occasion you need one) which means you can't conditionally insert declarations (i.e. variables, structs/classes, functions). > but beyond the novelty you'd hardly find a mature or reliable codebase written by a team of professionals using hacks like [string manipulation and mixins]. They are a wonderful hack when you need them and nothing else will do what you want. This is not unlike resorting to macros in C++, except that its hygienic, unlike macros. I'm not claiming the project is mature and I'm only one person, but reliable definitely out there. The most heinous set of string mixins i've ever written[1] has definitely got to be the code for generating wrappers to call the OpenCL object property querying functions (clGetDeviceInfo & friends). You need to pass a size and a void pointer to the address of the return object that you have to call once, twice or more (depending on the type of the queried property) to figure out how much memory you need to allocate to call it again. The important thing is that the interface[2] you use to drive this code generation is very clean and return on investment for getting the generic case correct is large. [1]: https://github.com/libmir/dcompute/blob/master/source/dcompute/driver/ocl/util.d#L71 https://github.com/libmir/dcompute/blob/master/source/dcompu... [2]: https://github.com/libmir/dcompute/blob/master/source/dcompute/driver/ocl/device.d#L172 https://github.com/libmir/dcompute/blob/master/source/dcompu...
    - renox 5y ago
      About the 'constexpr if' introducing a scope, they did it volontary against the very good (and fun) proposal of Andrei Alexandrescu, so there must be arguments against 'scopelesd constexpr if' even though I admit that I don't understand what they could be.. After all #if also doesn't introduce a scope!
  - Simon_O_Rourke 5y ago
    > Buggy implementations, releases that constantly break valid source code and make it a pain to work with dependencies along with a poor/unscalable garbage collector. I had the misfortune to have to use D professionally for a little over a year, and I can confirm all of the above are sadly true. The garbage collector was a headache from day one, and was never not a problem.
    - nicwilson 5y ago
      May I enquire when this was, and for what applications?
      - Simon_O_Rourke 5y ago
        It was writing back-end systems code around 2017. I had to help develop a whole specialized production and deployment/restart system just to get around memory issues in production.
- p0nce 5y ago
  D is the power of lisp macros with a native language and without a compiler in its runtime library.
- floatboth 5y ago
  Template Haskell and Rust proc macros aren't similar to pervasive CTFE. TH/procmacros just give you lexical tokens, you then parse them (typically adding a monstrous dependency on haskell-src-exts/syn) and emit AST. The way D is designed, you don't need such a special facility because CTFE is everywhere. You can run normal functions at compile time. You can have compile-time "static if" and "static foreach" statements, and use various introspection calls (confusingly called "traits") in those. Zig has similar "comptime" functionality. C++ now has "if constexpr" but not any of the introspection features.

dmead 5y ago

pretty much lisp macros with the constraints of a compile language. haskell templates are similar.

Kranar 5y ago
Most people don't compare D to Lisp or other languages with excellent metaprogramming. D is almost exclusively compared to C++ and C++ went through a painful metaprogramming phase of its own in the 2000s. While C++ is not by any means a great meta-language, it's improved considerably since that time. D still has some interesting metaprogramming features, but on the whole I don't think it's enough to justify the rest of the baggage that comes with the language. Buggy implementations, releases that constantly break valid source code and make it a pain to work with dependencies along with a poor/unscalable garbage collector. The ability to write a string literal that contains D code, perform raw string manipulations on it and then compile the result certainly sounds neat on paper and if you're working on a solo project it can be a fun exercise to blog about, but beyond the novelty you'd hardly find a mature or reliable codebase written by a team of professionals using hacks like these.
- mhh__ 5y ago
  No one does manipulation of D code as strings, and no one is really telling you to IMO. I do however work on a large-ish codebase with many professionals developers which makes extensive use of metaprogramming. We also don't have particularly big issues with moving from version to version so how old is your hearsay? C++ doesn't even have proper modules yet, it's still not as good as D in more regards than just metaprogramming.
  - pjmlp 5y ago
    It certainly has, using them on Visual Studio 2022 C++20 mode, and you don't need D's hack to import them inside functions.
    - mhh__ 5y ago
      Which hack would that be? So visual studio has an implementation not many people use, what of the other compilers? I'm still yet to even compile a project that actually uses them, and they must have spent a significant proportion of my entire life trying to get them in the standard.
      - pjmlp 5y ago
        It is on the standard, update yourself. Naturally not every compiler is yet fully C++20 compliant but they will get there, VC++ is there, next release of GCC is looking quite good, clang well lots of issues anyway now that Apple and Google resources went away and not everyone has the same love for upstream. As for the hack, go dive into D forums from like five years ago thereabouts, since import statements in D fully parse the source of the respective modules, the hack to speed compilation time was to move import statements into the function bodies so that they are only processed if that code is actually parsed. If I am not mistaken, Andrei was one of the persons suggesting this.
        mhh__ 5y ago
        I'm aware it's in the standard now, my point is that it's taken them a very long time to get to this state and what they do have isn't all that impressive. Concepts have possibly taken my actual whole life although I don't know when the first proposal was. That is a damp squib, of all the things I view as hacky in D that is not one of them. Local imports are favoured for stylistic reasons anyway, they don't actually make all that much difference for code that isn't dead anyway. What's wrong with importing things where they are used? IIRC Andrei proposed the local imports, but it was only a matter of removing an error message because they were banned at the time.
- WalterBright 5y ago
  You can use malloc/free and RAII if you prefer it to automatic memory management. It's nice to have a choice. Having a high performance GC requires inserting write gates into the executable code. This is only worthwhile if GC is the only allocation strategy the language uses. As the GC is optional in D, and is not actually used that much it is not worth the general slowdown of the write gates. > you'd hardly find a mature or reliable codebase written by a team of professionals using hacks like these. I wouldn't be so sure, as mature and reliable codebases use the hackish unhygienic preprocessor macros every day :-)
  - spijdar 5y ago
    > I wouldn't be so sure, as mature and reliable codebases use the hackish unhygienic preprocessor macros every day :-) Isn't it funny how the most mature software (and often by extension, "reliable" e.g. have been in use the longest with the most bugs beaten out by years of hacks and patches) seem so far away from "best practices"? I've always wondered about the root of this. Are programmers trudging along using "brute force" more effective, or is it just pure survivor bias on projects/products that people didn't give up on?
    - nicwilson 5y ago
      Probably because best practice and changes at a faster rate than coding standards for old codebases. I suspect this observation exhibits some survivorship bias as well, then again you get things like OpenSSL...
    - tzs 5y ago
      It's probably useful to distinguish between two kinds of "best practices". One kind is things dealing directly with the underlying thing you are trying to accomplish with a program. Say it is a program doing some sort of physics or engineering calculation that at some point needs to add a bunch of floating point numbers. Best practice might include using something like Kahan summation [1] to reduce the error. I'd expect to see that kind of best practice in both mature programs and new programs because it is something that directly relates to what the program actually is trying to do. The other kind of "best practices" are those concerning the environment and practice of programming itself. Many of those seem designed to deal with the case where there is a lot of turnover in who is working on the software. People join a project, work on it for a year or two, and then move off to something else. You adopt these best practices so that new people will be productive faster so you can get useful work out of them before they leave. So these best practices tend to change as fad and fashion come and go. The mature software that has been in use for a very long time I think is more likely to have a much higher percentage of its work done by people who have been working on it long term, and have a lower turnover, and when someone new does come on they stay longer. They don't need to have their best practices match the fad and fashion best practices used by the shorter term projects. [1] https://en.wikipedia.org/wiki/Kahan_summation_algorithm https://en.wikipedia.org/wiki/Kahan_summation_algorithm
- p0nce 5y ago
  Been parsing JSON at compile time with std.json for years, nothing ever broke. It wasn't even designed for this.
- nicwilson 5y ago
  As noted elsewhere it seems your experience is somewhat outdated: the releases of the LLVM D Compiler (one of the two compilers worth using for production builds, the other being GDC) are buffered to the bugs introduced in DMD (which is more stable than it used to be although there are still regressions), and there is a fork based GC available for linux, but as the GC will only ever trigger on allocation, don't use it and it won't collect. > While C++ is not by any means a great meta-language, it's improved considerably since that time. C++ has also painted itself into a corner multiple times too, which despite being technically an improvement over the status quo are lacking severely in their utility. C++ screwed up "constexpr if" big time by always introducing a scope (which costs you a pair of {}'s in the rare occasion you need one) which means you can't conditionally insert declarations (i.e. variables, structs/classes, functions). > but beyond the novelty you'd hardly find a mature or reliable codebase written by a team of professionals using hacks like [string manipulation and mixins]. They are a wonderful hack when you need them and nothing else will do what you want. This is not unlike resorting to macros in C++, except that its hygienic, unlike macros. I'm not claiming the project is mature and I'm only one person, but reliable definitely out there. The most heinous set of string mixins i've ever written[1] has definitely got to be the code for generating wrappers to call the OpenCL object property querying functions (clGetDeviceInfo & friends). You need to pass a size and a void pointer to the address of the return object that you have to call once, twice or more (depending on the type of the queried property) to figure out how much memory you need to allocate to call it again. The important thing is that the interface[2] you use to drive this code generation is very clean and return on investment for getting the generic case correct is large. [1]: https://github.com/libmir/dcompute/blob/master/source/dcompute/driver/ocl/util.d#L71 https://github.com/libmir/dcompute/blob/master/source/dcompu... [2]: https://github.com/libmir/dcompute/blob/master/source/dcompute/driver/ocl/device.d#L172 https://github.com/libmir/dcompute/blob/master/source/dcompu...
  - renox 5y ago
    About the 'constexpr if' introducing a scope, they did it volontary against the very good (and fun) proposal of Andrei Alexandrescu, so there must be arguments against 'scopelesd constexpr if' even though I admit that I don't understand what they could be.. After all #if also doesn't introduce a scope!
- Simon_O_Rourke 5y ago
  > Buggy implementations, releases that constantly break valid source code and make it a pain to work with dependencies along with a poor/unscalable garbage collector. I had the misfortune to have to use D professionally for a little over a year, and I can confirm all of the above are sadly true. The garbage collector was a headache from day one, and was never not a problem.
  - nicwilson 5y ago
    May I enquire when this was, and for what applications?
    - Simon_O_Rourke 5y ago
      It was writing back-end systems code around 2017. I had to help develop a whole specialized production and deployment/restart system just to get around memory issues in production.
p0nce 5y ago
D is the power of lisp macros with a native language and without a compiler in its runtime library.
floatboth 5y ago
Template Haskell and Rust proc macros aren't similar to pervasive CTFE. TH/procmacros just give you lexical tokens, you then parse them (typically adding a monstrous dependency on haskell-src-exts/syn) and emit AST. The way D is designed, you don't need such a special facility because CTFE is everywhere. You can run normal functions at compile time. You can have compile-time "static if" and "static foreach" statements, and use various introspection calls (confusingly called "traits") in those. Zig has similar "comptime" functionality. C++ now has "if constexpr" but not any of the introspection features.