6 ms·

DMD Compiler as a Library: A Call to Arms

72 points by ingve 3y ago

docandrew 3y ago
Given how fast the D compiler is, this might be a nice alternative to something like LuaJIT for an embedded scripting language in other apps.
- WalterBright 3y ago
  Many people do use D as a scripting language, replacing their use of bash. D programs can be compiled and run in one step. #!/usr/bin/env rdmd import std.stdio; void main() { writeln("Hello, world with automated script running!"); } > ./myprog.d Hello, world with automated script running! https://dlang.org/rdmd.html https://dlang.org/rdmd.html
  - thechao 3y ago
    Classically, you wait until your name is said three times, before appearing before the supplicant!
  - tripleo1 3y ago
    Do you also have: 1. Statically linked executables (essential but not crucial) 2. Dependency management (import tripleo.rtfm from "git:...#1234567") 3. A good AST API (jdt/roaster, javapoet, etc) 4. Tooling (cf spoon/soot) (I approve this comment, even given it's limited usefulness/expressivity)
    - WalterBright 3y ago
      1. yes 2. no 3. that's what this topic is about 4. yes
    - bachmeier 3y ago
      > 2. Dependency management (import tripleo.rtfm from "git:...#1234567") Not built into the compiler, but Dub, the official package manager distributed with the compiler, should be what I think your concise statement is referring to.
- p0nce 3y ago
  This is what HipremeEngine does for example. https://github.com/MrcSnm/HipremeEngine https://github.com/MrcSnm/HipremeEngine
UncleEntity 3y ago
Swift does something like this I believe. Not too sure on the details but I remember reading about how you can modify the parser to do...things. Looked interesting.
- favorited 3y ago
  It's part of the LLVM philosophy that the compiler tools should be usable as a library. There was a not insignificant amount of drama[0] when the folks working on LLD (the then-new LLVM linker) decided they weren't going to follow that pattern. It was another 5ish years before they revisited that decision[1], but it's not a simple task to retrofit a project with proper error handling (exit on error is fine for a binary, but not a library), memory management (leaking is faster than freeing memory for a short lived process, but isn't acceptable in a library), etc. [0]https://discourse.llvm.org/t/lld-status-update-and-performance-chart/43368/8 https://discourse.llvm.org/t/lld-status-update-and-performan... [1]https://discourse.llvm.org/t/rfc-revisiting-lld-as-a-library-design/58445 https://discourse.llvm.org/t/rfc-revisiting-lld-as-a-library...
  - mhh__ 3y ago
    I'd go a step further: the compiler should solely be a way to run the library, no extra code.
    - pjmlp 3y ago
      And packaged in a way that allows for both JIT and AOT workflows.
berkeleynerd 3y ago
This looks like more fun than sudoku or crossword puzzles.
- WalterBright 3y ago
  I can vouch for it being a lot more fun! Doing puzzles is a useless waste of time.
  - klyrs 3y ago
    Not true! Doing puzzles can help mitigate risk for dementia. Probably not necessary for those of us who do puzzling work day to day, but there is a solid benefit to society if people are in the habit of doing puzzles.
    - jraph 3y ago
      > puzzling work You don't say. Appalling, even!
    - WalterBright 3y ago
      Working in programming is often a puzzle, and something useful is accomplished beyond exercising your brain. For an analogy, if you walk to work every day instead of driving, you are accomplishing something useful beyond just the exercising. My dad always tried to get a residence that was about a 15 min walk to work.
      - deleted 3y ago
        [deleted]
WalterBright 3y ago
The author, Razvan Nitu, joined the D community a few years ago and is now a key member of the core Dlang team.
sixthDot 3y ago
The backend/frontend separation sounds right but the "sub" split of the frontend sounds odd. You dont care if in the AST some functions are doing too much. In my opinion the most important is the ability to _drive the compilation_, for example stop after lexical, stop after semantics. Then if too much code got compiled into the *.a that's unfortunate but not dramatic. BTW there's is also the split between the driver and the frontend, but I think this is done.
mhh__ 3y ago
The issue isn't really refactoring. This is kind of like a blind man clapping to echolocate his way around a maze. It's better than nothing but the issue is that the compiler just isn't amenable to non-batch work (e.g. the semantic analysis is all or nothing, it uses WAY too much memory[0], and it can't serialize it's state to the disk). You could already use the frontend as a library, almost no one does. It's not because the API. [0] SDC can pack a lot of types into a few bytes. This is the kind of thing real refactoring allows — this change (packing types, to be specific) is something I would happily chip in and help out with. Memory layout and locality is where performance lies in a compiler (that and doing less work).
- moonchild 3y ago
  'Compilers' should be incremental (and iterative), and the asymptotics are way more important than the constant factors; if the conceptual models are not adequate to effectively express very fine-grain incrementality, no amount of bitpacking is going to save you. https://arxiv.org/pdf/2104.01270.pdf https://arxiv.org/pdf/2104.01270.pdf as an example—this is not perfect, but very serviceable. Being able to serialise is good and valuable and useful, and enables some very interesting things, but you can still do a lot of useful things with just a persistent process working on in-memory structures.
  - mhh__ 3y ago
    Thx for link. Asymptotics do always win but consider but consider the price of the win i.e. bitpacking is cheap (and fun) — not even just an academic exercise, this is where you get to use your hard won intuition.
    - moonchild 3y ago
      Not saying it's not worth it, just, it's strange to put them in the same sentence when one is what makes the difference between usable and completely unusable. I lived with 20-60s dirty build times for 1-line changes at symmetry, solely because of redundant compile-time recomputation—I wouldn't wish that on anybody. 10x would have been on the upper edge of tolerable, but you certainly won't get 10x for 'cheap'—and even so, latency should rightfully be measured in the milliseconds, not seconds. (~10x, incidentally, is quoted as an upper bound for newctfe here https://forum.dlang.org/post/qxiggjwhvadbpdfkidvu@forum.dlang.org https://forum.dlang.org/post/qxiggjwhvadbpdfkidvu@forum.dlan..., had it ever materialised.)
codr7 3y ago
I wanted to like D for a long time, feels like Go without the limitations to me. But it's taking A LONG TIME to become usable.
- Alifatisk 3y ago
  I found the importC feature interesting, I tried the example code (with running the hello world in C using the importC flag) and the compiler threw lots of warnings. https://dlang.org/spec/importc.html#examples https://dlang.org/spec/importc.html#examples
  - WalterBright 3y ago
    I did not get any warnings running it on Ubuntu. What warnings are you seeing? What operating system?
    - Alifatisk 3y ago
      Thanks for checking up, I did it on my Winows 10 machine, it's kinda too late to start it up now and run it but I'll do it tomorrow and post both the code and the error message. Update: I did this quickly on my MBP. Code: #include <stdio.h> #include <stdlib.h> int main() { puts("Hello, world!"); return EXIT_SUCCESS; } Command: dmd test.c Error: ld: address=0x0 points to section(2) with no content in '/Users/james/.asdf/installs/dmd/2.107.0/dmd2/osx/lib/libphobos2.a[3233](config_a98_4c3.o)' clang: error: linker command failed with exit code 1 (use -v to see invocation) Error: linker exited with status 1
      - WalterBright 3y ago
        Thank you. Did you mean to #include stdio.h twice?
        Alifatisk 3y ago
        No I did not mean to include stdio.h twice. I don’t think I am doing that either?

72 points by ingve 3y ago

Given how fast the D compiler is, this might be a nice alternative to something like LuaJIT for an embedded scripting language in other apps.

WalterBright 3y ago
Many people do use D as a scripting language, replacing their use of bash. D programs can be compiled and run in one step. #!/usr/bin/env rdmd import std.stdio; void main() { writeln("Hello, world with automated script running!"); } > ./myprog.d Hello, world with automated script running! https://dlang.org/rdmd.html https://dlang.org/rdmd.html
- thechao 3y ago
  Classically, you wait until your name is said three times, before appearing before the supplicant!
- tripleo1 3y ago
  Do you also have: 1. Statically linked executables (essential but not crucial) 2. Dependency management (import tripleo.rtfm from "git:...#1234567") 3. A good AST API (jdt/roaster, javapoet, etc) 4. Tooling (cf spoon/soot) (I approve this comment, even given it's limited usefulness/expressivity)
  - WalterBright 3y ago
    1. yes 2. no 3. that's what this topic is about 4. yes
  - bachmeier 3y ago
    > 2. Dependency management (import tripleo.rtfm from "git:...#1234567") Not built into the compiler, but Dub, the official package manager distributed with the compiler, should be what I think your concise statement is referring to.
p0nce 3y ago
This is what HipremeEngine does for example. https://github.com/MrcSnm/HipremeEngine https://github.com/MrcSnm/HipremeEngine

UncleEntity 3y ago

Swift does something like this I believe. Not too sure on the details but I remember reading about how you can modify the parser to do...things. Looked interesting.

favorited 3y ago
It's part of the LLVM philosophy that the compiler tools should be usable as a library. There was a not insignificant amount of drama[0] when the folks working on LLD (the then-new LLVM linker) decided they weren't going to follow that pattern. It was another 5ish years before they revisited that decision[1], but it's not a simple task to retrofit a project with proper error handling (exit on error is fine for a binary, but not a library), memory management (leaking is faster than freeing memory for a short lived process, but isn't acceptable in a library), etc. [0]https://discourse.llvm.org/t/lld-status-update-and-performance-chart/43368/8 https://discourse.llvm.org/t/lld-status-update-and-performan... [1]https://discourse.llvm.org/t/rfc-revisiting-lld-as-a-library-design/58445 https://discourse.llvm.org/t/rfc-revisiting-lld-as-a-library...
- mhh__ 3y ago
  I'd go a step further: the compiler should solely be a way to run the library, no extra code.
  - pjmlp 3y ago
    And packaged in a way that allows for both JIT and AOT workflows.

berkeleynerd 3y ago

This looks like more fun than sudoku or crossword puzzles.

WalterBright 3y ago
I can vouch for it being a lot more fun! Doing puzzles is a useless waste of time.
- klyrs 3y ago
  Not true! Doing puzzles can help mitigate risk for dementia. Probably not necessary for those of us who do puzzling work day to day, but there is a solid benefit to society if people are in the habit of doing puzzles.
  - jraph 3y ago
    > puzzling work You don't say. Appalling, even!
  - WalterBright 3y ago
    Working in programming is often a puzzle, and something useful is accomplished beyond exercising your brain. For an analogy, if you walk to work every day instead of driving, you are accomplishing something useful beyond just the exercising. My dad always tried to get a residence that was about a 15 min walk to work.
    - deleted 3y ago
      [deleted]

WalterBright 3y ago

The author, Razvan Nitu, joined the D community a few years ago and is now a key member of the core Dlang team.

sixthDot 3y ago

The backend/frontend separation sounds right but the "sub" split of the frontend sounds odd. You dont care if in the AST some functions are doing too much. In my opinion the most important is the ability to _drive the compilation_, for example stop after lexical, stop after semantics. Then if too much code got compiled into the *.a that's unfortunate but not dramatic. BTW there's is also the split between the driver and the frontend, but I think this is done.

mhh__ 3y ago

The issue isn't really refactoring. This is kind of like a blind man clapping to echolocate his way around a maze. It's better than nothing but the issue is that the compiler just isn't amenable to non-batch work (e.g. the semantic analysis is all or nothing, it uses WAY too much memory[0], and it can't serialize it's state to the disk). You could already use the frontend as a library, almost no one does. It's not because the API. [0] SDC can pack a lot of types into a few bytes. This is the kind of thing real refactoring allows — this change (packing types, to be specific) is something I would happily chip in and help out with. Memory layout and locality is where performance lies in a compiler (that and doing less work).

moonchild 3y ago
'Compilers' should be incremental (and iterative), and the asymptotics are way more important than the constant factors; if the conceptual models are not adequate to effectively express very fine-grain incrementality, no amount of bitpacking is going to save you. https://arxiv.org/pdf/2104.01270.pdf https://arxiv.org/pdf/2104.01270.pdf as an example—this is not perfect, but very serviceable. Being able to serialise is good and valuable and useful, and enables some very interesting things, but you can still do a lot of useful things with just a persistent process working on in-memory structures.
- mhh__ 3y ago
  Thx for link. Asymptotics do always win but consider but consider the price of the win i.e. bitpacking is cheap (and fun) — not even just an academic exercise, this is where you get to use your hard won intuition.
  - moonchild 3y ago
    Not saying it's not worth it, just, it's strange to put them in the same sentence when one is what makes the difference between usable and completely unusable. I lived with 20-60s dirty build times for 1-line changes at symmetry, solely because of redundant compile-time recomputation—I wouldn't wish that on anybody. 10x would have been on the upper edge of tolerable, but you certainly won't get 10x for 'cheap'—and even so, latency should rightfully be measured in the milliseconds, not seconds. (~10x, incidentally, is quoted as an upper bound for newctfe here https://forum.dlang.org/post/qxiggjwhvadbpdfkidvu@forum.dlang.org https://forum.dlang.org/post/qxiggjwhvadbpdfkidvu@forum.dlan..., had it ever materialised.)

codr7 3y ago

I wanted to like D for a long time, feels like Go without the limitations to me. But it's taking A LONG TIME to become usable.

Alifatisk 3y ago
I found the importC feature interesting, I tried the example code (with running the hello world in C using the importC flag) and the compiler threw lots of warnings. https://dlang.org/spec/importc.html#examples https://dlang.org/spec/importc.html#examples
- WalterBright 3y ago
  I did not get any warnings running it on Ubuntu. What warnings are you seeing? What operating system?
  - Alifatisk 3y ago
    Thanks for checking up, I did it on my Winows 10 machine, it's kinda too late to start it up now and run it but I'll do it tomorrow and post both the code and the error message. Update: I did this quickly on my MBP. Code: #include <stdio.h> #include <stdlib.h> int main() { puts("Hello, world!"); return EXIT_SUCCESS; } Command: dmd test.c Error: ld: address=0x0 points to section(2) with no content in '/Users/james/.asdf/installs/dmd/2.107.0/dmd2/osx/lib/libphobos2.a[3233](config_a98_4c3.o)' clang: error: linker command failed with exit code 1 (use -v to see invocation) Error: linker exited with status 1
    - WalterBright 3y ago
      Thank you. Did you mean to #include stdio.h twice?
      - Alifatisk 3y ago
        No I did not mean to include stdio.h twice. I don’t think I am doing that either?