4 ms·

Writing a compiler is still almost always overkill, because implementing the compiler is one of the smallest parts of creating an effective programming language

Writing a compiler is still almost always overkill, because implementing the compiler is one of the smallest parts of creating an effective programming language. Let's say you're starting a project, and you're choosing a language to write it in. You have two options: Language A: 1. Has multiple mature compiler/interpreter implementations. 2. Has lots of tooling built around it. 3. Has extensive documentation. 4. Has a diverse community to hire programmers from. 5. Fits the problem domain okay. Language B: 1. Has one buggy implementation that returns mangled stack traces on errors. See Kernighan's Lever[1]. 2. Has no tooling. 3. The documentation is the compiler. So... no documentation. 4. The only programmer that knows it is the one who wrote the compiler. 5. Fits the problem domain as well as the programmer understood the problem domain when he started writing the compiler. I know which language Dan McKinley[2] would choose. :) You might say, "But my language uses the same homoiconic syntax of Lisp, so the tooling of Lisp carries over, and it doesn't require more documentation than just normal functions, and Lisp programmers can pick it up easily." To which I would respond, "Sounds like you took the long way to implementing a library." I'd level this criticism against a lot of Racket languages, which basically are just a shorthand for a bunch of require statements. I'd rather copy/paste those same require statements. The fact that Lisps make creating a compiler so easy is actually a downside, because it leads people to write compilers without thinking through the full secondary effects of doing so. This is one of many heads of the Hydra that is The Lisp Curse[3]. EDIT: Decided to say more. EDIT 2: Decided to say less. EDIT 3: Always be editing. [1] https://blogs.msdn.microsoft.com/alfredth/2009/01/08/are-you-smart-enough-to-debug-your-own-code/ https://blogs.msdn.microsoft.com/alfredth/2009/01/08/are-you... [2] https://mcfunley.com/choose-boring-technology https://mcfunley.com/choose-boring-technology [3] http://www.winestockwebdesign.com/Essays/Lisp_Curse.html http://www.winestockwebdesign.com/Essays/Lisp_Curse.html

justinmeiners 7y ago
> Sounds like you took the long way to implementing a library. You are right that DSLs should be about as natural as libraries. Linq in C#, regex, and SQL are good examples of DSLs you probably rely on regularly. I suggest looking at the symbolic differentiator and circuit simulator in SICP. It is composed of just a few functions and is far cleaner than a library of functions on a custom data structure.
- kerkeslager 7y ago
  > Linq in C#, regex, and SQL are good examples of DSLs you probably rely on regularly. Right, but with the exception of Linq, those all have addressed all the problems I brought up: 1. Has multiple mature compiler/interpreter implementations. 2. Has lots of tooling built around it. 3. Has extensive documentation. 4. Has a diverse community to hire programmers from. They were able to address these issues because the DSLs solved very common problems that a lot of people had, and even with that incentive, it took a lot of effort from a lot of people to get there. A new DSL you implement to solve your specific issue is not going to trivially address these issues. And Linq is an "exception that proves the rule". Linq has not taken off quite as well as the initial hype would have indicated. I'm not sure where it is now, because I haven't worked on .NET in over 4 years, but I remember times when it was unclear whether Linq would be supported in future Microsoft products. It's very difficult even for an organization with Microsoft's clout to create a DSL which is successful. > I suggest looking at the symbolic differentiator and circuit simulator in SICP. It is composed of just a few functions and is far cleaner than a library of functions on a custom data structure. I suggest looking at the 4 problems mentioned in my posts, and noting that all of them still apply to the SICP example(s) of DSLs.
  - justinmeiners 7y ago
    > 4 problems mentioned in my posts But we are comparing a small library of functions with a DSL of similar complexity. > You might say: But my language uses the same homoiconic syntax of Lisp, so the tooling of Lisp carries over, and it doesn't require more documentation than just normal functions, and Lisp programmers can pick it up easily. Doesn't that address them? Why are those 4 issues more applicable to a Lisp DSL than a library? To learn a Lisp DSL, I read the docs for the relevant ideas, and look at a few examples. What about that process is different than a library, except that it tends to be natural and flexible for certain problems? The SICP examples would be a major pain to implement as a typical library.
TeMPOraL 7y ago
You're still thinking of languages as these huge things that can be offered as products. It's not the way a Lisper thinks about it. > To which I would respond, "Sounds like you took the long way to implementing a library." And I'd respond, that library is the language. There's a spectrum of complexity of what you can call "a language". General-purpose programming languages like you seem to be considering are one end of that spectrum. The other end of that spectrum are the abstraction layers and APIs you're coding to. Take e.g. OpenGL API - you can view it as a set of functions, but once you consider the rules about calling them, you may as well call that a language. So when you're designing a "DSL" in a Lisp, it's literally no different than designing any other module. You have to name things, you have to figure out how they should be used. Lisps just doesn't force you to shoehorn your API into built-in language syntax. It doesn't force you to write "open-file" and "close-file" because your language only has functions and no built-in "with" construct; it lets you add the "with" constructs with four lines of code, making the API cleaner. Most DSLs sit somewhere between "API for a stack" and "API for OpenGL". They're just code - code that in some cases happens to run at compile time and operate on other code. But on the user level, on the API level, it's no different at all from using any other library. A documented DSL is no harder to use than a documented function library; a function library lacking documentation is no easier to use than similarly undocumented macro-based DSL. Some people seem to have an irrational fear of "less mainstream" things, even though they're not fundamentally more difficult than the mainstream things. I've been working with Lisp professionally for a while now, and I've seen plenty of really complex and hairy DSLs and had to debug some when they broke. Fixing a broken macro or its use is not fundamentally different from debugging Java code of similar complexity, but when the DSL works, it can vastly improve readability of the code that uses it.
- kerkeslager 7y ago
  > You're still thinking of languages as these huge things that can be offered as products. No, I'm really really not. You're assuming my criticisms are because I don't understand how DSLs work in Lisp, but I would request that you not make that assumption. > And I'd respond, that library is the language. There's a spectrum of complexity of what you can call "a language". General-purpose programming languages like you seem to be considering are one end of that spectrum. The other end of that spectrum are the abstraction layers and APIs you're coding to. Take e.g. OpenGL API - you can view it as a set of functions, but once you consider the rules about calling them, you may as well call that a language. You might as well not call it a language, though. I understand what you're saying about the spectrum, and I do agree that a library of functions is a DSL, but a library of functions is a special case of a DSL that doesn't have any of the usual downsides of a DSL. You can't really use a library of functions to justify the complexity of other DSLs. A library of macros isn't a library of functions, and a library of macros has all the downsides I talked about. And if you're talking about DSLs as being part of what makes Lisp special, then you're not talking about libraries of functions, because almost any language can do that. Let's not get lost in a semantic argument here: when we're talking about DSLs, libraries of functions aren't the central example of a DSL that we're talking about. > So when you're designing a "DSL" in a Lisp, it's literally no different than designing any other module. You have to name things, you have to figure out how they should be used. Lisps just doesn't force you to shoehorn your API into built-in language syntax. It doesn't force you to write "open-file" and "close-file" because your language only has functions and no built-in "with" construct; it lets you add the "with" constructs with four lines of code, making the API cleaner. Not being forced to shoehorn your API into the built-in language syntax is exactly the problem I'm talking about. Either you shoehorn your API into the built-in language syntax, or your users have to learn a new syntax, which mangles your stack traces through the macro expansions, breaks your tooling, isn't documented, and is only understood by you. Incidentally, Lisp (Common Lisp) definitely implements a with-like syntax. It's been a while since I used it, but I remember this clearly. > A documented DSL is no harder to use than a documented function library; a function library lacking documentation is no easier to use than similarly undocumented macro-based DSL. That is very much your opinion, and not one that is borne out by my experience. > Some people seem to have an irrational fear of "less mainstream" things, even though they're not fundamentally more difficult than the mainstream things. Again, please don't make assumptions about me. I'm not avoiding macros because they're less mainstream, I'm avoiding them because I've experienced a lot of pain from them. > Fixing a broken macro or its use is not fundamentally different from debugging Java code of similar complexity, but when the DSL works, it can vastly improve readability of the code that uses it. It's true that fixing a broken macro is not fundamentally different from debugging Java code of similar complexity. You can write bad code in any language. But if we're trying to write good code, limiting the complexity is a huge priority, and you can't deny that introducing macros introduces complexity. Comparing to Java misses the point: we can look at Lisp with macro-based DSLs versus Lisp without macro-based DSLs, and see the benefits there. Just to be clear: my criticisms here aren't of Lisp as a whole. I've been working a lot in Racket lately and there are a lot of things I like about it--it's batteries-included in the way Python used to be. I really hope Racket takes off more.
jimbokun 7y ago
"Sounds like you took the long way to implementing a library." And of course in Lisp, writing a compiler is just as easy as writing a library.
- kerkeslager 7y ago
  ...which doesn't address any of what I said.
angleofrepose 7y ago
> 3. The documentation is the compiler. So... no documentation. I don't understand this one. Shouldn't small compilers for small languages be excellent documentation? I can't really ground the point or my response, but it seems that clever tooling should be able to analyze the compiler to create good documentation. Information could include type annotation, a timeline of the creation of features of the DSL or any other number of things that could be kept during the creation of the compiler. > Sounds like you took the long way to implementing a library DSLs aren't Libraries. When would you want one over the other? The Racket languages I've seen have impressed me with their simplicity and focused scope, I would never think the same of C with includes or js with requires. The point is to be writing a file which is perfectly scoped to the issue, right? I imagine it would suck to "write" html if it were a library in C, rather than a file format with all the benefits that comes from being its own thing.

justinmeiners 7y ago

> Sounds like you took the long way to implementing a library. You are right that DSLs should be about as natural as libraries. Linq in C#, regex, and SQL are good examples of DSLs you probably rely on regularly. I suggest looking at the symbolic differentiator and circuit simulator in SICP. It is composed of just a few functions and is far cleaner than a library of functions on a custom data structure.

kerkeslager 7y ago
> Linq in C#, regex, and SQL are good examples of DSLs you probably rely on regularly. Right, but with the exception of Linq, those all have addressed all the problems I brought up: 1. Has multiple mature compiler/interpreter implementations. 2. Has lots of tooling built around it. 3. Has extensive documentation. 4. Has a diverse community to hire programmers from. They were able to address these issues because the DSLs solved very common problems that a lot of people had, and even with that incentive, it took a lot of effort from a lot of people to get there. A new DSL you implement to solve your specific issue is not going to trivially address these issues. And Linq is an "exception that proves the rule". Linq has not taken off quite as well as the initial hype would have indicated. I'm not sure where it is now, because I haven't worked on .NET in over 4 years, but I remember times when it was unclear whether Linq would be supported in future Microsoft products. It's very difficult even for an organization with Microsoft's clout to create a DSL which is successful. > I suggest looking at the symbolic differentiator and circuit simulator in SICP. It is composed of just a few functions and is far cleaner than a library of functions on a custom data structure. I suggest looking at the 4 problems mentioned in my posts, and noting that all of them still apply to the SICP example(s) of DSLs.
- justinmeiners 7y ago
  > 4 problems mentioned in my posts But we are comparing a small library of functions with a DSL of similar complexity. > You might say: But my language uses the same homoiconic syntax of Lisp, so the tooling of Lisp carries over, and it doesn't require more documentation than just normal functions, and Lisp programmers can pick it up easily. Doesn't that address them? Why are those 4 issues more applicable to a Lisp DSL than a library? To learn a Lisp DSL, I read the docs for the relevant ideas, and look at a few examples. What about that process is different than a library, except that it tends to be natural and flexible for certain problems? The SICP examples would be a major pain to implement as a typical library.

TeMPOraL 7y ago

You're still thinking of languages as these huge things that can be offered as products. It's not the way a Lisper thinks about it. > To which I would respond, "Sounds like you took the long way to implementing a library." And I'd respond, that library is the language. There's a spectrum of complexity of what you can call "a language". General-purpose programming languages like you seem to be considering are one end of that spectrum. The other end of that spectrum are the abstraction layers and APIs you're coding to. Take e.g. OpenGL API - you can view it as a set of functions, but once you consider the rules about calling them, you may as well call that a language. So when you're designing a "DSL" in a Lisp, it's literally no different than designing any other module. You have to name things, you have to figure out how they should be used. Lisps just doesn't force you to shoehorn your API into built-in language syntax. It doesn't force you to write "open-file" and "close-file" because your language only has functions and no built-in "with" construct; it lets you add the "with" constructs with four lines of code, making the API cleaner. Most DSLs sit somewhere between "API for a stack" and "API for OpenGL". They're just code - code that in some cases happens to run at compile time and operate on other code. But on the user level, on the API level, it's no different at all from using any other library. A documented DSL is no harder to use than a documented function library; a function library lacking documentation is no easier to use than similarly undocumented macro-based DSL. Some people seem to have an irrational fear of "less mainstream" things, even though they're not fundamentally more difficult than the mainstream things. I've been working with Lisp professionally for a while now, and I've seen plenty of really complex and hairy DSLs and had to debug some when they broke. Fixing a broken macro or its use is not fundamentally different from debugging Java code of similar complexity, but when the DSL works, it can vastly improve readability of the code that uses it.

kerkeslager 7y ago
> You're still thinking of languages as these huge things that can be offered as products. No, I'm really really not. You're assuming my criticisms are because I don't understand how DSLs work in Lisp, but I would request that you not make that assumption. > And I'd respond, that library is the language. There's a spectrum of complexity of what you can call "a language". General-purpose programming languages like you seem to be considering are one end of that spectrum. The other end of that spectrum are the abstraction layers and APIs you're coding to. Take e.g. OpenGL API - you can view it as a set of functions, but once you consider the rules about calling them, you may as well call that a language. You might as well not call it a language, though. I understand what you're saying about the spectrum, and I do agree that a library of functions is a DSL, but a library of functions is a special case of a DSL that doesn't have any of the usual downsides of a DSL. You can't really use a library of functions to justify the complexity of other DSLs. A library of macros isn't a library of functions, and a library of macros has all the downsides I talked about. And if you're talking about DSLs as being part of what makes Lisp special, then you're not talking about libraries of functions, because almost any language can do that. Let's not get lost in a semantic argument here: when we're talking about DSLs, libraries of functions aren't the central example of a DSL that we're talking about. > So when you're designing a "DSL" in a Lisp, it's literally no different than designing any other module. You have to name things, you have to figure out how they should be used. Lisps just doesn't force you to shoehorn your API into built-in language syntax. It doesn't force you to write "open-file" and "close-file" because your language only has functions and no built-in "with" construct; it lets you add the "with" constructs with four lines of code, making the API cleaner. Not being forced to shoehorn your API into the built-in language syntax is exactly the problem I'm talking about. Either you shoehorn your API into the built-in language syntax, or your users have to learn a new syntax, which mangles your stack traces through the macro expansions, breaks your tooling, isn't documented, and is only understood by you. Incidentally, Lisp (Common Lisp) definitely implements a with-like syntax. It's been a while since I used it, but I remember this clearly. > A documented DSL is no harder to use than a documented function library; a function library lacking documentation is no easier to use than similarly undocumented macro-based DSL. That is very much your opinion, and not one that is borne out by my experience. > Some people seem to have an irrational fear of "less mainstream" things, even though they're not fundamentally more difficult than the mainstream things. Again, please don't make assumptions about me. I'm not avoiding macros because they're less mainstream, I'm avoiding them because I've experienced a lot of pain from them. > Fixing a broken macro or its use is not fundamentally different from debugging Java code of similar complexity, but when the DSL works, it can vastly improve readability of the code that uses it. It's true that fixing a broken macro is not fundamentally different from debugging Java code of similar complexity. You can write bad code in any language. But if we're trying to write good code, limiting the complexity is a huge priority, and you can't deny that introducing macros introduces complexity. Comparing to Java misses the point: we can look at Lisp with macro-based DSLs versus Lisp without macro-based DSLs, and see the benefits there. Just to be clear: my criticisms here aren't of Lisp as a whole. I've been working a lot in Racket lately and there are a lot of things I like about it--it's batteries-included in the way Python used to be. I really hope Racket takes off more.

jimbokun 7y ago

"Sounds like you took the long way to implementing a library." And of course in Lisp, writing a compiler is just as easy as writing a library.

kerkeslager 7y ago
...which doesn't address any of what I said.

angleofrepose 7y ago

> 3. The documentation is the compiler. So... no documentation. I don't understand this one. Shouldn't small compilers for small languages be excellent documentation? I can't really ground the point or my response, but it seems that clever tooling should be able to analyze the compiler to create good documentation. Information could include type annotation, a timeline of the creation of features of the DSL or any other number of things that could be kept during the creation of the compiler. > Sounds like you took the long way to implementing a library DSLs aren't Libraries. When would you want one over the other? The Racket languages I've seen have impressed me with their simplicity and focused scope, I would never think the same of C with includes or js with requires. The point is to be writing a file which is perfectly scoped to the issue, right? I imagine it would suck to "write" html if it were a library in C, rather than a file format with all the benefits that comes from being its own thing.