3 ms·

> The downside is that you end up building a custom language in the process You'll do it in many complex programs in many programming languages. Creating the n

by lispm 8y ago

> The downside is that you end up building a custom language in the process You'll do it in many complex programs in many programming languages. Creating the necessary infrastructure in a programming language is called 'Greenspunning'. If we for example need to work with 'state machines', I would implement in Lisp the machinery for it and give it a nicer source code user interface via some macro. The effect could be that the particular state machine definition sees a reduction in code size by, say, factor ten. Now productivity may depend on the size of the code -> productivity goes up. Readability and maintainability of the code goes up, too. Now, if we program the same thing in Java or C++ - would we want to manually code the state machine without linguistic abstraction, where we would like to get shorter source code? Probably not. There are now several ways to work around this. One would be to have an XML scheme describing state machines and a translator to Java code. Or defining a custom external DSL for it. This approach is quite popular. Now you need to learn Java and your custom configuration language, plus its implementation. Lisp just has a different solution for it: developers tend to implement kind of an internal macro-based DSL for these problems. The solution for that may look different from what Java or C++ uses - but they also need to abstract the code. If the code does not -> then you have here the reason why Lisp teams are smaller -> they are more productive. If a C++ program gets large enough, team size may go up ten times... I once heard an example for that from a Lucent manager who worked with a team of hundred people using Lisp.

desc 8y ago
Good point, but creating infrastructure doesn't necessarily mean creating idioms on the compiler level: neither DSLs nor macros. I would reject both without solid evidence that a good method and object API in the host language couldn't solve the problem first. My approach in C++, Java, C#, JS, etc would be to provide classes and methods which assist in building the state machine definition. If their existing compiler capabilities can be used to optimise it, so much the better, but I'd stick to the host language unless I had other reasons to delve into building my own language. As a concrete example: some time ago we tried creating a domain-specific language to define some fairly hefty structures in our application. The DSL was more concise and was intended to provide a 'pit of success' for adding new such structures, by adding some extra verification and getting rid of boilerplate which might get miswritten or forgotten. Some years later we finally found the time to throw out the DSL and replace it with straightforward (albeit slightly more verbose) compiled code with helper methods. Which, funnily enough, was a direct translation of what the DSL interpreter/compiler was doing anyway under the hood. We threw out an extra build/start-up phase, a few thousand lines of pointless glue, and a massive debugging headache. We gained the strong typing which our platform provides by default. The mistake we made was that the DSL was merely an abbreviation of code which was otherwise running in the exact same context as everything else. It was a syntax hack. It was something which could be done better by simply making proper use of the idioms and capabilities of our host platform, and ignoring the syntax entirely. If you don't need to mess with syntax, you probably don't need metaprogramming. And syntax is not really the hard part of programming (if it's consistent), it should be just a minor and necessary hurdle before you get to the important things, so you probably don't need to mess with syntax either. Boilerplate and bloat can be solved with good API design, but only if the pointless repetition is really pointless repetition. If it's not easy to fix with a few helper functions, odds are it's not actually as common and repetitive as you'd like to believe. Creating an entirely new language just to slim things down a bit is rarely appropriate for a single project. The caveat here of course is that, with Lisp, there's no difference between macros and API (the attempt to differentiate is itself meaningless). And that's fine, but it's not the case for other platforms which lack Lisp-style macros, and it doesn't help with accessibility. Layering things is an important part of how we think and sticking to common concepts in a given layer are a useful tradeoff. (It was rather fun to abuse C# `this[]` getters to force assignment of a function result via the compiler, at one stage. While useful as a temporary refactoring tool, I'm glad that hack has finally left our codebase.) (The dark side of 'no DSL' is the fluent interface. These are really hard to design well, and most are terrible. The examples always read nicely of course, because they're basically the design documents.) (I'm also aware that there are valid use cases for building DSLs. I've just never encountered such a use case myself...)
- lispm 8y ago
  > provide classes and methods which assist in building the state machine definition Now you shift the complexity to a language which is potentially bloated and less suitable to express domain level concepts in a concise way. We've see a lot of OO-architectures which try to recover flexibility by providing complex meta-level mechanisms. For a state-machine one would implement kind of an interpreter over an OO-data model - or a code generator from that oo-model -> greenspunning. > It was something which could be done better by simply making proper use of the idioms and capabilities of our host platform, and ignoring the syntax entirely For Lisp this would be natural. One can easily hide an implementation behind a domain-level descriptive representation of the problem. The distance between both is very small. It's actually what I would try to approach: working on a descriptive level with domain concepts and hiding the implementation. > Creating an entirely new language just to slim things down a bit is rarely appropriate for a single project. That depends on the 'single project' and its size. Larger applications usually contain a multitude of such tailored notations and machineries to implement them. > The caveat here of course is that, with Lisp, there's no difference between macros and API The API consists of exported and documented macros. > Layering things is an important part of how we think and sticking to common concepts in a given layer are a useful tradeoff. Sure. If you look at the Common Lisp Object System, it was originally an extension to Common Lisp to implement an object system with classes, functions, methods, inheritance, etc. The original implementation was in several layers: The lowest level was a layer of objects. CLOS implemented in terms of itself. Like the definition of a class for classes. It's a bit unusual, since it is an implementation in itself. The next layer was a layer of extensible functions and classes. Like creating classes by calling functions. The developer layer is a level of macros where, for example, classes are specified via macros. The macros assemble the necessary language constructs - like how to descriptively define classes in a convenient notation. The layers are documented and the lower layers are collections of protocols over classes and functions. This layered language approach in Lisp is described in the book 'The Art of the Meta-Object Protocol', short AMOP. https://en.wikipedia.org/wiki/The_Art_of_the_Metaobject_Protocol https://en.wikipedia.org/wiki/The_Art_of_the_Metaobject_Prot...

by lispm 8y ago

desc 8y ago

Good point, but creating infrastructure doesn't necessarily mean creating idioms on the compiler level: neither DSLs nor macros. I would reject both without solid evidence that a good method and object API in the host language couldn't solve the problem first. My approach in C++, Java, C#, JS, etc would be to provide classes and methods which assist in building the state machine definition. If their existing compiler capabilities can be used to optimise it, so much the better, but I'd stick to the host language unless I had other reasons to delve into building my own language. As a concrete example: some time ago we tried creating a domain-specific language to define some fairly hefty structures in our application. The DSL was more concise and was intended to provide a 'pit of success' for adding new such structures, by adding some extra verification and getting rid of boilerplate which might get miswritten or forgotten. Some years later we finally found the time to throw out the DSL and replace it with straightforward (albeit slightly more verbose) compiled code with helper methods. Which, funnily enough, was a direct translation of what the DSL interpreter/compiler was doing anyway under the hood. We threw out an extra build/start-up phase, a few thousand lines of pointless glue, and a massive debugging headache. We gained the strong typing which our platform provides by default. The mistake we made was that the DSL was merely an abbreviation of code which was otherwise running in the exact same context as everything else. It was a syntax hack. It was something which could be done better by simply making proper use of the idioms and capabilities of our host platform, and ignoring the syntax entirely. If you don't need to mess with syntax, you probably don't need metaprogramming. And syntax is not really the hard part of programming (if it's consistent), it should be just a minor and necessary hurdle before you get to the important things, so you probably don't need to mess with syntax either. Boilerplate and bloat can be solved with good API design, but only if the pointless repetition is really pointless repetition. If it's not easy to fix with a few helper functions, odds are it's not actually as common and repetitive as you'd like to believe. Creating an entirely new language just to slim things down a bit is rarely appropriate for a single project. The caveat here of course is that, with Lisp, there's no difference between macros and API (the attempt to differentiate is itself meaningless). And that's fine, but it's not the case for other platforms which lack Lisp-style macros, and it doesn't help with accessibility. Layering things is an important part of how we think and sticking to common concepts in a given layer are a useful tradeoff. (It was rather fun to abuse C# `this[]` getters to force assignment of a function result via the compiler, at one stage. While useful as a temporary refactoring tool, I'm glad that hack has finally left our codebase.) (The dark side of 'no DSL' is the fluent interface. These are really hard to design well, and most are terrible. The examples always read nicely of course, because they're basically the design documents.) (I'm also aware that there are valid use cases for building DSLs. I've just never encountered such a use case myself...)

lispm 8y ago
> provide classes and methods which assist in building the state machine definition Now you shift the complexity to a language which is potentially bloated and less suitable to express domain level concepts in a concise way. We've see a lot of OO-architectures which try to recover flexibility by providing complex meta-level mechanisms. For a state-machine one would implement kind of an interpreter over an OO-data model - or a code generator from that oo-model -> greenspunning. > It was something which could be done better by simply making proper use of the idioms and capabilities of our host platform, and ignoring the syntax entirely For Lisp this would be natural. One can easily hide an implementation behind a domain-level descriptive representation of the problem. The distance between both is very small. It's actually what I would try to approach: working on a descriptive level with domain concepts and hiding the implementation. > Creating an entirely new language just to slim things down a bit is rarely appropriate for a single project. That depends on the 'single project' and its size. Larger applications usually contain a multitude of such tailored notations and machineries to implement them. > The caveat here of course is that, with Lisp, there's no difference between macros and API The API consists of exported and documented macros. > Layering things is an important part of how we think and sticking to common concepts in a given layer are a useful tradeoff. Sure. If you look at the Common Lisp Object System, it was originally an extension to Common Lisp to implement an object system with classes, functions, methods, inheritance, etc. The original implementation was in several layers: The lowest level was a layer of objects. CLOS implemented in terms of itself. Like the definition of a class for classes. It's a bit unusual, since it is an implementation in itself. The next layer was a layer of extensible functions and classes. Like creating classes by calling functions. The developer layer is a level of macros where, for example, classes are specified via macros. The macros assemble the necessary language constructs - like how to descriptively define classes in a convenient notation. The layers are documented and the lower layers are collections of protocols over classes and functions. This layered language approach in Lisp is described in the book 'The Art of the Meta-Object Protocol', short AMOP. https://en.wikipedia.org/wiki/The_Art_of_the_Metaobject_Protocol https://en.wikipedia.org/wiki/The_Art_of_the_Metaobject_Prot...