8 ms·

A programmable programming language (2018) [pdf]

68 points by azhenley 7y ago

sam_lowry_ 7y ago
LISP?
- DannyB2 7y ago
  Racket is a Lisp. One that facilitates building other languages.
  - schwurb 7y ago
    You could characterize Racket as THE lisp when it comes to well-behaved and well-researched metaprograming.
- kleer001 7y ago
  That was my first thought.
yters 7y ago
I used Racket for a couple projects, but meta programming hasn't turned out to be super useful. When I did use meta programming, I got pretty confused when returning to my code after leaving it for awhile. I know PG claims meta programming was the secret to ViaWeb's success, but I'm not sure how exactly that was the case, and even if true whether that is still relevant today. I have found Python to hit the sweet spot much better in terms of rapid prototyping, and providing a runway to making the prototype more rigorous and maintainable.
- skohan 7y ago
  > I have found Python to hit the sweet spot much better in terms of rapid prototyping, and providing a runway to making the prototype more rigorous and maintainable. I've found Swift to be extremely good for this. It might be hair slower than python to get a project up and running in that very early stage, but it has a similar feeling of "writing the intent of the program not the syntax of the language" and the type system and compile time checking help alleviate a lot of the growing pains you might run into when a project starts getting medium-sized.
- wry_discontent 7y ago
  I've found Python to inhibit me a lot from moving quickly past the very initial stages. I'm fond of Clojure for that, but it tends to de-emphasize macros as a good way of doing things. Most things are just regular fns.
  - yters 7y ago
    Do you find it is easy to come back to your Clojure code and understand what is going on?
    - fctorial 7y ago
      Yeah, tried it on a year old code, looks fine.
    - wry_discontent 7y ago
      The issue with that, I've found, is that you do actually have to read the code. I do JS professionally, and you can definitely tell some of what's going on just from the shape of the code, but I've found Clojure doesn't do that. That said, after a couple minutes looking at something, it's usually clear.
- Ididntdothis 7y ago
  “I know PG claims meta programming was the secret to ViaWeb's success, but I'm not sure how exactly that was the case, and even if true whether that is still relevant today” I can totally see this if you are alone or a very small tight team is working on the code. You will know what’s going on but once you start adding more people it will be very hard for them to work with such a codebase. I have done some really elegant stuff that made me super productive but looking at the code three years later I myself can barely figure out how it works or how to make changes. other people have almost no chance. Once you grow it’s probably good to stick to paradigms a lot of people in the Industry are familiar with.
  - phlakaton 7y ago
    > I have done some really elegant stuff that made me super productive but looking at the code base three years later I myself can barely figure out how it works or how to make changes. Could well apply to me and some of my old C++ projects. Well, except for the super-productive part. PG would say that it's not that the abstractions afforded by metaprogramming are incomprehensible. It's just that, if you are a habitual Blub programmer, everything up the power curve from you is always going to appear weird and hairy. This sort of argument is a terrible one to have with generalizations, though. You should look at specific cases where metaprogramming has been used and analyze them on a case-by-case basis. Sure, a bad, leaky, poorly-documented abstraction is going to lead to grief. On the other hand, umpteen thousands of programmers use the metamagic of Rails on a daily basis and barely bat an eyelash. (Or so it seems to me.)
    - Ididntdothis 7y ago
      Rails is a great example. For me it’s one level further up from simple meta programming. Writing stuff for yourself is one thing but writing stuff that’s universally usable for a lot of people is much harder.
- jlarocco 7y ago
  After years of using Common Lisp, my opinion on "meta-programming" is that it's wonderful when used sparingly and in just the right way, but a bad idea 99% of the time. If it's not completely obvious what (not how) a macro does just by looking at the code that calls it, then it's almost certainly a bad macro. For example, there's a CL library, iterate, that introduces several new iteration constructs. There are a bunch of complicated macros under the hood, but as an end user I never have to think about them, and the provided interface is obvious, even without knowing anything about the iterate library. For example: (iterate (for (key . item) in alist) (collect key into keys) (collect item into items) (finally (return (values keys items)))) I would definitely consider, "Just because you can doesn't mean you should" to be the number one rule of metaprogramming.
- ralphc 7y ago
  I think of Lisp vs. Python similarly to PostScript vs PDF. In PostScript you have a full language, can do anything but you often have to do it yourself. PDF's aren't extensible but have _just about_ all the functionality you need to produce a document. An example of macros you often see is surrounding a block of code with open/close pairs to make sure you don't exit the code leaving dangling resources. File open/close, database connections, etc. Python has the __enter__ and __exit__ protocols to handle that. You can't do every bit of metaprogramming in Python, but over the years they've covered close to every contingency a developer comes across.
  - lispm 7y ago
    Lisp uses advising for that. In the modern form these are :before, :after and :around methods in CLOS. The difference between some Lisps and CPython is this: Lisp is its own implementation language and provides lower level control flow constructs, which can be compiled to efficient code.
- pdonis 7y ago
  > I know PG claims meta programming was the secret to ViaWeb's success, but I'm not sure how exactly that was the case IIRC, PG said the advantage of Lisp was being able to add features much more quickly than the competition, so that (a) they could be adding features almost daily that the competition would take weeks or months to match, and (b) whenever the competition added a new feature that looked like a good idea, they could implement it within a day or two, so by the time the press came around to ask them if they had it too, they could say yes. I also remember him saying that Lisp macros were about 25% of Viaweb's code, so a significant part of the advantage of Lisp was metaprogamming. > even if true whether that is still relevant today. It's probably not that much of an advantage any more if your competition is other startups, since they probably have access to basically the same metaprogramming tools that you do. It might still be a significant advantage if your competition is large, mature businesses whose business model you are trying to disrupt. > I have found Python to hit the sweet spot much better in terms of rapid prototyping, and providing a runway to making the prototype more rigorous and maintainable. I have found this to be true as well. But I also find that I do a significant amount of metaprogramming in Python, and one of the key things to transitioning to more rigorous and maintainable code is to make sure that whatever metaprogramming needs to stay there for the long term is packaged into a library or a tool whose internals won't need to change.
  - alwalo 7y ago
    Could you write something more on metaprogramming in Python ? And how does this compare with Lisp, what do you use it for ? Any good resources to learn this ?
    - pdonis 7y ago
      Metaprogramming is a fairly broad term, but some of the kinds of things I often find myself doing in Python are: - Writing functions that make functions: often I'll find that I want a family of functions that all follow a the same general pattern. In Lisp this would generally be done with a macro; in Python it can usually be done by writing a function that returns a function. The "maker" function will generally have some parameters that fill in specific parts of the general pattern; the function that gets returned is therefore a closure over those parameters (i.e., it has specific values of them filled in). Doing this with a "maker" function is less error prone (particularly if there are a large number of variations on the general pattern) and follows the DRY (don't repeat yourself) principle, which I think is a very good one. - Writing decorators that add a particular common behavior (such as logging or debug output) to a variety of different functions or classes or methods. This is a common case. (In fact one could argue that decorators in general are examples of metaprogramming.) - Using data to construct code. Database programming is a common application of this: for example, I have a database library I commonly use (it's on GitLab here: https://gitlab.com/pdonis/plib3-dbtools https://gitlab.com/pdonis/plib3-dbtools) which constructs typed namedtuples specific to a database table's fields, and wraps rows returned from queries in them for easier field access. More complicated database ORMs use similar techniques. Unfortunately I don't know of any good resources on metaprogramming in general.
agalunar 7y ago
One of my best friends did his Masters work designing a programming language that can be modified on the fly; I'd highly recommend the paper: https://dspace.mit.edu/bitstream/handle/1721.1/113144/1017990524-MIT.pdf https://dspace.mit.edu/bitstream/handle/1721.1/113144/101799...
- nielsbot 7y ago
  The API for building grammars seems straightforward to use.
jimbob45 7y ago
I feel like there's something so comforting about having a concrete language you can rest your back up against though. Once you start throwing in #DEFINEs and typedefs, I start to question whether I can trust anything I'm seeing on the screen to not be hiding crucial secrets from me. Maybe it's just me.
- kd0amg 7y ago
  You don't know what all those library calls do behind the scenes either.
- bob1029 7y ago
  Having constraints imposed upon me by my tools is the only way anything ever gets done. If you put me in front of a blank canvas, and then tell me I can alter the nature of the canvas itself to whatever extent I desire (I.e. in hopes of making using it easier), I'll never get to the point where I'm actually painting on it. Or, perhaps I do manage to produce an end result, but no one else can understand what it means. Even with a mountain of constraints, there is still usually far too much subjectivity in software development. This language seems to me to be moving the needle in a very wrong direction. Having a stable basis that is standardized and everyone can (mostly) agree upon is the foundation for growing a technical team or community. Being able to type "LINQ GroupBy Select examples" into your preferred search engine of choice and receive actionable advice is clearly an advantage for productivity. Sure, in a purely academic sense there is almost certainly a more ideal solution. But, if you are seeking to actually get things done, constraints are the name of the game.
  - discreteevent 7y ago
    I remember some interview with Ulrich Schnauss (electronic artist) where he went back to plain synthesisers for the same reason. Too many things to tweak meant he wasn't able to focus on the piece.
- zomglings 7y ago
  What you said really resonates with me - especially re: resting your back up against it. Just built my first single-page website with React and I ran up against the new Javascript ES6 import system. It makes no sense that people keep introducing this kind of unnecessary complexity into their tools. What was wrong with good old "require"? This is just one instance among many many many. It feels like I'm aging a year every day.
Koshkin 7y ago
The usefulness of this depends on how you want to solve the problem at hand. One approach to programming is to create a DSL for the problem domain and then solve the problem using that DSL. Languages (such as Common Lisp) that include advanced means of metaprogramming are perfect from this perspective. Otherwise, one has to create an API to approximate the DSL - which could be rather hairy, so one may try to manage the complexity of this particular representation by either using a preprocessor or even, as an extreme measure, by writing a separate language processor (compiler or interpreter) for the DSL.
jmull 7y ago
I feel like general purpose programming languages very widely already let you express DSLs, with semantics, names, structure, etc. appropriate to the domain. What they do restrict, to various degrees, is the syntax. So what we're talking about here is custom/customizable syntax to go with your DSL. That can be nice, but it comes at a high cost -- e.g., developers will have to understand the syntax, it will need to be maintained and extended as requirements grow or change. Of course it's worth it in some cases, and tools to help do it are great when it is. But I just don't see that there's a general problem that we don't have enough custom language syntax to deal with.
bitwize 7y ago
Meh, in practice most people will use JavaScript and Babel.
- iLemming 7y ago
  Yeah, and that is a problem with the majority of programmers. They readily choose a seemingly easy path instead of something that leads to simplicity. Lisp is a fantastic idea that (even after over six decades) is still very much relevant today. Once you get used to program with a true REPL, every other (non-lispy) language starts feeling like frustratingly over-engineered, bloated, and unnecessarily complicated. Waiting for compilation cycles and annoying syntax quirks would slowly turn what once was a joy into a "just a job." It feels incredibly liberating and joyous to be able to grab any part of your program and without any preceding ceremony just to "eval" it and see what happens. And let me prematurely answer the comments that probably will be thrown here. No, other languages (most of them) don't have real REPLs. What they have, at best can be categorized as "interactive shells". Younger folks who are learning, juniors, seasoned developers who for one or another reason, never had any exposure to a Lisp, please do yourselves a favor. Try Racket, Clojure, Clojurescript, Fennel, Chez. Give Lisp an honest, heartfelt attempt to learn it. Follow the work of people like Harold Abelson and Gerald Sussman, Matthias Felleisen, Dan Friedman, Rich Hickey, Guy Steele, Richard Gabriel, Alan Kay, and many others. Once you attain familiarity with a Lisp to the sufficient level, perhaps then you will get it. You will see that it is indeed a lifetime investment that would allow you to reap the rewards for the entirety of your career. I am telling you all this, not from the top of some ivory tower. I never went to CS academia. Like most of you, I am altogether a self-taught programmer. I am honestly begging you not to make the same mistake that I did. For many years I have ignored Lisp due to my self-righteous ignorance. Unknowingly, I delayed my true growth, and I sincerely regret it.
  - lispm 7y ago
    > No, other languages (most of them) don't have real REPLs. What they have, at best can be categorized as "interactive shells". > Try Racket, Clojure, Clojurescript, Fennel, Chez. Give Lisp an honest, heartfelt attempt to learn it. Compared to actual Lisps, some of these have mostly interactive shells. https://www.youtube.com/watch?v=o4-YnLpLgtk https://www.youtube.com/watch?v=o4-YnLpLgtk

68 points by azhenley 7y ago

sam_lowry_ 7y ago

LISP?

DannyB2 7y ago
Racket is a Lisp. One that facilitates building other languages.
- schwurb 7y ago
  You could characterize Racket as THE lisp when it comes to well-behaved and well-researched metaprograming.
kleer001 7y ago
That was my first thought.

yters 7y ago

I used Racket for a couple projects, but meta programming hasn't turned out to be super useful. When I did use meta programming, I got pretty confused when returning to my code after leaving it for awhile. I know PG claims meta programming was the secret to ViaWeb's success, but I'm not sure how exactly that was the case, and even if true whether that is still relevant today. I have found Python to hit the sweet spot much better in terms of rapid prototyping, and providing a runway to making the prototype more rigorous and maintainable.

skohan 7y ago
> I have found Python to hit the sweet spot much better in terms of rapid prototyping, and providing a runway to making the prototype more rigorous and maintainable. I've found Swift to be extremely good for this. It might be hair slower than python to get a project up and running in that very early stage, but it has a similar feeling of "writing the intent of the program not the syntax of the language" and the type system and compile time checking help alleviate a lot of the growing pains you might run into when a project starts getting medium-sized.
wry_discontent 7y ago
I've found Python to inhibit me a lot from moving quickly past the very initial stages. I'm fond of Clojure for that, but it tends to de-emphasize macros as a good way of doing things. Most things are just regular fns.
- yters 7y ago
  Do you find it is easy to come back to your Clojure code and understand what is going on?
  - fctorial 7y ago
    Yeah, tried it on a year old code, looks fine.
  - wry_discontent 7y ago
    The issue with that, I've found, is that you do actually have to read the code. I do JS professionally, and you can definitely tell some of what's going on just from the shape of the code, but I've found Clojure doesn't do that. That said, after a couple minutes looking at something, it's usually clear.
Ididntdothis 7y ago
“I know PG claims meta programming was the secret to ViaWeb's success, but I'm not sure how exactly that was the case, and even if true whether that is still relevant today” I can totally see this if you are alone or a very small tight team is working on the code. You will know what’s going on but once you start adding more people it will be very hard for them to work with such a codebase. I have done some really elegant stuff that made me super productive but looking at the code three years later I myself can barely figure out how it works or how to make changes. other people have almost no chance. Once you grow it’s probably good to stick to paradigms a lot of people in the Industry are familiar with.
- phlakaton 7y ago
  > I have done some really elegant stuff that made me super productive but looking at the code base three years later I myself can barely figure out how it works or how to make changes. Could well apply to me and some of my old C++ projects. Well, except for the super-productive part. PG would say that it's not that the abstractions afforded by metaprogramming are incomprehensible. It's just that, if you are a habitual Blub programmer, everything up the power curve from you is always going to appear weird and hairy. This sort of argument is a terrible one to have with generalizations, though. You should look at specific cases where metaprogramming has been used and analyze them on a case-by-case basis. Sure, a bad, leaky, poorly-documented abstraction is going to lead to grief. On the other hand, umpteen thousands of programmers use the metamagic of Rails on a daily basis and barely bat an eyelash. (Or so it seems to me.)
  - Ididntdothis 7y ago
    Rails is a great example. For me it’s one level further up from simple meta programming. Writing stuff for yourself is one thing but writing stuff that’s universally usable for a lot of people is much harder.
jlarocco 7y ago
After years of using Common Lisp, my opinion on "meta-programming" is that it's wonderful when used sparingly and in just the right way, but a bad idea 99% of the time. If it's not completely obvious what (not how) a macro does just by looking at the code that calls it, then it's almost certainly a bad macro. For example, there's a CL library, iterate, that introduces several new iteration constructs. There are a bunch of complicated macros under the hood, but as an end user I never have to think about them, and the provided interface is obvious, even without knowing anything about the iterate library. For example: (iterate (for (key . item) in alist) (collect key into keys) (collect item into items) (finally (return (values keys items)))) I would definitely consider, "Just because you can doesn't mean you should" to be the number one rule of metaprogramming.
ralphc 7y ago
I think of Lisp vs. Python similarly to PostScript vs PDF. In PostScript you have a full language, can do anything but you often have to do it yourself. PDF's aren't extensible but have _just about_ all the functionality you need to produce a document. An example of macros you often see is surrounding a block of code with open/close pairs to make sure you don't exit the code leaving dangling resources. File open/close, database connections, etc. Python has the __enter__ and __exit__ protocols to handle that. You can't do every bit of metaprogramming in Python, but over the years they've covered close to every contingency a developer comes across.
- lispm 7y ago
  Lisp uses advising for that. In the modern form these are :before, :after and :around methods in CLOS. The difference between some Lisps and CPython is this: Lisp is its own implementation language and provides lower level control flow constructs, which can be compiled to efficient code.
pdonis 7y ago
> I know PG claims meta programming was the secret to ViaWeb's success, but I'm not sure how exactly that was the case IIRC, PG said the advantage of Lisp was being able to add features much more quickly than the competition, so that (a) they could be adding features almost daily that the competition would take weeks or months to match, and (b) whenever the competition added a new feature that looked like a good idea, they could implement it within a day or two, so by the time the press came around to ask them if they had it too, they could say yes. I also remember him saying that Lisp macros were about 25% of Viaweb's code, so a significant part of the advantage of Lisp was metaprogamming. > even if true whether that is still relevant today. It's probably not that much of an advantage any more if your competition is other startups, since they probably have access to basically the same metaprogramming tools that you do. It might still be a significant advantage if your competition is large, mature businesses whose business model you are trying to disrupt. > I have found Python to hit the sweet spot much better in terms of rapid prototyping, and providing a runway to making the prototype more rigorous and maintainable. I have found this to be true as well. But I also find that I do a significant amount of metaprogramming in Python, and one of the key things to transitioning to more rigorous and maintainable code is to make sure that whatever metaprogramming needs to stay there for the long term is packaged into a library or a tool whose internals won't need to change.
- alwalo 7y ago
  Could you write something more on metaprogramming in Python ? And how does this compare with Lisp, what do you use it for ? Any good resources to learn this ?
  - pdonis 7y ago
    Metaprogramming is a fairly broad term, but some of the kinds of things I often find myself doing in Python are: - Writing functions that make functions: often I'll find that I want a family of functions that all follow a the same general pattern. In Lisp this would generally be done with a macro; in Python it can usually be done by writing a function that returns a function. The "maker" function will generally have some parameters that fill in specific parts of the general pattern; the function that gets returned is therefore a closure over those parameters (i.e., it has specific values of them filled in). Doing this with a "maker" function is less error prone (particularly if there are a large number of variations on the general pattern) and follows the DRY (don't repeat yourself) principle, which I think is a very good one. - Writing decorators that add a particular common behavior (such as logging or debug output) to a variety of different functions or classes or methods. This is a common case. (In fact one could argue that decorators in general are examples of metaprogramming.) - Using data to construct code. Database programming is a common application of this: for example, I have a database library I commonly use (it's on GitLab here: https://gitlab.com/pdonis/plib3-dbtools https://gitlab.com/pdonis/plib3-dbtools) which constructs typed namedtuples specific to a database table's fields, and wraps rows returned from queries in them for easier field access. More complicated database ORMs use similar techniques. Unfortunately I don't know of any good resources on metaprogramming in general.

agalunar 7y ago

One of my best friends did his Masters work designing a programming language that can be modified on the fly; I'd highly recommend the paper: https://dspace.mit.edu/bitstream/handle/1721.1/113144/1017990524-MIT.pdf https://dspace.mit.edu/bitstream/handle/1721.1/113144/101799...

nielsbot 7y ago
The API for building grammars seems straightforward to use.

jimbob45 7y ago

I feel like there's something so comforting about having a concrete language you can rest your back up against though. Once you start throwing in #DEFINEs and typedefs, I start to question whether I can trust anything I'm seeing on the screen to not be hiding crucial secrets from me. Maybe it's just me.

kd0amg 7y ago
You don't know what all those library calls do behind the scenes either.
bob1029 7y ago
Having constraints imposed upon me by my tools is the only way anything ever gets done. If you put me in front of a blank canvas, and then tell me I can alter the nature of the canvas itself to whatever extent I desire (I.e. in hopes of making using it easier), I'll never get to the point where I'm actually painting on it. Or, perhaps I do manage to produce an end result, but no one else can understand what it means. Even with a mountain of constraints, there is still usually far too much subjectivity in software development. This language seems to me to be moving the needle in a very wrong direction. Having a stable basis that is standardized and everyone can (mostly) agree upon is the foundation for growing a technical team or community. Being able to type "LINQ GroupBy Select examples" into your preferred search engine of choice and receive actionable advice is clearly an advantage for productivity. Sure, in a purely academic sense there is almost certainly a more ideal solution. But, if you are seeking to actually get things done, constraints are the name of the game.
- discreteevent 7y ago
  I remember some interview with Ulrich Schnauss (electronic artist) where he went back to plain synthesisers for the same reason. Too many things to tweak meant he wasn't able to focus on the piece.
zomglings 7y ago
What you said really resonates with me - especially re: resting your back up against it. Just built my first single-page website with React and I ran up against the new Javascript ES6 import system. It makes no sense that people keep introducing this kind of unnecessary complexity into their tools. What was wrong with good old "require"? This is just one instance among many many many. It feels like I'm aging a year every day.

Koshkin 7y ago

The usefulness of this depends on how you want to solve the problem at hand. One approach to programming is to create a DSL for the problem domain and then solve the problem using that DSL. Languages (such as Common Lisp) that include advanced means of metaprogramming are perfect from this perspective. Otherwise, one has to create an API to approximate the DSL - which could be rather hairy, so one may try to manage the complexity of this particular representation by either using a preprocessor or even, as an extreme measure, by writing a separate language processor (compiler or interpreter) for the DSL.

jmull 7y ago

I feel like general purpose programming languages very widely already let you express DSLs, with semantics, names, structure, etc. appropriate to the domain. What they do restrict, to various degrees, is the syntax. So what we're talking about here is custom/customizable syntax to go with your DSL. That can be nice, but it comes at a high cost -- e.g., developers will have to understand the syntax, it will need to be maintained and extended as requirements grow or change. Of course it's worth it in some cases, and tools to help do it are great when it is. But I just don't see that there's a general problem that we don't have enough custom language syntax to deal with.

bitwize 7y ago

Meh, in practice most people will use JavaScript and Babel.

iLemming 7y ago
Yeah, and that is a problem with the majority of programmers. They readily choose a seemingly easy path instead of something that leads to simplicity. Lisp is a fantastic idea that (even after over six decades) is still very much relevant today. Once you get used to program with a true REPL, every other (non-lispy) language starts feeling like frustratingly over-engineered, bloated, and unnecessarily complicated. Waiting for compilation cycles and annoying syntax quirks would slowly turn what once was a joy into a "just a job." It feels incredibly liberating and joyous to be able to grab any part of your program and without any preceding ceremony just to "eval" it and see what happens. And let me prematurely answer the comments that probably will be thrown here. No, other languages (most of them) don't have real REPLs. What they have, at best can be categorized as "interactive shells". Younger folks who are learning, juniors, seasoned developers who for one or another reason, never had any exposure to a Lisp, please do yourselves a favor. Try Racket, Clojure, Clojurescript, Fennel, Chez. Give Lisp an honest, heartfelt attempt to learn it. Follow the work of people like Harold Abelson and Gerald Sussman, Matthias Felleisen, Dan Friedman, Rich Hickey, Guy Steele, Richard Gabriel, Alan Kay, and many others. Once you attain familiarity with a Lisp to the sufficient level, perhaps then you will get it. You will see that it is indeed a lifetime investment that would allow you to reap the rewards for the entirety of your career. I am telling you all this, not from the top of some ivory tower. I never went to CS academia. Like most of you, I am altogether a self-taught programmer. I am honestly begging you not to make the same mistake that I did. For many years I have ignored Lisp due to my self-righteous ignorance. Unknowingly, I delayed my true growth, and I sincerely regret it.
- lispm 7y ago
  > No, other languages (most of them) don't have real REPLs. What they have, at best can be categorized as "interactive shells". > Try Racket, Clojure, Clojurescript, Fennel, Chez. Give Lisp an honest, heartfelt attempt to learn it. Compared to actual Lisps, some of these have mostly interactive shells. https://www.youtube.com/watch?v=o4-YnLpLgtk https://www.youtube.com/watch?v=o4-YnLpLgtk