3 ms·

Python as a language is so good that it becomes difficult to concisely describe why one wants metaprogramming. Generally speaking the reason one wants metaprog

by gchpaco 17y ago

Python as a language is so good that it becomes difficult to concisely describe why one wants metaprogramming. Generally speaking the reason one wants metaprogramming is to make the language of the domain you are thinking in and the language of the program as close as possible. So in banking we talk of accounts and transactions, not rows in a database. The need for this is so acute in Java it makes my teeth ache. Python is much more flexible; so much so that this becomes a difficult to explain urge, but it is still there. An example may be worthwhile. Consider SCons. Know anybody who uses it? There were a number of issues going on there (one of which was the general lameness of the Cons infrastructure--I'm sorry, I use the fact that make is a dependency computation engine, all the damned time, and ant and Cons and every other lame-ass "make replacement" that doesn't do this drives me up the wall) but IMHO the decision to make the syntax raw, executable Python code did not help. Now, granted, I consider myself a Lisp hacker, so maybe you'll just write this off entirely. But the key thing I have with Python, the key complaint I have, is that there are fundamental domain assumptions baked into the language that are sometimes appropriate (numpy is an enormous success, for example) and sometimes aren't (SCons, HTML template system du jour, configuration files, etc.), and I can't change them. Edit: an example came to me as I thought about this. So how's this as an explanation: in Java and Python, you can have as many nouns as you want. You can have as many verbs as you like. Adverbs are hard. Pronouns are almost impossible. And syntax? Forget it. It makes for an interesting contrast with mathematical notation, which takes time to learn and is inconsistent and only governed by social constraints but, at its best, is so good at communicating information that it fades into the background, just as = or 0 do.

jnoller 17y ago
"Now, granted, I consider myself a Lisp hacker, so maybe you'll just write this off entirely." Now why would I write you off? :) I would like to see examples of what you're talking about however.
- gchpaco 17y ago
  It's hard to quantify the benefits of system-qua-language without pointing you at Maxima or something (a computer algebra system well integrated with the underlying programming language). But here's a small example of the sort of thing I want to do with metaprogramming in Python. In C, C++, Java, and almost every commonly used language, parsing is usually done by writing a parser in a different language (yacc, ANTLR, whatever) which is then compiled to source code which is then compiled again. In a language with good metaprogramming facilities, the parser is written in the language; macros in Lisp-descendants, parser combinators in Haskell (itself probably about as flexible syntactically as you can get without actually having macros), whatever the hell Forth people call it, etc. This has, at least in theory (a lot of parser combinators in Haskell are impossibly naive, but that's an implementation detail or so I claim), no demerits over the C approach, but it integrates better with the language. This is not just a difference in taste, either. As a different example, Python had the problem wherein regexp strings were getting impossibly difficult to write. They added raw strings to the language to get around this. Schemers had the same problem, but one of them (Shivers) wrote a regexp language with escapes to the underlying Scheme implementation that compiled to regexps but made it possible to read them like they were any other kind of code, instead of weird terse utterances. You can still do the weird terse utterances if you like, but you can also write regexps like they were any other kind of Scheme code. When I was doing web programming in 2000, I hacked an SQL generation library in Python. It felt awkward to use and was verbose but I deemed it better than being paranoid about SQL injection attacks. If I had good metaprogramming facilities, I might have been able to implement the database comprehensions they have in C# now (the LINQ facilities). I probably wouldn't have decided to do so, because even in 2000 I had some taste, but I could have and experienced all the benefits the C# people claim to have. Another thing that came up during that same project was that I was trying to do what we now call continuation-based web frameworks in a language with no explicit continuation apparatus. This is similar to what PG described doing in Viaweb, but I didn't have a metaprogramming setup to help me out with it. I did the CPS conversion manually, and lemme tell you it was a royal pain in the arse and contributed significantly to the difficulty in implementing that setup. If I had good metaprogramming frameworks I could CPS convert the code automatically, as PG did. Would you like more? :)
  - rw 17y ago
    A Lisper myself, what I really want from Python are its libraries. I don't care about its syntactical "elegance" at all.
  - arohner 17y ago
    Excellent points. > In a language with good metaprogramming facilities, the parser is written in the language ... it integrates better with the language. Your point was about how a good language makes it easy to parse another language into your desired langauge. I recently noticed the converse is true too. A good language makes it easy to output code in another language In Clojure (and I suspect other lisps), it is quite common to see clojure literals used to generate code in other languages. i.e. in compojure, a webapp framework, the html generation code is entirely clojure literals: (html [:body [:div [:a {:href "www.google.com"} "click me"]]]) generates: <body><div><a href="www.google.com">click me</a></div></body> There are plenty of examples of this for html generation, SQL generation, and in one CL library, even C source code generation.
    - catch404 17y ago
      Is it possible to do a similar construct in python? This is what I'm really after, a Lisp like ability to generate html, sql combined with the rest of the Python language. Perhaps I should be looking into ruby more? or a python parser? I'm working on a similar technique for python (using lists and function lookups but it's still more verbose than the lisp style ones) ('p','this is a paragraph') vs (p, "this is a paragraph") as a small example
  - j_baker 17y ago
    "In a language with good metaprogramming facilities, the parser is written in the language; macros in Lisp-descendants, parser combinators in Haskell (itself probably about as flexible syntactically as you can get without actually having macros), whatever the hell Forth people call it, etc." You mean like PyPy? http://codespeak.net/pypy/dist/pypy/doc/ http://codespeak.net/pypy/dist/pypy/doc/
    - 17y ago
daeken 17y ago
I agree entirely -- I love Python and it's incredibly flexible, I just frequently find myself wanting more. There's rarely something I /can't/ do, but I often find myself wanting to add, say, pattern matching in an intuitive way. I guess the best way to put it is that Python has a lot of flexibility in what you can do, but not so much in how you do it.
aristus 17y ago
Metaprogramming is like freedom of speech: everyone wants it, but wish the idiots who abuse it would shut the hell up. :) When your language is programmable, you have to accept that static analysis, even a simple grep for function calls, is essentially impossible if you don't absolutely trust your fellow programmers not to do something too clever. I think that's why Lisps are considered lone wolf languages.

by gchpaco 17y ago

jnoller 17y ago

"Now, granted, I consider myself a Lisp hacker, so maybe you'll just write this off entirely." Now why would I write you off? :) I would like to see examples of what you're talking about however.

gchpaco 17y ago
It's hard to quantify the benefits of system-qua-language without pointing you at Maxima or something (a computer algebra system well integrated with the underlying programming language). But here's a small example of the sort of thing I want to do with metaprogramming in Python. In C, C++, Java, and almost every commonly used language, parsing is usually done by writing a parser in a different language (yacc, ANTLR, whatever) which is then compiled to source code which is then compiled again. In a language with good metaprogramming facilities, the parser is written in the language; macros in Lisp-descendants, parser combinators in Haskell (itself probably about as flexible syntactically as you can get without actually having macros), whatever the hell Forth people call it, etc. This has, at least in theory (a lot of parser combinators in Haskell are impossibly naive, but that's an implementation detail or so I claim), no demerits over the C approach, but it integrates better with the language. This is not just a difference in taste, either. As a different example, Python had the problem wherein regexp strings were getting impossibly difficult to write. They added raw strings to the language to get around this. Schemers had the same problem, but one of them (Shivers) wrote a regexp language with escapes to the underlying Scheme implementation that compiled to regexps but made it possible to read them like they were any other kind of code, instead of weird terse utterances. You can still do the weird terse utterances if you like, but you can also write regexps like they were any other kind of Scheme code. When I was doing web programming in 2000, I hacked an SQL generation library in Python. It felt awkward to use and was verbose but I deemed it better than being paranoid about SQL injection attacks. If I had good metaprogramming facilities, I might have been able to implement the database comprehensions they have in C# now (the LINQ facilities). I probably wouldn't have decided to do so, because even in 2000 I had some taste, but I could have and experienced all the benefits the C# people claim to have. Another thing that came up during that same project was that I was trying to do what we now call continuation-based web frameworks in a language with no explicit continuation apparatus. This is similar to what PG described doing in Viaweb, but I didn't have a metaprogramming setup to help me out with it. I did the CPS conversion manually, and lemme tell you it was a royal pain in the arse and contributed significantly to the difficulty in implementing that setup. If I had good metaprogramming frameworks I could CPS convert the code automatically, as PG did. Would you like more? :)
- rw 17y ago
  A Lisper myself, what I really want from Python are its libraries. I don't care about its syntactical "elegance" at all.
- arohner 17y ago
  Excellent points. > In a language with good metaprogramming facilities, the parser is written in the language ... it integrates better with the language. Your point was about how a good language makes it easy to parse another language into your desired langauge. I recently noticed the converse is true too. A good language makes it easy to output code in another language In Clojure (and I suspect other lisps), it is quite common to see clojure literals used to generate code in other languages. i.e. in compojure, a webapp framework, the html generation code is entirely clojure literals: (html [:body [:div [:a {:href "www.google.com"} "click me"]]]) generates: <body><div><a href="www.google.com">click me</a></div></body> There are plenty of examples of this for html generation, SQL generation, and in one CL library, even C source code generation.
  - catch404 17y ago
    Is it possible to do a similar construct in python? This is what I'm really after, a Lisp like ability to generate html, sql combined with the rest of the Python language. Perhaps I should be looking into ruby more? or a python parser? I'm working on a similar technique for python (using lists and function lookups but it's still more verbose than the lisp style ones) ('p','this is a paragraph') vs (p, "this is a paragraph") as a small example
- j_baker 17y ago
  "In a language with good metaprogramming facilities, the parser is written in the language; macros in Lisp-descendants, parser combinators in Haskell (itself probably about as flexible syntactically as you can get without actually having macros), whatever the hell Forth people call it, etc." You mean like PyPy? http://codespeak.net/pypy/dist/pypy/doc/ http://codespeak.net/pypy/dist/pypy/doc/
  - 17y ago

daeken 17y ago

I agree entirely -- I love Python and it's incredibly flexible, I just frequently find myself wanting more. There's rarely something I /can't/ do, but I often find myself wanting to add, say, pattern matching in an intuitive way. I guess the best way to put it is that Python has a lot of flexibility in what you can do, but not so much in how you do it.

aristus 17y ago

Metaprogramming is like freedom of speech: everyone wants it, but wish the idiots who abuse it would shut the hell up. :) When your language is programmable, you have to accept that static analysis, even a simple grep for function calls, is essentially impossible if you don't absolutely trust your fellow programmers not to do something too clever. I think that's why Lisps are considered lone wolf languages.