4 ms·

Ask HN: Writing DSL in Python

26 points by oltmans 18y ago

Hey Hackers, I was reading this blog article http://weblog.jamisbuck.org/2006/4/20/writing-domain-specific-languages and I've no idea how can one go about writing a DSL in Python. I've rudimentary knowledge of Python and I'm interesting in writing a DSL just for fun and knowledge. Can anyone please point out what do I need to learn to be able to write a DSL in Python? Any help is appreciated.

apgwoz 18y ago
Study up on context managers, decorators, and meta-classes, which will assist in creating DSLs. However, you don't really need to use these features to build DSLs. The use of method chaining (think jQuery) can lead to interesting languages. Or, take Django's chaining for making queries. User.objects.filter(first_name='Andrew').order_by('last_name') Or, you can get more complicated and do something like described at: http://code.activestate.com/recipes/534150/ http://code.activestate.com/recipes/534150/ EDIT: s/be vital/assist/ on line 1.
- cabalamat 18y ago
  > Study up on context managers, decorators, and meta-classes, which will be vital to creating DSLs. However, you don't really need to use these features to build DSLs. Make your mind up. Either they're vital or they're not. You can't have both.
  - apgwoz 18y ago
    No. I won't make my mind up. I'm not providing a recipe for DSLs in Python. I'm simply stating ingredients and saying "now go create."
    - petercooper 18y ago
      I think he/she was more criticizing that you effectively said "X, Y and Z are vital" and then "But X, Y and Z are not vital." If you'd said "useful" instead of "vital" it'd have been fine logically, but "vital" implies necessity.
      - apgwoz 18y ago
        Yeah. I ended up editing the original post when I realized it later.
- russell 18y ago
  apgwoz has a excellent suggestion. Python has a clean syntax, so you may not need to invent your own. Meta-classes allow you to do the DSL magic behind the scenes. It's less fun than inventing a whole new language, but you can get something working in a couple of hours. Even more useful are the mind stretching exercises inherent in understanding meta-classes and their friends. You will see programming in a whole new light.
jlc 18y ago
I ran into this a while back: http://blog.brianbeck.com/post/53538107/python-dsl-i http://blog.brianbeck.com/post/53538107/python-dsl-i
ivank 18y ago
You could look at SQLAlchemy, Storm, Schevo, or Nevow Stan. Or even BeautifulSoup's element-finding.
nuclear_eclipse 18y ago
I still haven't quite grasped the concept behind what a DSL really is, and what makes your code a DSL versus any other program or API. Any insight that can be offered, especially in the context of something other than Ruby and Lisp? I just want to know what all the hubbub is about, and why it matters to me?
- cabalamat 18y ago
  A DSL is a language that is specific to a domain. DSLs may or may not be Turing-complete, but typically you wouldn't write the whole of a big project in one. The difference between a DSL and an API? An API uses the host language's syntax, but a DSL introduces its own syntax, which must then be parsed.
  - apgwoz 18y ago
    > The difference between a DSL and an API? An API uses the host language's syntax, but a DSL introduces its own syntax, which must then be parsed. Well, this is a conundrum then. You can't just create new syntax in languages that don't support a way to extend the syntax (see Lisp macros, and I even get to mention TCL here). So, most of what we normally call "DSLs," (i.e. method chaining languages, overloading operators) aren't. So, I guess we should start calling them Domain Specific APIs--DSAPIs, if you will.
    - cabalamat 18y ago
      > So, most of what we normally call "DSLs," (i.e. method chaining languages, overloading operators) aren't. I don't call these things DSLs. Other people might.
      - wilkes 18y ago
        http://www.martinfowler.com/bliki/DomainSpecificLanguage.htmlFowler http://www.martinfowler.com/bliki/DomainSpecificLanguage.htm... distinguishes between Internal and External DSLs. Internal DSLs are a style of programming, usually non-idiomatic. External DSls require parsing.
tocomment 18y ago
I think it just means writing a programming language but a little one. So use pyparsing as your parser. The May 2008 Python magazine has a good article on making an interpreter/compiler for brainf*ck using PyParsing.
- wilkes 18y ago
  I've really enjoyed using PyParsing. O'Reilly has a ShortCut book on it that is worthwhile. http://oreilly.com/catalog/9780596514235/ http://oreilly.com/catalog/9780596514235/
cabalamat 18y ago
I have implemented two languages in Python, one a DSL, the other a general-purpose language. Both times I used the SPARK parsing framework http://pages.cpsc.ucalgary.ca/~aycock/spark/ http://pages.cpsc.ucalgary.ca/~aycock/spark/ which I recommend.
bayareaguy 18y ago
I would recommend taking a quick look at how SCons and Conary use python. You may also want to consider implementing your DSL using a python parser generator such as Ply. SCons - http://www.scons.org/doc/HTML/scons-man.html#lbBB http://www.scons.org/doc/HTML/scons-man.html#lbBB Conary - http://wiki.rpath.com/wiki/Conary:Recipe_Structure http://wiki.rpath.com/wiki/Conary:Recipe_Structure Ply - http://www.dalkescientific.com/writings/NBN/parsing_with_ply.html http://www.dalkescientific.com/writings/NBN/parsing_with_ply...
nostrademons 18y ago
I used yapps2 for the DSL parts of GameClay. It's probably not the best Python parser generator out there, but it was installable via apt, simple to use, and familiar to people who have used Antlr. Had a simple DSL written in 2 days. Also, if it's just a quick-n-dirty hack, don't underestimate the power of regexp/split() parsing. Yeah, everyone says you shouldn't do it, and you probably shouldn't if you're planning on maintaining things for a while, but you can get going really fast without learning any additional tools. And for maybe 80-90% of the things you'd use a DSL for, it's perfectly adequate.
intellectronica 18y ago
You can customize the behaviour of Python objects in ways which may allow you to implement DSLs (for example, by responding to arbitrary messages, or by constructing callables). A good example to look at might be https://storm.canonical.com/ https://storm.canonical.com/ which utilizes every trick in the book to allows you to write SQL-like queries in Python.
gruseom 18y ago
Although I got excited about it at first, I've come to the conclusion that this DSL stuff is seriously overrated. I've seen few successful examples and many silly ones. There are certainly good little specialized languages -- Common Lisp's LOOP is an example, as are regular expressions -- but these are for programmers, not domain experts.
thorax 18y ago
No joke: take a look at the LOLpython source for an example of creating a variant language that can compile to Python bytecode. It's one of many methods to create a DSL, though you often don't need access to all of Python in the DSL itself.
jgalvez 18y ago
http://github.com/galvez/gae-rest/tree/master/xmlbuilder.py http://github.com/galvez/gae-rest/tree/master/xmlbuilder.py

26 points by oltmans 18y ago

apgwoz 18y ago

Study up on context managers, decorators, and meta-classes, which will assist in creating DSLs. However, you don't really need to use these features to build DSLs. The use of method chaining (think jQuery) can lead to interesting languages. Or, take Django's chaining for making queries. User.objects.filter(first_name='Andrew').order_by('last_name') Or, you can get more complicated and do something like described at: http://code.activestate.com/recipes/534150/ http://code.activestate.com/recipes/534150/ EDIT: s/be vital/assist/ on line 1.

cabalamat 18y ago
> Study up on context managers, decorators, and meta-classes, which will be vital to creating DSLs. However, you don't really need to use these features to build DSLs. Make your mind up. Either they're vital or they're not. You can't have both.
- apgwoz 18y ago
  No. I won't make my mind up. I'm not providing a recipe for DSLs in Python. I'm simply stating ingredients and saying "now go create."
  - petercooper 18y ago
    I think he/she was more criticizing that you effectively said "X, Y and Z are vital" and then "But X, Y and Z are not vital." If you'd said "useful" instead of "vital" it'd have been fine logically, but "vital" implies necessity.
    - apgwoz 18y ago
      Yeah. I ended up editing the original post when I realized it later.
russell 18y ago
apgwoz has a excellent suggestion. Python has a clean syntax, so you may not need to invent your own. Meta-classes allow you to do the DSL magic behind the scenes. It's less fun than inventing a whole new language, but you can get something working in a couple of hours. Even more useful are the mind stretching exercises inherent in understanding meta-classes and their friends. You will see programming in a whole new light.

jlc 18y ago

I ran into this a while back: http://blog.brianbeck.com/post/53538107/python-dsl-i http://blog.brianbeck.com/post/53538107/python-dsl-i

ivank 18y ago

You could look at SQLAlchemy, Storm, Schevo, or Nevow Stan. Or even BeautifulSoup's element-finding.

nuclear_eclipse 18y ago

I still haven't quite grasped the concept behind what a DSL really is, and what makes your code a DSL versus any other program or API. Any insight that can be offered, especially in the context of something other than Ruby and Lisp? I just want to know what all the hubbub is about, and why it matters to me?

cabalamat 18y ago
A DSL is a language that is specific to a domain. DSLs may or may not be Turing-complete, but typically you wouldn't write the whole of a big project in one. The difference between a DSL and an API? An API uses the host language's syntax, but a DSL introduces its own syntax, which must then be parsed.
- apgwoz 18y ago
  > The difference between a DSL and an API? An API uses the host language's syntax, but a DSL introduces its own syntax, which must then be parsed. Well, this is a conundrum then. You can't just create new syntax in languages that don't support a way to extend the syntax (see Lisp macros, and I even get to mention TCL here). So, most of what we normally call "DSLs," (i.e. method chaining languages, overloading operators) aren't. So, I guess we should start calling them Domain Specific APIs--DSAPIs, if you will.
  - cabalamat 18y ago
    > So, most of what we normally call "DSLs," (i.e. method chaining languages, overloading operators) aren't. I don't call these things DSLs. Other people might.
    - wilkes 18y ago
      http://www.martinfowler.com/bliki/DomainSpecificLanguage.htmlFowler http://www.martinfowler.com/bliki/DomainSpecificLanguage.htm... distinguishes between Internal and External DSLs. Internal DSLs are a style of programming, usually non-idiomatic. External DSls require parsing.

tocomment 18y ago

I think it just means writing a programming language but a little one. So use pyparsing as your parser. The May 2008 Python magazine has a good article on making an interpreter/compiler for brainf*ck using PyParsing.

wilkes 18y ago
I've really enjoyed using PyParsing. O'Reilly has a ShortCut book on it that is worthwhile. http://oreilly.com/catalog/9780596514235/ http://oreilly.com/catalog/9780596514235/

cabalamat 18y ago

I have implemented two languages in Python, one a DSL, the other a general-purpose language. Both times I used the SPARK parsing framework http://pages.cpsc.ucalgary.ca/~aycock/spark/ http://pages.cpsc.ucalgary.ca/~aycock/spark/ which I recommend.

bayareaguy 18y ago

I would recommend taking a quick look at how SCons and Conary use python. You may also want to consider implementing your DSL using a python parser generator such as Ply. SCons - http://www.scons.org/doc/HTML/scons-man.html#lbBB http://www.scons.org/doc/HTML/scons-man.html#lbBB Conary - http://wiki.rpath.com/wiki/Conary:Recipe_Structure http://wiki.rpath.com/wiki/Conary:Recipe_Structure Ply - http://www.dalkescientific.com/writings/NBN/parsing_with_ply.html http://www.dalkescientific.com/writings/NBN/parsing_with_ply...

nostrademons 18y ago

I used yapps2 for the DSL parts of GameClay. It's probably not the best Python parser generator out there, but it was installable via apt, simple to use, and familiar to people who have used Antlr. Had a simple DSL written in 2 days. Also, if it's just a quick-n-dirty hack, don't underestimate the power of regexp/split() parsing. Yeah, everyone says you shouldn't do it, and you probably shouldn't if you're planning on maintaining things for a while, but you can get going really fast without learning any additional tools. And for maybe 80-90% of the things you'd use a DSL for, it's perfectly adequate.

intellectronica 18y ago

You can customize the behaviour of Python objects in ways which may allow you to implement DSLs (for example, by responding to arbitrary messages, or by constructing callables). A good example to look at might be https://storm.canonical.com/ https://storm.canonical.com/ which utilizes every trick in the book to allows you to write SQL-like queries in Python.

gruseom 18y ago

Although I got excited about it at first, I've come to the conclusion that this DSL stuff is seriously overrated. I've seen few successful examples and many silly ones. There are certainly good little specialized languages -- Common Lisp's LOOP is an example, as are regular expressions -- but these are for programmers, not domain experts.

thorax 18y ago

No joke: take a look at the LOLpython source for an example of creating a variant language that can compile to Python bytecode. It's one of many methods to create a DSL, though you often don't need access to all of Python in the DSL itself.

jgalvez 18y ago

http://github.com/galvez/gae-rest/tree/master/xmlbuilder.py http://github.com/galvez/gae-rest/tree/master/xmlbuilder.py