6 ms·

Utterly sensible HTML templating in Common Lisp

71 points by sgrove 16y ago

rorrr 16y ago
Sensible html templating exists already. It's called HAML. http://haml-lang.com/ http://haml-lang.com/
- sgrove 16y ago
  Actually, while developing this I was really struck by the similarities. With the automatic indenting in emacs for lisp (really quite good), it looks very similar, and it's impossible to leave a tag unclosed, but there's a further gain for using parentheses in that with paren-highlighting, you can tell exactly which tag your in. While I converted several dozen pages to this system from straight html, I was saved again and again by this. HAML's great, and I disagree with the other "I hate haml" post on the frontpage today, but this approach actually alleviates the issues pointed out with haml's whitespace-is-meaningful approach. Anyway, it's all in good fun.
  - rapind 16y ago
    Yup I'm a huge haml fan, and saw the similarities. I thought your post was great. Very clean, and you can keep optimizing it. I'm learning CL right now and this added some fuel to the motivation fire ;)
Vekz 16y ago
I am spoiled by nodejs and its live server side DOM rendering. Having one layer of template code reusable on the client and server is too powerful. You would think people would be sick of writing duplicate server templates and ajax templates. To me anything else is a step in the wrong direction.
- sgrove 16y ago
  I think you're pointing out Dave Glass' technique over at yahoo, right? This one http://www.yuiblog.com/blog/2010/04/09/node-js-yui-3-dom-manipulation-oh-my/ http://www.yuiblog.com/blog/2010/04/09/node-js-yui-3-dom-man... ? The ideas there are definitely very powerful... and just plain sensible. I'd like to find a way to work this in, but it will take time. Do you see any fundamental reason a lisp equivalent of this technique couldn't be reproduced, using something like parenscript?
  - sedachv 16y ago
    It's been done already: http://code.google.com/p/cl-closure-template/ http://code.google.com/p/cl-closure-template/
    - sgrove 16y ago
      That's extremely different from the technique that Glass is demoing - in fact, it's the antithesis. It takes care of unifying the server/client templates (by introducing an unnecessary layer, imo), but it's very obviously string-based output rather than live DOM manipulation like Enlive or YUI on node.js
swannodette 16y ago
I still think the new gold standard for functional and extensible HTML templating is Enlive - people should look into this before coming up with variations of the same old wheel - https://github.com/cgrand/enlive https://github.com/cgrand/enlive
- sgrove 16y ago
  Enlive is awesome, I used it while working in clojure quite a bit. I definitely prefer it, personally, to something like Clojure's hiccup (which vana-templating is very similar to). The tradeoff is that new-comers who are already familiar with front-end design are going to find it tough to transition - it's a fundamental problem: do you optimize for the newcomer, or for the experienced developer? Enlive has a lot in common with Vekz' suggestion elsewhere in the comments, and I feel like there's some unifying theme underlying all of this, that should make it easy for newcomers, but allow experienced developers to transform the DOM programmatically. I'm searching for it now, and I'm sure there'll be a lot of false-starts and dead-ends, but the only sure way not to get there is to not try.
- nomurrcy 16y ago
  I agree completely. Enlive has really blown me away.
- xtacy 16y ago
  I think there's a difference: Enlive is meant to manipulate the DOM using familiar CSS/jQuery like selectors. The article however, is a way to _write_ the HTML in the first place. I've myself been using a declarative style HTML output engine that I wrote in Python. It's simple, easier, outputs correct HTML and it merges well with my workflow. I don't have to write HTML, nor JavaScript to programmatically generate what I need. It may sound naive, but I find myself comfortable indenting code in Python than write matching HTML tags. If there are errors, the Python code wouldn't run, but HTML code would still render, though it wouldn't be what I want. Relevant: http://code.google.com/p/zen-coding/ http://code.google.com/p/zen-coding/
jasonlotito 16y ago
Anything that requires you to write HTML in something that isn't HTML isn't sensible. Sure, go ahead, add tokens to the HTML that can be parsed out and rendered differently, stored in a cache or something. But as soon as you're forced to write in another language to get HTML, you've lost. Edit: To elaborate, as some have asked: I'm speaking in terms of HTML templating. Having the software render your HTML form for you, or rending out a table full of data is fine. But by and large, the HTML should remain HTML. Granted, in this case, your not writing HTML, your writing code to generate a form. Not sitting there adding HTML attributes, which is handled for you. As to why I disagree with this approach? At it's core, portability. You can now no longer take an HTML file and use it. You have to rewrite everything into your strange, personal, customized, non-standard language and hope everything comes out alright. Their is also the learning curve in bringing anyone else on board. You also suddenly lack the ability to use any of the tools associated with HTML. Abstraction for the sack of abstraction is silly. If you have multiple functions for every single tag, your essentially taking HTML and just making the syntax your own. This "(ul '(id "navbar" class "horizontal_list")" is no better than HTML, but it isn't special or unique. It's cumbersome. Your also limited by the framework (which you hope will add new tags as they appear, and support the customization HTML offers). I'm not against HTML generating code. I'm against rewriting HTML as code. Maybe someone can explain how (ul '(id "navbar" class "horizontal_list")... is better than <ul id="navbar" class="horizontal_list">... Edit 2: Also, it should be fairly obviously, but all of this is of course just my opinion. =)
- gnaritas 16y ago
  Not true, the are advantages to programatically generated HTML using a mini language that simply can't be ignored. See http://seaside.st http://seaside.st
- thecombjelly 16y ago
  How have you lost? I write my HTML in a very similar way to the linked article, and I don't think I lose at all. In fact, I much prefer it. It is more consistent. All of my code is now (in my case) Scheme. This makes it easier to change things as you don't have to go between two different languages. And it is simpler. My favorite reason is the fact that I can use S-Exps. I no longer have to type the ending tag. I remove the possibility for spelling it wrong, or forgetting the slash. And it takes less space and is easier to get the placement correct, as you just utilize paren matching.
marcusbooster 16y ago
Not trying to pick on this particular project because it seems cool enough, but, I don't get all this html templating. With ajax and websockets on the way, why are we still generating html in the application? This all seems like a solution to a problem we should be leaving behind.
- sgrove 16y ago
  You're definitely right, actually. I've been working a lot in clojure, node.js, and html5 <canvas> recently on various projects (more posts later), and there's a definitely feeling of reinventing the wheel, and doing it poorly. That said, there are a ton of different approaches - At the oak.js meetup, I saw a sammy.js app embedded into couchdb. Definitely not my style, but a cool idea anyway. I'm very eager to get this into several html5 apps that are currently node.js or rails backends and for particular reasons would be well suited for a lisp backend. The goal is to tie in very closely with the html5 landscape, and drop a lot of the legacy stuff that other frameworks care about.
- lemming 16y ago
  on the way You said it - websockets aren't here yet. Plus I think there are still plenty of good applications for static HTML. For starters, even full AJAX apps should work without JS enabled if you care about accessibility.
  - ekidd 16y ago
    For starters, even full AJAX apps should work without JS enabled if you care about accessibility. This should be just about possible using Node.js these days: You can run your entire client-side templating system—and even the <canvas>—on the server, and then server static HTML and PNGs to the browser. Unfortunately, quite a bit of elbow grease is still required.
    - lemming 16y ago
      But my point is - you're still templating. The only thing you're gaining is using the same language on the client and the server. I'm not saying that's a small thing, but the OP could have achieved the same using parenscript. The GP seemed to be implying that we should only generate our HTML programatically on the client. Thinking about it more, I don't agree with this. I don't write webapps these days, but in our platform we do a lot of code generation. We use a programmatic method to do this because we have to manipulate the intermediate representation a lot - add or remove fields added by earlier phases in the generation etc. This is much more flexible but it's much harder to maintain, and looking at the code generation code it's very hard to see what the end result will be. If we were simply generating code in a single pass I'd definitely use templating, it's just much simpler, easier to develop, and easier to see what the end result will be. Add in the need with HTML to work with designers and I think templating will be with us for some time to come.
zachbeane 16y ago
You should learn about WITH-OUTPUT-TO-STRING. (string-downcase (string thing)) should be written (string-downcase thing) The fdefinition trick is not a good one. Define the functions at compile-time, not at runtime.
- chaitanya 16y ago
  > (string-downcase (string thing)) should be written (string-downcase thing) I learnt this trick only after working on CL for a couple of years!
  - zachbeane 16y ago
    Maybe http://l1sp.org/search/designator http://l1sp.org/search/designator will teach you a few more tricks and save you some time...
    - chaitanya 16y ago
      Nice. Today I learnt about the file position designator :)
- sgrove 16y ago
  I was originally using macros, but I prefer to use functions instead. I'll give it some thought, you may be right in this case. And thanks for the (string-downcase thing) trick, I'll update that right now :)
  - zachbeane 16y ago
    Functions are good. Defining them at runtime with (setf (fdefinition ...) ...) is rarely good.
astine 16y ago
This looks to me like an HTML generation library, not an HTML templating tool. Common Lisp has a number of both, and I'm wondering what advantage this would have over cl-who, for example?
chaitanya 16y ago
For those who didn't know already, cl-who (http://weitz.de/cl-who/ http://weitz.de/cl-who/) is a well known sexp-to-html converter. This one works a little differently from cl-who (uses functions instead of macros), but the input is similar.
philjackson 16y ago
One a little like it for emacs: https://github.com/philjackson/xmlgen https://github.com/philjackson/xmlgen
DanielRibeiro 16y ago
Can also be done, in runtime including, more cleanly and without parenthesis with coffeescript: http://news.ycombinator.com/item?id=1786002 http://news.ycombinator.com/item?id=1786002
its2010already 16y ago
The problem I always see with practically all templating systems is the lack of proper escaping of input data. For example, the function 'tag-name' really needs to HTML-escape the input parameter content. The simple example (tag-name "div" "Hello world") breaks down when "Hello World" is replaced by a variable reference. Fortunately this is an relatively easy problem to solve. A better definition for tag-name is something like: (defun tag-name (name content) (html (format nil "<~A>~A</~A>" name (to-html content) name)) The function to-html will return the HTML-escaped version of content. The function html returns an object/string that marked as already being HTML-escaped. They work together so that (to-html (html x)) just returns x. A similar type system can be set up to handle url escaping rules. If the templating system doesn't keep track of what is escaped and what isn't then that burden is left to the programmer with all of the associated risks of producing malformed output and creating opportunities for injection attacks.
jhrobert 16y ago
Am I missing something or is this code defining "global scope" functions whose names include things like a, b, i, p, command, etc...? What are the chances that such global names will collide with something else?
- bodhi 16y ago
  You could easily put them into a separate package if you wish: http://gigamonkeys.com/book/programming-in-the-large-packages-and-symbols.html http://gigamonkeys.com/book/programming-in-the-large-package...
KirinDave 16y ago
Or don't template at all. Let your designers design documents, then you write code to modify the markup at key integration points: https://github.com/cgrand/enlive https://github.com/cgrand/enlive This has some really nice properties: 1. The only thing your devs and designers need to agree on is some basic semantic markup and tag IDs. Then they can make a full comp (complete with sample data) for their purposes. You simply rewrite that data. 2. The approach is very declarative, which makes it a lot easier to reason about what is actually being shown and what is actually changing. 3. It's amenable to all the same functional properties of the sexp->xml stuff that you see here. Enlive's approach is a nearly 100% win for doing site templating over these sorts of sexp->xml libraries.
nickik 16y ago
Are there any big diffrence to clojure's hiccup?

71 points by sgrove 16y ago

rorrr 16y ago

Sensible html templating exists already. It's called HAML. http://haml-lang.com/ http://haml-lang.com/

sgrove 16y ago
Actually, while developing this I was really struck by the similarities. With the automatic indenting in emacs for lisp (really quite good), it looks very similar, and it's impossible to leave a tag unclosed, but there's a further gain for using parentheses in that with paren-highlighting, you can tell exactly which tag your in. While I converted several dozen pages to this system from straight html, I was saved again and again by this. HAML's great, and I disagree with the other "I hate haml" post on the frontpage today, but this approach actually alleviates the issues pointed out with haml's whitespace-is-meaningful approach. Anyway, it's all in good fun.
- rapind 16y ago
  Yup I'm a huge haml fan, and saw the similarities. I thought your post was great. Very clean, and you can keep optimizing it. I'm learning CL right now and this added some fuel to the motivation fire ;)

Vekz 16y ago

I am spoiled by nodejs and its live server side DOM rendering. Having one layer of template code reusable on the client and server is too powerful. You would think people would be sick of writing duplicate server templates and ajax templates. To me anything else is a step in the wrong direction.

sgrove 16y ago
I think you're pointing out Dave Glass' technique over at yahoo, right? This one http://www.yuiblog.com/blog/2010/04/09/node-js-yui-3-dom-manipulation-oh-my/ http://www.yuiblog.com/blog/2010/04/09/node-js-yui-3-dom-man... ? The ideas there are definitely very powerful... and just plain sensible. I'd like to find a way to work this in, but it will take time. Do you see any fundamental reason a lisp equivalent of this technique couldn't be reproduced, using something like parenscript?
- sedachv 16y ago
  It's been done already: http://code.google.com/p/cl-closure-template/ http://code.google.com/p/cl-closure-template/
  - sgrove 16y ago
    That's extremely different from the technique that Glass is demoing - in fact, it's the antithesis. It takes care of unifying the server/client templates (by introducing an unnecessary layer, imo), but it's very obviously string-based output rather than live DOM manipulation like Enlive or YUI on node.js

swannodette 16y ago

I still think the new gold standard for functional and extensible HTML templating is Enlive - people should look into this before coming up with variations of the same old wheel - https://github.com/cgrand/enlive https://github.com/cgrand/enlive

sgrove 16y ago
Enlive is awesome, I used it while working in clojure quite a bit. I definitely prefer it, personally, to something like Clojure's hiccup (which vana-templating is very similar to). The tradeoff is that new-comers who are already familiar with front-end design are going to find it tough to transition - it's a fundamental problem: do you optimize for the newcomer, or for the experienced developer? Enlive has a lot in common with Vekz' suggestion elsewhere in the comments, and I feel like there's some unifying theme underlying all of this, that should make it easy for newcomers, but allow experienced developers to transform the DOM programmatically. I'm searching for it now, and I'm sure there'll be a lot of false-starts and dead-ends, but the only sure way not to get there is to not try.
nomurrcy 16y ago
I agree completely. Enlive has really blown me away.
xtacy 16y ago
I think there's a difference: Enlive is meant to manipulate the DOM using familiar CSS/jQuery like selectors. The article however, is a way to _write_ the HTML in the first place. I've myself been using a declarative style HTML output engine that I wrote in Python. It's simple, easier, outputs correct HTML and it merges well with my workflow. I don't have to write HTML, nor JavaScript to programmatically generate what I need. It may sound naive, but I find myself comfortable indenting code in Python than write matching HTML tags. If there are errors, the Python code wouldn't run, but HTML code would still render, though it wouldn't be what I want. Relevant: http://code.google.com/p/zen-coding/ http://code.google.com/p/zen-coding/

jasonlotito 16y ago

Anything that requires you to write HTML in something that isn't HTML isn't sensible. Sure, go ahead, add tokens to the HTML that can be parsed out and rendered differently, stored in a cache or something. But as soon as you're forced to write in another language to get HTML, you've lost. Edit: To elaborate, as some have asked: I'm speaking in terms of HTML templating. Having the software render your HTML form for you, or rending out a table full of data is fine. But by and large, the HTML should remain HTML. Granted, in this case, your not writing HTML, your writing code to generate a form. Not sitting there adding HTML attributes, which is handled for you. As to why I disagree with this approach? At it's core, portability. You can now no longer take an HTML file and use it. You have to rewrite everything into your strange, personal, customized, non-standard language and hope everything comes out alright. Their is also the learning curve in bringing anyone else on board. You also suddenly lack the ability to use any of the tools associated with HTML. Abstraction for the sack of abstraction is silly. If you have multiple functions for every single tag, your essentially taking HTML and just making the syntax your own. This "(ul '(id "navbar" class "horizontal_list")" is no better than HTML, but it isn't special or unique. It's cumbersome. Your also limited by the framework (which you hope will add new tags as they appear, and support the customization HTML offers). I'm not against HTML generating code. I'm against rewriting HTML as code. Maybe someone can explain how (ul '(id "navbar" class "horizontal_list")... is better than <ul id="navbar" class="horizontal_list">... Edit 2: Also, it should be fairly obviously, but all of this is of course just my opinion. =)

gnaritas 16y ago
Not true, the are advantages to programatically generated HTML using a mini language that simply can't be ignored. See http://seaside.st http://seaside.st
thecombjelly 16y ago
How have you lost? I write my HTML in a very similar way to the linked article, and I don't think I lose at all. In fact, I much prefer it. It is more consistent. All of my code is now (in my case) Scheme. This makes it easier to change things as you don't have to go between two different languages. And it is simpler. My favorite reason is the fact that I can use S-Exps. I no longer have to type the ending tag. I remove the possibility for spelling it wrong, or forgetting the slash. And it takes less space and is easier to get the placement correct, as you just utilize paren matching.

marcusbooster 16y ago

Not trying to pick on this particular project because it seems cool enough, but, I don't get all this html templating. With ajax and websockets on the way, why are we still generating html in the application? This all seems like a solution to a problem we should be leaving behind.

sgrove 16y ago
You're definitely right, actually. I've been working a lot in clojure, node.js, and html5 <canvas> recently on various projects (more posts later), and there's a definitely feeling of reinventing the wheel, and doing it poorly. That said, there are a ton of different approaches - At the oak.js meetup, I saw a sammy.js app embedded into couchdb. Definitely not my style, but a cool idea anyway. I'm very eager to get this into several html5 apps that are currently node.js or rails backends and for particular reasons would be well suited for a lisp backend. The goal is to tie in very closely with the html5 landscape, and drop a lot of the legacy stuff that other frameworks care about.
lemming 16y ago
on the way You said it - websockets aren't here yet. Plus I think there are still plenty of good applications for static HTML. For starters, even full AJAX apps should work without JS enabled if you care about accessibility.
- ekidd 16y ago
  For starters, even full AJAX apps should work without JS enabled if you care about accessibility. This should be just about possible using Node.js these days: You can run your entire client-side templating system—and even the <canvas>—on the server, and then server static HTML and PNGs to the browser. Unfortunately, quite a bit of elbow grease is still required.
  - lemming 16y ago
    But my point is - you're still templating. The only thing you're gaining is using the same language on the client and the server. I'm not saying that's a small thing, but the OP could have achieved the same using parenscript. The GP seemed to be implying that we should only generate our HTML programatically on the client. Thinking about it more, I don't agree with this. I don't write webapps these days, but in our platform we do a lot of code generation. We use a programmatic method to do this because we have to manipulate the intermediate representation a lot - add or remove fields added by earlier phases in the generation etc. This is much more flexible but it's much harder to maintain, and looking at the code generation code it's very hard to see what the end result will be. If we were simply generating code in a single pass I'd definitely use templating, it's just much simpler, easier to develop, and easier to see what the end result will be. Add in the need with HTML to work with designers and I think templating will be with us for some time to come.

zachbeane 16y ago

You should learn about WITH-OUTPUT-TO-STRING. (string-downcase (string thing)) should be written (string-downcase thing) The fdefinition trick is not a good one. Define the functions at compile-time, not at runtime.

chaitanya 16y ago
> (string-downcase (string thing)) should be written (string-downcase thing) I learnt this trick only after working on CL for a couple of years!
- zachbeane 16y ago
  Maybe http://l1sp.org/search/designator http://l1sp.org/search/designator will teach you a few more tricks and save you some time...
  - chaitanya 16y ago
    Nice. Today I learnt about the file position designator :)
sgrove 16y ago
I was originally using macros, but I prefer to use functions instead. I'll give it some thought, you may be right in this case. And thanks for the (string-downcase thing) trick, I'll update that right now :)
- zachbeane 16y ago
  Functions are good. Defining them at runtime with (setf (fdefinition ...) ...) is rarely good.

astine 16y ago

This looks to me like an HTML generation library, not an HTML templating tool. Common Lisp has a number of both, and I'm wondering what advantage this would have over cl-who, for example?

chaitanya 16y ago

For those who didn't know already, cl-who (http://weitz.de/cl-who/ http://weitz.de/cl-who/) is a well known sexp-to-html converter. This one works a little differently from cl-who (uses functions instead of macros), but the input is similar.

philjackson 16y ago

One a little like it for emacs: https://github.com/philjackson/xmlgen https://github.com/philjackson/xmlgen

DanielRibeiro 16y ago

Can also be done, in runtime including, more cleanly and without parenthesis with coffeescript: http://news.ycombinator.com/item?id=1786002 http://news.ycombinator.com/item?id=1786002

its2010already 16y ago

The problem I always see with practically all templating systems is the lack of proper escaping of input data. For example, the function 'tag-name' really needs to HTML-escape the input parameter content. The simple example (tag-name "div" "Hello world") breaks down when "Hello World" is replaced by a variable reference. Fortunately this is an relatively easy problem to solve. A better definition for tag-name is something like: (defun tag-name (name content) (html (format nil "<~A>~A</~A>" name (to-html content) name)) The function to-html will return the HTML-escaped version of content. The function html returns an object/string that marked as already being HTML-escaped. They work together so that (to-html (html x)) just returns x. A similar type system can be set up to handle url escaping rules. If the templating system doesn't keep track of what is escaped and what isn't then that burden is left to the programmer with all of the associated risks of producing malformed output and creating opportunities for injection attacks.

jhrobert 16y ago

Am I missing something or is this code defining "global scope" functions whose names include things like a, b, i, p, command, etc...? What are the chances that such global names will collide with something else?

bodhi 16y ago
You could easily put them into a separate package if you wish: http://gigamonkeys.com/book/programming-in-the-large-packages-and-symbols.html http://gigamonkeys.com/book/programming-in-the-large-package...

KirinDave 16y ago

Or don't template at all. Let your designers design documents, then you write code to modify the markup at key integration points: https://github.com/cgrand/enlive https://github.com/cgrand/enlive This has some really nice properties: 1. The only thing your devs and designers need to agree on is some basic semantic markup and tag IDs. Then they can make a full comp (complete with sample data) for their purposes. You simply rewrite that data. 2. The approach is very declarative, which makes it a lot easier to reason about what is actually being shown and what is actually changing. 3. It's amenable to all the same functional properties of the sexp->xml stuff that you see here. Enlive's approach is a nearly 100% win for doing site templating over these sorts of sexp->xml libraries.

nickik 16y ago

Are there any big diffrence to clojure's hiccup?