17 ms·

Lisp and the Web: Creating Web Apps Through Lisp and GCE

162 points by abhinav22 5y ago

fractallyte 5y ago
You could cut out a chunk of configuration by using Caddy web server (https://caddyserver.com/ https://caddyserver.com/) instead of Nginx. Configuration is very simple (for example, reverse proxying is just three lines - for the entire config!), and SSL works out of the box!
- abhinav22 5y ago
  Thanks! I’ll have a look. I find Certbot a bit fragile to use, but not sure if that’s just the problem space (since you have to verify domain control before issuing SSL certificates). Perhaps this will be a more robust approach
MawKKe 5y ago
https://github.com/ashok-khanna/common-lisp-by-example/blob/41544bd5838a032964be5472812ad6d475f5ca19/example-server/blog-server.lisp#L27 https://github.com/ashok-khanna/common-lisp-by-example/blob/... Formatting queries like this is not a good idea at all. See: https://en.wikipedia.org/wiki/SQL_injection https://en.wikipedia.org/wiki/SQL_injection
- fidesomnes 5y ago
  is there a better method for lisp?
  - hashkb 5y ago
    Something like https://github.com/fukamachi/sxql https://github.com/fukamachi/sxql
  - abhinav22 5y ago
    The issue with my code is that somebody can put in arbitrary SQL queries within the data field (I think - not sure if it is 100% possible because their data would be enclosed within the programs SQL query, which may break their attempts) Best way around it is to put in checks to prevent the code from being anything nefarious. I’ll have a think and figure out what’s the best way of doing this, but I assume not constructing the sql strings from scratch but rather using postmodern’s API (postmodern is Common Lisp’s excellent interface to PostgreSQL) makes sense, or the alternate solution recommended in the other reply to your post.
    - stevelosh 5y ago
      Postmodern can do the right thing if you use prepared queries. See `defprepared`.
      - abhinav22 5y ago
        Thanks! I will use that
        abhinav22 5y ago
        For those interested, refer to https://sites.google.com/site/sabraonthehill/postmodern-examples/postmodern-parametized-statements https://sites.google.com/site/sabraonthehill/postmodern-exam... Thanks also to kagevf on #lisp irc channel for his help here
- abhinav22 5y ago
  That’s a good point. I will update the code to prevent any potential SQL injections so that others follow the right practice. Very nice catch! Thank you very much.
abhinav22 5y ago
Author: I wanted to consolidate some of the steps I did to get the basic functionality for Common Lisp web applications (CRUD, User Authentication, Domains, etc.) as its a big pain to keep revisiting and troubleshooting which step I did wrong. So I saved these notes in guide form, hopefully its useful for others. The trick is to extend a lot of this, all the hard parts are here - the rest is just basic Lisp, HTML, JavaScript, building upon this platform. Highly recommend Lisp for Web Development, its *much, much better* than any of the other frameworks out there! So easy to use. Do learn HTML, CSS & JavaScript though. Esp. JavaScript as you would want to offload some of the computation to client-side to reduce your server costs. Big Thanks to u/linnilbobo on Reddit for his code on user authentication in Hunchentoot - that was one of the missing pieces for me.
- manjana 5y ago
  > Highly recommend Lisp for Web Development, its much, much better than any of the other frameworks out there! Could you elaborate a bit on this? I've never written a single line of Lisp so I get a bit curious.
  - abhinav22 5y ago
    There are many views on this (so please don’t take my word as gospel), but from my experience, Lisp gives the following benefits: - A fantastic and easy to use open source web server in Hunchentoot - Very easy integration into databases like PostgreSQL. Just looking at my guide shows how easy all this is (assuming you know lisp, but if you look at the number of lines of code - it’s not that much) - Live debugging - I.e. I can recompile certain functions and don’t need to restart my web server! That’s a huge win in my opinion - It’s hard to explain, but I find Lisp very transparent. With knowledge of JavaScript, HTML, CSS and Lisp, I am very comfortable building any web app. Whereas when I use various web frameworks like react, angular, etc, I have to spend so much learning it’s idiosyncrasies (which to me is a waste of time since these frameworks keep changing), and then usually I hit a brick wall somewhere. Other frameworks in my opinion obfuscate too much - they solve for easy wins (which they do), but anything more complex and you are fighting against the language, spending hours on Stack Overflow for solutions to what should be simple problems. In my opinion I spent 1-2 weeks setting up a CRUD app with Angular and still don’t know the best way to use it. With Lisp, I did the same in one day. It was that experience that pushed to me to learning and using Lisp (I was a complete beginner at the time). For me that was enough proof - Stuff like NPM I don’t like because there are so many dependencies (which break from time to time) and I’m worried somebody could be putting in bugs / trackers in these packages. I don’t like having many dependencies - Ability to write DSLs (Domain Specific Language) where I can easily write macros to generate JavaScript and HTML for me from Lisp, increasing the cohesiveness of the whole program All that said, Lisp by itself is not enough. HTML/CSS/JavaScript are much more important - Lisp is the magic server side plug for all of this, and in my opinion, better than server side JavaScript because Lisp is a very well thought and well featured language (JavaScript is as well, but I think Lisp, at least to me, is slightly more enjoyable to write in). I find the issue with Lisp is that it’s not used much, so outside of some very good packages, there are many parts which are easier to do in other languages / frameworks with much bigger ecosystems. For me, web development is the exception, because all you need is HTML/CSS/JS and a good server side language. Other languages are also very good, so this is just a personal opinion. If you like to invest your time in one core language, and not have to keep learning new web frameworks, I recommend Lisp. It’s very good for solo programmers or small teams, but I do think it’s not suitable in larger teams as I do find lisp code can be a bit harder to read and also most people don’t use lisp. So in my uneducated view - if you have spare time, if you like working from first principles and have a scientific / mathematical mindset, and if you are working alone or in small teams, I highly recommend Lisp for web dev. If you are more business minded and want to scale your project quickly and hire more developers, then using an off the shelf package makes more sense. If you are time poor, stick to what you know - the differences aren’t THAT great, and it’s better to build something now than to spend most of your time learning a new language (unless you have time to devote to it - in which I highly recommend learning Lisp :) It really makes me smile every time I code in Lisp :)
phoe-krk 5y ago
Not to be confused with the book Lisp for the Web [2015] by Adam Tornhill, https://leanpub.com/lispweb https://leanpub.com/lispweb
- tartoran 5y ago
  This looks interesting. I started learning Scheme but felt that CL has some interesting pieces and this book seems quite inviting. I wish I could order a paperback for this book
kreetx 5y ago
Seems to be paywalled?
- abhinav22 5y ago
  Argh! Fixed now - sorry about that. Pressed the wrong button when publishing on Medium (ps I really like Medium, sometimes it gets a bit of a rap, but it’s very easy to write and they are quite user friendly in my opinion at least :)
  - notagoodidea 5y ago
    To be picky, Medium may be writer-friendly but never was really reader-friendly. The page are heavy with banners blocking the view, no possibilities to know in advance which article is paywalled or not. I do not have a problem with authors publishing on Medium, just Medium itself. Besides that, nice and interesting write up!
    - abhinav22 5y ago
      Thanks! I agree with you, it’s not perfect for readers and until I caved and signed up as a member, I also was annoyed at hitting my monthly limit of articles. They have amazing SEO so inevitably many articles I need while researching a solution led me to Medium.com It’s just a very hard market to serve (long form reading / journalism etc). That’s why I’m not that bullish on substack because it’s hard to make it good for readers and profitable for writers / the startup. Medium paid me $10, and I have about 10,000 reads in the last year. I didn’t do it for the money, but that’s an indication for those who do want to write for a side income - it’s a hard business. For writing, its very good - to the extent, I use it as one of my main word processors lol. Then just copy it to something else for formatting. Very distraction free and the font is nice to make you want to write more. It would be great if a new competitor did what Medium did, but made it better for readers. IMO the business model should be like Wordpress but for writers, together with an overall platform with high quality SEO. Make the money selling the software and a bit with memberships on the main site. I feel for Medium and it’s writers - there’s so little money in it, that to some extent they have to resort to click bait titles, bad UI for readers, and high amount of not amazing quality content. Perhaps a donation model is best.
bilegeek 5y ago
Thank you for this. You get into the intermediate steps, which just about every other tutorial I've seen on this glosses over.
- abhinav22 5y ago
  You are most welcome :) Yes, this hopefully is a complete guide from start to finish and without any gaps (which are not easy for us to solve without having a reasonable amount of background knowledge!) Thanks for the kind words
dj_mc_merlin 5y ago
I skimmed over the article, but I think it misses my favourite part: running a Swank server for live debugging! Something along the lines of: (bt:make-thread (lambda () (swank:create-server :port port :dont-close t)) then connect with M-x slime-connect I've debugged many tricky to reproduce issues like this. Also it's just damn cool.
- abhinav22 5y ago
  Would you be able to expand on this? I’m very basic when it comes to debugging in Lisp :) So you are running a second server to test issues away from production? I thought this is possible in other languages / frameworks, so I must be missing something I read the cook book but still didn’t get the main benefits of it. However the quote was very cool: That’s how Ron Garret debugged the Deep Space 1 spacecraft from the earth in 1999: We were able to debug and fix a race condition that had not shown up during ground testing. (Debugging a program running on a $100M piece of hardware that is 100 million miles away is an interesting experience. Having a read-eval-print loop running on the spacecraft proved invaluable in finding and fixing the problem.
  - kagevf 5y ago
    This repo has an example of running hunchentoot with a swank server: https://github.com/vseloved/flight-recorder/blob/master/hunch.lisp https://github.com/vseloved/flight-recorder/blob/master/hunc...
  - dj_mc_merlin 5y ago
    You have your normal application, accessible on some port serving HTTP requests (or whatever). Then you start a thread (with `bt`, bordeaux-threads, a very common multi-threading library) and run a Swank server listening on some other port on it. Swank is the backend used by SLIME, it allows manipulation of the Lisp environment. A remote server is still the same Lisp environment as when you run it on your machine, so you can use Emacs to connect to it (M-x slime-connect) and debug live. Swank doesn't allow remote connections though, otherwise anyone could connect and mess with your server. You must connect through the loopback interface. Easiest way is to set up an ssh tunnel, like: ssh abc@example.com -L 4200:localhost:4200 Whether you can call the Swank server a second server or not is a somewhat philosophical question I guess. There's one application serving different kinds of requests on different ports.
exdsq 5y ago
Does anyone here use Common Lisp for their day job? I'd like to try it out some time (and am currently working through Paradigms of AI Programming by Norvig) but it feels like the language is so bendable that it'd be a minefield for a large team.
- fidesomnes 5y ago
  I used to work at an AI firm that used it. Large Lisp codebases are a minefield with near impossible security and debugging issues. I still love it as a programming language tho.
  - medo-bear 5y ago
    There are a number of large companies that use CL quite effectively for safety critial purposes. Can you please give more details about your experience?
    - exdsq 5y ago
      Out of interest can you give some examples?
      - medo-bear 5y ago
        siscog in portugal is one example but there are others, historical and current. it is also used in quantum computing and financial markets. gramarly and google flights are perhaps the most famous current users of the software. edit: i adjusted my answer because i thought i was answering to parent
  - abhinav22 5y ago
    I would love to hear more on the security challenges and debugging issues, if you have time. It’s not used as much in industry, so any insights are really very valuable. Also so cool that you worked in AI! That’s my dream field :)
    - 5y ago
kaeland 5y ago
I really enjoyed your article. Thank you for writing this.

162 points by abhinav22 5y ago

You could cut out a chunk of configuration by using Caddy web server (https://caddyserver.com/ https://caddyserver.com/) instead of Nginx. Configuration is very simple (for example, reverse proxying is just three lines - for the entire config!), and SSL works out of the box!

abhinav22 5y ago
Thanks! I’ll have a look. I find Certbot a bit fragile to use, but not sure if that’s just the problem space (since you have to verify domain control before issuing SSL certificates). Perhaps this will be a more robust approach

MawKKe 5y ago

https://github.com/ashok-khanna/common-lisp-by-example/blob/41544bd5838a032964be5472812ad6d475f5ca19/example-server/blog-server.lisp#L27 https://github.com/ashok-khanna/common-lisp-by-example/blob/... Formatting queries like this is not a good idea at all. See: https://en.wikipedia.org/wiki/SQL_injection https://en.wikipedia.org/wiki/SQL_injection

fidesomnes 5y ago
is there a better method for lisp?
- hashkb 5y ago
  Something like https://github.com/fukamachi/sxql https://github.com/fukamachi/sxql
- abhinav22 5y ago
  The issue with my code is that somebody can put in arbitrary SQL queries within the data field (I think - not sure if it is 100% possible because their data would be enclosed within the programs SQL query, which may break their attempts) Best way around it is to put in checks to prevent the code from being anything nefarious. I’ll have a think and figure out what’s the best way of doing this, but I assume not constructing the sql strings from scratch but rather using postmodern’s API (postmodern is Common Lisp’s excellent interface to PostgreSQL) makes sense, or the alternate solution recommended in the other reply to your post.
  - stevelosh 5y ago
    Postmodern can do the right thing if you use prepared queries. See `defprepared`.
    - abhinav22 5y ago
      Thanks! I will use that
      - abhinav22 5y ago
        For those interested, refer to https://sites.google.com/site/sabraonthehill/postmodern-examples/postmodern-parametized-statements https://sites.google.com/site/sabraonthehill/postmodern-exam... Thanks also to kagevf on #lisp irc channel for his help here
abhinav22 5y ago
That’s a good point. I will update the code to prevent any potential SQL injections so that others follow the right practice. Very nice catch! Thank you very much.

abhinav22 5y ago

Author: I wanted to consolidate some of the steps I did to get the basic functionality for Common Lisp web applications (CRUD, User Authentication, Domains, etc.) as its a big pain to keep revisiting and troubleshooting which step I did wrong. So I saved these notes in guide form, hopefully its useful for others. The trick is to extend a lot of this, all the hard parts are here - the rest is just basic Lisp, HTML, JavaScript, building upon this platform. Highly recommend Lisp for Web Development, its *much, much better* than any of the other frameworks out there! So easy to use. Do learn HTML, CSS & JavaScript though. Esp. JavaScript as you would want to offload some of the computation to client-side to reduce your server costs. Big Thanks to u/linnilbobo on Reddit for his code on user authentication in Hunchentoot - that was one of the missing pieces for me.

manjana 5y ago
> Highly recommend Lisp for Web Development, its much, much better than any of the other frameworks out there! Could you elaborate a bit on this? I've never written a single line of Lisp so I get a bit curious.
- abhinav22 5y ago
  There are many views on this (so please don’t take my word as gospel), but from my experience, Lisp gives the following benefits: - A fantastic and easy to use open source web server in Hunchentoot - Very easy integration into databases like PostgreSQL. Just looking at my guide shows how easy all this is (assuming you know lisp, but if you look at the number of lines of code - it’s not that much) - Live debugging - I.e. I can recompile certain functions and don’t need to restart my web server! That’s a huge win in my opinion - It’s hard to explain, but I find Lisp very transparent. With knowledge of JavaScript, HTML, CSS and Lisp, I am very comfortable building any web app. Whereas when I use various web frameworks like react, angular, etc, I have to spend so much learning it’s idiosyncrasies (which to me is a waste of time since these frameworks keep changing), and then usually I hit a brick wall somewhere. Other frameworks in my opinion obfuscate too much - they solve for easy wins (which they do), but anything more complex and you are fighting against the language, spending hours on Stack Overflow for solutions to what should be simple problems. In my opinion I spent 1-2 weeks setting up a CRUD app with Angular and still don’t know the best way to use it. With Lisp, I did the same in one day. It was that experience that pushed to me to learning and using Lisp (I was a complete beginner at the time). For me that was enough proof - Stuff like NPM I don’t like because there are so many dependencies (which break from time to time) and I’m worried somebody could be putting in bugs / trackers in these packages. I don’t like having many dependencies - Ability to write DSLs (Domain Specific Language) where I can easily write macros to generate JavaScript and HTML for me from Lisp, increasing the cohesiveness of the whole program All that said, Lisp by itself is not enough. HTML/CSS/JavaScript are much more important - Lisp is the magic server side plug for all of this, and in my opinion, better than server side JavaScript because Lisp is a very well thought and well featured language (JavaScript is as well, but I think Lisp, at least to me, is slightly more enjoyable to write in). I find the issue with Lisp is that it’s not used much, so outside of some very good packages, there are many parts which are easier to do in other languages / frameworks with much bigger ecosystems. For me, web development is the exception, because all you need is HTML/CSS/JS and a good server side language. Other languages are also very good, so this is just a personal opinion. If you like to invest your time in one core language, and not have to keep learning new web frameworks, I recommend Lisp. It’s very good for solo programmers or small teams, but I do think it’s not suitable in larger teams as I do find lisp code can be a bit harder to read and also most people don’t use lisp. So in my uneducated view - if you have spare time, if you like working from first principles and have a scientific / mathematical mindset, and if you are working alone or in small teams, I highly recommend Lisp for web dev. If you are more business minded and want to scale your project quickly and hire more developers, then using an off the shelf package makes more sense. If you are time poor, stick to what you know - the differences aren’t THAT great, and it’s better to build something now than to spend most of your time learning a new language (unless you have time to devote to it - in which I highly recommend learning Lisp :) It really makes me smile every time I code in Lisp :)

phoe-krk 5y ago

Not to be confused with the book Lisp for the Web [2015] by Adam Tornhill, https://leanpub.com/lispweb https://leanpub.com/lispweb

tartoran 5y ago
This looks interesting. I started learning Scheme but felt that CL has some interesting pieces and this book seems quite inviting. I wish I could order a paperback for this book

kreetx 5y ago

Seems to be paywalled?

abhinav22 5y ago
Argh! Fixed now - sorry about that. Pressed the wrong button when publishing on Medium (ps I really like Medium, sometimes it gets a bit of a rap, but it’s very easy to write and they are quite user friendly in my opinion at least :)
- notagoodidea 5y ago
  To be picky, Medium may be writer-friendly but never was really reader-friendly. The page are heavy with banners blocking the view, no possibilities to know in advance which article is paywalled or not. I do not have a problem with authors publishing on Medium, just Medium itself. Besides that, nice and interesting write up!
  - abhinav22 5y ago
    Thanks! I agree with you, it’s not perfect for readers and until I caved and signed up as a member, I also was annoyed at hitting my monthly limit of articles. They have amazing SEO so inevitably many articles I need while researching a solution led me to Medium.com It’s just a very hard market to serve (long form reading / journalism etc). That’s why I’m not that bullish on substack because it’s hard to make it good for readers and profitable for writers / the startup. Medium paid me $10, and I have about 10,000 reads in the last year. I didn’t do it for the money, but that’s an indication for those who do want to write for a side income - it’s a hard business. For writing, its very good - to the extent, I use it as one of my main word processors lol. Then just copy it to something else for formatting. Very distraction free and the font is nice to make you want to write more. It would be great if a new competitor did what Medium did, but made it better for readers. IMO the business model should be like Wordpress but for writers, together with an overall platform with high quality SEO. Make the money selling the software and a bit with memberships on the main site. I feel for Medium and it’s writers - there’s so little money in it, that to some extent they have to resort to click bait titles, bad UI for readers, and high amount of not amazing quality content. Perhaps a donation model is best.

bilegeek 5y ago

Thank you for this. You get into the intermediate steps, which just about every other tutorial I've seen on this glosses over.

abhinav22 5y ago
You are most welcome :) Yes, this hopefully is a complete guide from start to finish and without any gaps (which are not easy for us to solve without having a reasonable amount of background knowledge!) Thanks for the kind words

dj_mc_merlin 5y ago

I skimmed over the article, but I think it misses my favourite part: running a Swank server for live debugging! Something along the lines of: (bt:make-thread (lambda () (swank:create-server :port port :dont-close t)) then connect with M-x slime-connect I've debugged many tricky to reproduce issues like this. Also it's just damn cool.

abhinav22 5y ago
Would you be able to expand on this? I’m very basic when it comes to debugging in Lisp :) So you are running a second server to test issues away from production? I thought this is possible in other languages / frameworks, so I must be missing something I read the cook book but still didn’t get the main benefits of it. However the quote was very cool: That’s how Ron Garret debugged the Deep Space 1 spacecraft from the earth in 1999: We were able to debug and fix a race condition that had not shown up during ground testing. (Debugging a program running on a $100M piece of hardware that is 100 million miles away is an interesting experience. Having a read-eval-print loop running on the spacecraft proved invaluable in finding and fixing the problem.
- kagevf 5y ago
  This repo has an example of running hunchentoot with a swank server: https://github.com/vseloved/flight-recorder/blob/master/hunch.lisp https://github.com/vseloved/flight-recorder/blob/master/hunc...
- dj_mc_merlin 5y ago
  You have your normal application, accessible on some port serving HTTP requests (or whatever). Then you start a thread (with `bt`, bordeaux-threads, a very common multi-threading library) and run a Swank server listening on some other port on it. Swank is the backend used by SLIME, it allows manipulation of the Lisp environment. A remote server is still the same Lisp environment as when you run it on your machine, so you can use Emacs to connect to it (M-x slime-connect) and debug live. Swank doesn't allow remote connections though, otherwise anyone could connect and mess with your server. You must connect through the loopback interface. Easiest way is to set up an ssh tunnel, like: ssh abc@example.com -L 4200:localhost:4200 Whether you can call the Swank server a second server or not is a somewhat philosophical question I guess. There's one application serving different kinds of requests on different ports.

exdsq 5y ago

Does anyone here use Common Lisp for their day job? I'd like to try it out some time (and am currently working through Paradigms of AI Programming by Norvig) but it feels like the language is so bendable that it'd be a minefield for a large team.

fidesomnes 5y ago
I used to work at an AI firm that used it. Large Lisp codebases are a minefield with near impossible security and debugging issues. I still love it as a programming language tho.
- medo-bear 5y ago
  There are a number of large companies that use CL quite effectively for safety critial purposes. Can you please give more details about your experience?
  - exdsq 5y ago
    Out of interest can you give some examples?
    - medo-bear 5y ago
      siscog in portugal is one example but there are others, historical and current. it is also used in quantum computing and financial markets. gramarly and google flights are perhaps the most famous current users of the software. edit: i adjusted my answer because i thought i was answering to parent
- abhinav22 5y ago
  I would love to hear more on the security challenges and debugging issues, if you have time. It’s not used as much in industry, so any insights are really very valuable. Also so cool that you worked in AI! That’s my dream field :)
  - 5y ago

kaeland 5y ago

I really enjoyed your article. Thank you for writing this.