4 ms·

I wrote some high-performance lisp applications in production. My main was was a threaded queue worker, driven off beanstalkd, that processed large volumes of d

I wrote some high-performance lisp applications in production. My main was was a threaded queue worker, driven off beanstalkd, that processed large volumes of data. This was highly successful and didn't require overly-provisioned servers (was using CCL/linux x64/linode VPS). That said, I don't think I'll ever use lisp for front-facing web stuff again. I wrote cl-async, and afterwards and used it in production in a few places, then a bit later started using node heavily. The tooling for CL is just not there. Quicklisp is great, but not being able to pick and choose package versions is limiting, and having to rewrite every single thing for async was a pain (since CL doesn't have continuations, any sort of async that avoids CPS is impossible). I was hoping I would get more community support on cl-async, and I did get many hands helping, but eventually it just became too much to maintain. So if you're doing network stuff on CL, it's probably best to just use threaded, which IMO just doesn't handle the same amount of load evented does, at least when dealing with web APIs. One thing I used lisp for briefly which I want to do again soon is game dev. Being able to redefine/fix portions of your program while it's running is fantastic, especially when you have huge amounts of state to deal with like you do in games. I wouldn't exactly call that running CL in production, though.

fiddlerwoaroof 9y ago
I'm a little bit disappointed that you've decided to stop maintaining cl-async, although I appreciate the difficulty of maintaining such a project.
- orthecreedence 9y ago
  Hopefully it's still useful! The nice thing about CL is that it never fucking changes so you can write something and 25 years later it still runs. Really, the only moving targets are the bundled ASDF version implementations ship with and libuv itself. I've had some reports of cl-libuv not compiling against the latest libuvs, but that shouldn't be too hard to fix (I still merge PRs =]). All in all, I think I just hit one too many problems using cl-async/wookie in production to justify using it vs Node which just kind of works and has a plethora of maintained packages.
jaccarmac 9y ago
FWIW, Qlot solves the version pinning problem. Not to discredit your experience, just wanted to say that there are solutions today for the same issue. I'm interested exactly what the pain points are with CPS. Do you mean manually rewriting in continuations is tedious or is there a more insidious issue. I haven't tried writing async code in Lisp but macros would appear to eliminate continuation-writing-by-hand.
- orthecreedence 9y ago
  > macros would appear to eliminate continuation-writing-by-hand They don't =]. There's cl-cont, but it's like building a house on a foundation of toothpicks. Not to discount the work the author put in, but you can only do very specific things with it, and last time I looked, error/exception handling was pretty much completely missing. > I'm interested exactly what the pain points are with CPS Basically, for example, if I'm going to convert cl-postgres to use async, I have to rip out the protocol parsing from the networking (they are often globbed together) then I have to replace the networking with async and tie it back into the protocol parsing. This in itself isn't too bad, but then anything that calls the new async cl-postgres has to be converted to expect async...all the way up. This can be somewhat alleviated by promises (https://orthecreedence.github.io/blackbird/ https://orthecreedence.github.io/blackbird/) but they still don't change the fact that the entire stack is unwound once you call into async. Continuations would allow building green threads/coroutines which would eliminate CPS and allow you to preserve your stack while calling into async, but CL is pretty much written in stone to not have continuations (or coroutines et al) at this point. Qlot looks handy, thanks for the tip.
  - TeMPOraL 9y ago
    > then I have to replace the networking with async and tie it back into the protocol parsing. This in itself isn't too bad, but then anything that calls the new async cl-postgres has to be converted to expect async...all the way up. Isn't this the very consequence of going async? I'm reminded of the article on this recently featured here[1]. Once you introduce async, you poison everything up to the event loop (which, with usual async implementations, you don't even see or control). [0] - http://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/ http://journal.stuffwithstuff.com/2015/02/01/what-color-is-y... + https://news.ycombinator.com/item?id=16732948 https://news.ycombinator.com/item?id=16732948
sigjuice 9y ago
About picking and choosing package versions; isn’t it straightforward to keep whatever versions of packages you want under ~/quicklisp/local-projects ? Or is there a need for a more sophisticated tool for this?
- orthecreedence 9y ago
  This works, sure, but when setting up a new server, you're going to have to remember which package version you want for each of your dependencies. It's much easier to just have a Cargo.toml or package.json and run one command that verifies and installs exactly what you need. I guess containerization solves much of this problem now, but back when I was heavily into CL, containers were just starting to peek their heads out into the world at large.
  - patrickg_zill 9y ago
    Solaris and openvz containers have been around since 2005 or so.
    - sigjuice 9y ago
      Solaris is not that accessible to most people the way Linux is. And containers are overkill when all you need is the ability to put things in a sub-directory.
martinflack 9y ago
I use cl-async and want to thank you for it. Anything further from your experience and retrospective working on the project would be invaluable to document somewhere for the CL community at some point. Just your insight here in the above comment is very interesting. Over time things may improve.
- orthecreedence 9y ago
  Thanks for the nice note! I'm glad you still find it useful. I do hope things improve over time. I'd love to come back to CL, and actually plan to sometime soon. I'd like to get my pet project, Turtl (which was using cl-async/Wookie in production and is now migrating to node), in a stable place (and hopefully making money from it at some point) and then start dipping my toe into game programming again. I'll almost definitely use CL over anything else. Maybe I'll write the GPG-verifying, version-pinning quicklisp wrapper I've always wanted while I'm at it =]. As far as retrospective, I think I suffered from the part of the lisp community I had always hoped wasn't true (or hoped I could change): people are much quicker to build their own version than to improve on something existing. I think if Wookie had gotten more attention and cl-async had had a few more drivers written (postgres, mysql, etc) it could have gotten a foothold. It didn't though, and I got tired of hacking through the digital reeds with my machete when I already knew javascript pretty well and node had all this stuff figured out already. It's funny, after using Express for ~4 years, I keep having the urge to go back and implement what I've learned into Wookie. I have to tell myself that it's probably best to just let it sit for now. On another note, my dog, Wookie, who the project was named after, died last week. So it seems even more appropriate to let the whole thing rest. I might re-use the name if I come back to the CL world, though. Seems its doppleganger project got more focus anyway.
vseloved 9y ago
Thanks for your work on cl-async/blackbird: we also used it in our recent project (and you should have gotten a couple of related requests)! I don't think you should be discouraged though: your contribution is very valuable and surely was noted, just that the whole async approach is, in my view, not in line with the general direction of where Lisp is most applicable, currently (we should admit that it's a niche language). After all, having an evented server means that it doesn't perform a lot of useful computations, most of the work is just serving some kind of existing content. I.e. you could say that the application logic is marginalized and commoditized. And Lisp is the best choice for the opposite cases, those that require and emphasize elaborate application logic and intense computation...

fiddlerwoaroof 9y ago

I'm a little bit disappointed that you've decided to stop maintaining cl-async, although I appreciate the difficulty of maintaining such a project.

orthecreedence 9y ago
Hopefully it's still useful! The nice thing about CL is that it never fucking changes so you can write something and 25 years later it still runs. Really, the only moving targets are the bundled ASDF version implementations ship with and libuv itself. I've had some reports of cl-libuv not compiling against the latest libuvs, but that shouldn't be too hard to fix (I still merge PRs =]). All in all, I think I just hit one too many problems using cl-async/wookie in production to justify using it vs Node which just kind of works and has a plethora of maintained packages.

jaccarmac 9y ago

FWIW, Qlot solves the version pinning problem. Not to discredit your experience, just wanted to say that there are solutions today for the same issue. I'm interested exactly what the pain points are with CPS. Do you mean manually rewriting in continuations is tedious or is there a more insidious issue. I haven't tried writing async code in Lisp but macros would appear to eliminate continuation-writing-by-hand.

orthecreedence 9y ago
> macros would appear to eliminate continuation-writing-by-hand They don't =]. There's cl-cont, but it's like building a house on a foundation of toothpicks. Not to discount the work the author put in, but you can only do very specific things with it, and last time I looked, error/exception handling was pretty much completely missing. > I'm interested exactly what the pain points are with CPS Basically, for example, if I'm going to convert cl-postgres to use async, I have to rip out the protocol parsing from the networking (they are often globbed together) then I have to replace the networking with async and tie it back into the protocol parsing. This in itself isn't too bad, but then anything that calls the new async cl-postgres has to be converted to expect async...all the way up. This can be somewhat alleviated by promises (https://orthecreedence.github.io/blackbird/ https://orthecreedence.github.io/blackbird/) but they still don't change the fact that the entire stack is unwound once you call into async. Continuations would allow building green threads/coroutines which would eliminate CPS and allow you to preserve your stack while calling into async, but CL is pretty much written in stone to not have continuations (or coroutines et al) at this point. Qlot looks handy, thanks for the tip.
- TeMPOraL 9y ago
  > then I have to replace the networking with async and tie it back into the protocol parsing. This in itself isn't too bad, but then anything that calls the new async cl-postgres has to be converted to expect async...all the way up. Isn't this the very consequence of going async? I'm reminded of the article on this recently featured here[1]. Once you introduce async, you poison everything up to the event loop (which, with usual async implementations, you don't even see or control). [0] - http://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/ http://journal.stuffwithstuff.com/2015/02/01/what-color-is-y... + https://news.ycombinator.com/item?id=16732948 https://news.ycombinator.com/item?id=16732948

sigjuice 9y ago

About picking and choosing package versions; isn’t it straightforward to keep whatever versions of packages you want under ~/quicklisp/local-projects ? Or is there a need for a more sophisticated tool for this?

orthecreedence 9y ago
This works, sure, but when setting up a new server, you're going to have to remember which package version you want for each of your dependencies. It's much easier to just have a Cargo.toml or package.json and run one command that verifies and installs exactly what you need. I guess containerization solves much of this problem now, but back when I was heavily into CL, containers were just starting to peek their heads out into the world at large.
- patrickg_zill 9y ago
  Solaris and openvz containers have been around since 2005 or so.
  - sigjuice 9y ago
    Solaris is not that accessible to most people the way Linux is. And containers are overkill when all you need is the ability to put things in a sub-directory.

martinflack 9y ago

I use cl-async and want to thank you for it. Anything further from your experience and retrospective working on the project would be invaluable to document somewhere for the CL community at some point. Just your insight here in the above comment is very interesting. Over time things may improve.

orthecreedence 9y ago
Thanks for the nice note! I'm glad you still find it useful. I do hope things improve over time. I'd love to come back to CL, and actually plan to sometime soon. I'd like to get my pet project, Turtl (which was using cl-async/Wookie in production and is now migrating to node), in a stable place (and hopefully making money from it at some point) and then start dipping my toe into game programming again. I'll almost definitely use CL over anything else. Maybe I'll write the GPG-verifying, version-pinning quicklisp wrapper I've always wanted while I'm at it =]. As far as retrospective, I think I suffered from the part of the lisp community I had always hoped wasn't true (or hoped I could change): people are much quicker to build their own version than to improve on something existing. I think if Wookie had gotten more attention and cl-async had had a few more drivers written (postgres, mysql, etc) it could have gotten a foothold. It didn't though, and I got tired of hacking through the digital reeds with my machete when I already knew javascript pretty well and node had all this stuff figured out already. It's funny, after using Express for ~4 years, I keep having the urge to go back and implement what I've learned into Wookie. I have to tell myself that it's probably best to just let it sit for now. On another note, my dog, Wookie, who the project was named after, died last week. So it seems even more appropriate to let the whole thing rest. I might re-use the name if I come back to the CL world, though. Seems its doppleganger project got more focus anyway.

vseloved 9y ago

Thanks for your work on cl-async/blackbird: we also used it in our recent project (and you should have gotten a couple of related requests)! I don't think you should be discouraged though: your contribution is very valuable and surely was noted, just that the whole async approach is, in my view, not in line with the general direction of where Lisp is most applicable, currently (we should admit that it's a niche language). After all, having an evented server means that it doesn't perform a lot of useful computations, most of the work is just serving some kind of existing content. I.e. you could say that the application logic is marginalized and commoditized. And Lisp is the best choice for the opposite cases, those that require and emphasize elaborate application logic and intense computation...