8 ms·
Running Lisp in Production
- BlanketLogic 11y agoVery informative. Thank you. Anyone here has any experience with the GCs of Allegro or LispWorks or any other commercial Lisp implementations?
- ChargingWookie 11y agoI've worked a lot with LispWorks and tuning the gc had the same method of programmatically calling a full GC after every N operations. We also found setting the gc threshold to high amount helped a bunch. Supposedly they have a concurrent GC in the works but I haven't played with it.
- BlanketLogic 11y agoThanks. > I've worked a lot with LispWorks and tuning the gc had the same method of programmatically calling a full GC after every N operations. What is "full GC", here? Do you mean, even the "older" generations? (Assuming Lispworks also has generational GC a la sbcl) In other words, Would it have helped if the implementation was a mark-compact rather than generational?
- ChargingWookie 11y agoWe normally used http://www.lispworks.com/documentation/lw60/LW/html/lw-712.htm http://www.lispworks.com/documentation/lw60/LW/html/lw-712.h... with the full set to nil.
- lispm 11y agoLispWorks has a lot of features in its GC. Years ago it was used in a demanding telco application, an ATM switch. It was also used on a space mission experiment. Generally the runtime is very very nice. Franz Inc uses Allegro CL in a large database. They tuned the GC quite a bit for that. But there were also other GC demanding applications on Allegro CL, for example in CAD and 3D design. They are now working on a concurrent GC, something which is still rare in the Lisp world.
- lambdaelite 11y agoWhy are concurrent GCs rare?
- 0xCMP 11y agoI am guessing[1] that GCs are easier to code correctly without the concurrency and that a GC language is already expected to be slower so it doesn't make sense to do a concurrent GC. Also possibly, the language doesn't support concurrency well. Like a Python or Ruby. [1] just an educated guess. I have no real knowledge of GCs other than skimming how they work in articles and runtime/language docs.
- lispm 11y agoMostly because there are very very few people in the world who are able to develop such a complex thing for Lisp (or similar runtimes). Since the market is relatively small and many applications are not overly concurrent, there is very little money to support the development.
- vseloved 11y agoare there many platforms, besides JVM and .Net, that have good-quality concurrent GCs?
- fredyr 11y agoThe Erlang Vm (BEAM), is another one at least.
- the_why_of_y 11y agoIt doesn't actually implement concurrent GC, although what it does implement is far simpler and has a similar effect (low latencies) as concurrent GC. Each Erlang process has a separate heap that is collected independently; because the process heap is usually small a stop-the-process collection does not take much time. The downside is that sending messages between processes requires copying all the data that is sent between process heaps.
- zenogais 11y agoThis was fantastic. Thank you.
- deleted 11y ago[deleted]
- davexunit 11y agoAwesome stuff. Articles like this are what we Lispers/Schemers need to show that our languages can be used for "real work"(tm).
- dfischer 11y agoIs it worthwhile to explore Clojure for web-dev seriously or more as a toy?
- davexunit 11y agoSeriously. I have my misgivings with Clojure in comparison to other Lisps, but it's a serious programming language.
- jodah 11y agoI'm the opposite. The idea that Clojure is just enough Lisp - that it strips down some of the syntax a bit - is one of the benefits, IMO.
- loumf 11y agoIt's extremely serious. The deployment is the very well understood JVM and standard web servers. There's a common HTTP middleware framework (Ring), lots of choices for HTML generation and ClojureScript allows some code-sharing with your client side (compiles to JS). On the back-end, you could use any Java library. On the front-end, you could use any JS framework (e.g. see https://github.com/omcljs/om https://github.com/omcljs/om)
- sgrove 11y agoI suspect swannodette (dnolen)'s talk at EuroClojure should be a pretty compelling showcase of how Clojure is quickly blossoming in the web-dev space.
- reitzensteinm 11y agoIt was a cool talk, but it didn't really address adoption. It featured (to the best of my memory): A new data model for om next. Instead of cursors, components have a composable datomic style DSL to locate their data, which is pluggable and can be from local memory or a cached server side query. Using this approach you can request data in new ways from the client without adding server side code. An update on cljs in cljs (small example but showing lots of plumbing work done) The path to react native, repls and dynamic code loading on the device (was demoed) via ambly
- lkrubner 11y agoGood lord, I would go insane if I ran into a bug like this: "We've built an esoteric application (even by Lisp standards), and in the process have hit some limits of our platform. One unexpected thing was heap exhaustion during compilation. We rely heavily on macros, and some of the largest ones expand into thousands of lines of low-level code. It turned out that SBCL compiler implements a lot of optimizations that allow us to enjoy quite fast generated code, but some of which require exponential time and memory resources. "
- vseloved 11y agoYeah, but that's something you need to be prepared for. Such bugs happen in literally every platform (for instance, I had similar trouble with the JVM). So the question is not how to avoid them, but how to cope. Usually, there are 2 ways: - workaround - investigation (and in this case, if you're on a closed platform you're busted)
- apalmer 11y agoI dont really understand the downvotes... I agree with you would go nuts. My two cents, if your macro is expanding to thousands of lines of code, your doing something wrong I think. I would expect Macros to expand out to a few lines of code which might have function calls that themselves may contain however many lines of code... but expanding to thousands of lines INLINE via macros seems wrong.
- vseloved 11y agoI have pointed to the call-with-* style which is a general "best practice" for that (it's even mentioned in Google CL Style guide). However, expanding to low-level stuff also has it's benefits for a clearly delimited space (mainly, performance) if you know what you're doing
- jeremiep 11y agoSome macros expand to a lot of code, especially when doing more in-depth transformations such as those performed by core.async in Clojure where they transform standard sequential code into a state machine with exactly the same semantic but with the added ability to execute, yield and resume like a coroutine.
- jon-wood 11y agoApparently they use "JVM languages", JavaScript, Python, Go, Lisp and Erlang in production. I may be in the minority, but that would drive me mad. I assume they're not routinely jumping between those stacks multiple times a day, but even so is there really that much benefit that it's worth keeping track of how to do things in that many different environments?
- jessaustin 11y agoSince "proper service encapsulation" is mentioned, it may be that each team uses whatever they like, and as long as your component speaks http you don't have to look at what other components are doing.
- krvss 11y agoThis is exactly how it is (except sometimes it is not HTTP but message queuing etc)
- jon-wood 11y agoThat makes sense, having spent the last five years on a development team that has only recently grown to four people I can forget that not every team needs all their developers to be able to work on any project!
- vseloved 11y agoAs for me, adaptability is one of the important traits of a senior engineer. Surely, you don't have to be an expert in every platform, but you also shouldn't go mad if you need to do some work outside of your comfort zone occasionally. Besides, every language has its strong and weak points: if you're putting arbitrary limits here, you're just limiting what you can do and the people you're going to get in a team. At Grammarly, we always erred on the side of more freedom and it worked not so bad for us so far. Although, there are different companies, each with a unique story...
- davelnewton 11y agoWhile a Single Language to Rule Them All would be cool, I ultimately prefer using the "best" language for the job based on specific requirements. The "best" might change over time, too. It can be a headache to manage massively-polyglot environments. At the same time, it's also pretty great for a variety of reasons. I mean, we regularly use different data stores, messaging solutions, frameworks, etc. and I don't see why languages shouldn't also be up for shuffling.
- avodonosov 11y agoThanks, great post and lot of useful references.
- deleted 11y ago[deleted]
- vseloved 11y agoWell, the HDF5 problem was actually not on the Lisp side ;) But, in general, do you really believe that there are no issues with libraries in other languages? I've had my share in Python or on the JVM, as well. The whole point in the article was to show that there are some challenges, but they didn't become critical to our operation.
- deleted 11y ago[deleted]
- plinkplonk 11y agoAll right, i'm deleting my comment. I don't want to start a Clojure vs CL flame war. That wasn't the intent and I thought I'd made it very clear.
- mrottenkolber 11y agoSo you would pick a different language based on the lack of an existing Jenkins interface library? They would lack 90% of Common Lisp then. Not a good trade if you ask me. Edit: I assume you are down-voted because most of your post is wrong information.
- plinkplonk 11y agoHuh? I said nothing about picking a language only based on a Jenkins interface library. I just noted that every problem they had which they noted in this post seems to arise from the gnarliness of the CL ecosystem, and wouldn't happen with Clojure. Jeez. Where is the "wrong information" in that.
- PuercoPop 11y agoGnarly CL ecosysten? That seems a tad manichean, Clojure certainly has more libraries but CL has its fair share as well. And if you want to tap into the Java ecosystem there is always ABCL. Clojure is a language with diferent design sensibilites than CL.
- Grue3 11y agoCommon Lisp's macros and grammar go together like bread and butter. A grammar module in the app I built [1] uses macros to generate huge amounts of repetitive code. [1] https://github.com/tshatrov/ichiran/blob/master/dict-grammar.lisp https://github.com/tshatrov/ichiran/blob/master/dict-grammar... I wonder if they're still hiring Lispers. I once passed on the opportunity to work in their Kiev office, but I might give it a shot again.
- bliti 11y agoFound my reading material for tonight. Thank you.
- muraiki 11y agoWow, I checked out your ichi.moe app. It's awesome! I'm definitely going to use this as I work on re-learning the Japanese I've forgotten over the years. I've done some Racket and Clojure but never done CL... I'll have to check it out.
- brobinson 11y agoThanks for pointing this out. I'm learning Japanese now and this will be massively helpful.
- f00biebletch 11y agoThey are hiring - contact kevin dot mcintire at grammarly dot com
- welshguy 11y agoI love it :0) I tried grammarly, and typed in a remembered poem. It informed me that it had detected significant plagiarism. Edit: It's still not advice I would pay for, though.
- PuercoPop 11y agoOne of the things I would have liked to see on the article is how do they handle the deployment itself. Do they build an executable with build app? To they used sb-daemon? An home-grown solution using sb-posix:fork?
- lisper 11y agoI don't know how they do it, but I have run several Lisp production systems. I use CCL, which has a really fast compiler. So I just load the code from source. The deployment process then becomes: git pull, quit and restart Lisp. (The ccl-init file loads the code.)
- PuercoPop 11y agoAnd in that case how do you handle the monitoring of the system?
- lisper 11y agoI don't understand the question. Once it's running, it's like any other server, and you monitor it like you would any other server written in any other language.
- WalterGR 11y agoSpeculation here, but based on the context, I think the question is: Some daemon-izer solutions will monitor the daemon and e.g. restart it if it's unresponsive. How do you handle this slash how is this handled in the Common Lisp world?
- lisper 11y agoAs a last resort, if Lisp becomes totally unresponsive, you kill the process and restart, same as any other language. But it's pretty rare to lose the REPL, so usually you can fix any unexpected problems through that.
- 11y ago
- akssri 11y ago> but we value choice and freedom over rules and processes. Which I exactly why I feel Lisp doesn't see much use elsewhere :(
- outworlder 11y agoAw, now they have disclosed their secret weapon! [1] [http://www.paulgraham.com/avg.html http://www.paulgraham.com/avg.html]
- orthecreedence 11y agoGreat article, and good reminder on using trace. Every time I rediscover trace, I can't remember how I ever forgot to use it in the first place for most of my problems. I used CL in a production environment a while back for a threaded queue worker and nowadays as the app server for my turtl project, and I still have yet to run into problems. It seems like you guys managed to push the boundaries and find workable solutions, which is really great. Thanks for the writeup!
- eruditely 11y agoWhy not racket?
- dkvasnicka 11y agoI was thinking the same question and the reply would probably be "performance". I don't think untyped Racket can currently compete with SBCL and I'm not sure they'd be interested in Typed Racket. But I'd love to be proven wrong on both of those things ;)
- copperx 11y agoSeconded. Why not racket?
- Queue29 11y agoWhy racket?
- white-flame 11y agoWe deploy distributed, multi-language, centrally Lisp/SBCL servers as well. A few specifics that I'd point out: Many of SBCL's optimizations are fine grained selectable, using internal SB-* declarations. I know I was at least able to turn off all optimizations for debug/disasm clarity, while specifically enabling tail recursion so that our main loop wouldn't blow up the stack in that build configuration. These aren't in the main documentation; I asked in the #sbcl IRC channel on FreeNode. You can directly set the size of the nursery with sb-ext:bytes-consed-between-gcs, as opposed to overprovisioning the heap to influence the nursery size. While we've run in the 8-24GB heap ranges depending on deployment, a minimum nursery size of 1GB seems to give us the best performance as well. We're looking at much larger heap sizes now, so who knows what will work best. While we haven't hit heap exhaustion conditions during compilation, we did hit multi-minute compilation lags for large macros (18,000 LoC from a first-level expansion). That was a reported performance bug in SBCL and has been fixed a while back. Since the Debian upstream for SBCL lags the official releases quite a bit, it's always a manual job to fetch the latest versions, but quite worth it. Great read, and really familiar. :-)
- vseloved 11y agoWe played with sb-ext:bytes-consed-between-gcs, but couldn't find the right balance. That's why we were surprised with the result of the oversized heap experiment
- white-flame 11y agoYou might want to do some googling about it, but I also seem to remember that SBCL might decide to use large or small MMU page granularity depending on the size of the heap. That might be the watershed trigger for performance. (or just some random mis-remembered nonsense)
- vseloved 11y agoCan you also point to the particular version of SBCL that fixed for long-running compilation? We have recently upgraded to one of the latest version, but I think I've missed this change - I'm interested to check it in more detail.
- mud_dauber 11y agoWow. I can't ever remember reading about a consumer-facing app using Common Lisp. Ever.
- wtbob 11y ago> I can't ever remember reading about a consumer-facing app using Common Lisp. There was ViaWeb: http://www.paulgraham.com/avg.html http://www.paulgraham.com/avg.html I'm told that Orbitz uses or used it a lot too.
- jon-wood 11y agoYou're using one ;)
- cpach 11y agoNope, HN is written in Arc (which runs on top of Racket) :-p
- jjawssd 11y agoOnce you go deep enough you are fucked no matter what language you choose. Might as well pick one that doesn't beat you up too much.
- doomrobo 11y agoSlightly off-topic, but does anybody know of a kind of "LISP Challange" set? I recently started the Matasano challenges[0] and I found them really well-suited to my style of learning (learning by doing and expanding by reading relevant material, enabled by my own internal motivation). Is there anything like that that has a relatively small set of condensed yet rich challenges that demonstrate key elements from LISPy functional programming? I read some of SICP but reading long form really puts a damper on my motivation/excitement. Also there were a lot of exercises (with a lot of overlap in concepts) so I didn't know what to do and what not to do, since I wasn't about to do every single one. Any pointers would be much appreciated! [0] http://cryptopals.com http://cryptopals.com
- digaozao 11y agoHi, I don't know if it fits what you want. But there is something like that for clojure[0] . I liked the language a lot. I didn't have the chance to use it in production yet. Another option is hacker rank with challenges [1] [0]https://www.4clojure.com/ https://www.4clojure.com/ [1]https://www.hackerrank.com/ https://www.hackerrank.com/
- jarcane 11y agohttp://exercism.io http://exercism.io supports Common Lisp, Scheme, Clojure, and Emacs Lisp, plus a ton of others, and runs through a neat little CLI app locally, so you can use your own editor (I've been doing Rust challenges in Emacs, frex.) http://www.codewars.com/ http://www.codewars.com/ also supports Clojure, and Haskell, which is not a Lisp but is FP. Hacker Rank pretty much supports everything, but the reason for this is that it handles all the tests through stdio instead of a test suite, resulting in a lot of irritating boilerplate code.
- dkvasnicka 11y agoWell, that stdio thing is actually what I love about Hacker Rank :) I don't think the boilerplate code is a problem, if you use a reasonable language. Racket is a pleasure to use over there, Clojure slightly less so but wouldn't call it irritating... But yeah, I've seen some Java solutions on HR...