14 ms·

The Common Lisp Condition System – Upcoming Book

183 points by phoe-krk 6y ago

phoe-krk 6y ago
Author here. The book has just gone to production and will be available at the end of the year in dead-tree and electronic formats. The book happened because I got annoyed at the fact that there is no good reference that shows how the condition system works, what are the foundations required for it to work, what are the basic building blocks that it is composed of, and what are its uses, both in the domain of handling exceptional situations and, more importantly and less obviously, outside that domain. The book also contains a description of a full implementation of a condition system that can (and hopefully will!) be used outside Common Lisp. There's hope that some languages will pick up these parts of Common Lisp that fully decouple the acts and means of signaling conditions from the acts and means of handling them, therefore allowing for means of dynamically controlled control flow that are currently very hard to achieve in other languages. The accompanying code is available at https://github.com/phoe/tclcs-code https://github.com/phoe/tclcs-code and https://github.com/phoe/portable-condition-system https://github.com/phoe/portable-condition-system - it will be copied over to the Apress repository soon. Linked Reddit thread: https://www.reddit.com/r/lisp/comments/hrjzs8/ https://www.reddit.com/r/lisp/comments/hrjzs8/ AMA, I guess.
- jshaqaw 6y ago
  The book looks really interesting. Congrats!
  - phoe-krk 6y ago
    Thanks. Hope that it indeed is interesting, and that it doesn't contain sizeable errors anymore. The Lisp community has been a great help when it came to reviewing and editing the hell out of the manuscript, both language- and content-wise.
- fovc 6y ago
  Looks cool! Is there a TOC and/or sample section available? Couldn't see one through the site but would like to get a sense of contents and style before buying
  - phoe-krk 6y ago
    Samples should be available later; the book has only gone to production several days ago, so there's nothing properly formatted for a sample section yet. As for ToC, hold on, give me a few minutes - I'll extract it for you from the manuscript. EDIT: ToC is here, https://plaster.tymoon.eu/view/1956 https://plaster.tymoon.eu/view/1956
- Archit3ch 6y ago
  Is this a book for Common Lisp beginners? The description claims so, but dynamic control flow doesn't seem like a beginner topic.
  - phoe-krk 6y ago
    I'll daringly say that it is. The book explicitly contains a tutorial section that describes dynamic variables, non-local transfers of control, and lexical closures - the three building blocks that allow for construction of a condition system. The part that comes afterwards constructs the condition system piece by piece, function after function - and only after we've constructed something from scratch, we draw parallels to relevant parts of the condition system. There's also a short tutorial for macro writing as an appendix, since we will write some macros as a part of the condition system. Overall, this book should be able to introduce a programming into using the Common Lisp condition system. For the very basic basics, I will still want the reader to start with Practical Common Lisp, but once the reader knows the basics of syntax, evaluation rules, and basic data types in Lisp, they should be able to follow the book from the beginning.
    - Akronymus 6y ago
      Where would you put it in relation to htdp and sicp? I personally am working my way through htdp at this time, and plan on doing sicp afterwards, and would be very interested in maybe slotting your book in there somewhere.
      - phoe-krk 6y ago
        I'd put TCLCS after SICP and HTDP; these are introductory books to programming in general and therefore are prerequisites to working with a condition system. Or, in other words, one shouldn't read a book about handling exceptional situations if one doesn't understand what are exceptional situations or why one would need to handle them.
        Akronymus 6y ago
- aidenn0 6y ago
  > The book also contains a description of a full implementation of a condition system that can (and hopefully will!) be used outside Common Lisp Do you foresee any issues with other languages only having dynamic (i.e. THROW) non-local transfers of control rather than lexical as well (i.e. GO). I find it convenient for when I want to conditionally unwind the stack to have lexical transfers, and obviously it's been demonstrated you can implement GO with just THROW, so there's no loss of generality, but I wonder what the ergonomic impact will be.
  - phoe-krk 6y ago
    None, other than for sheer performance. (THROW will perform worse than GO and RETURN since the list of catch tag will need to be traversed on each throw.) As you mentioned, one can implement both GO and RETURN via THROW plus gensyms - see https://web.archive.org/web/20191008050814/http://home.pipeline.com/~hbaker1/MetaCircular.html https://web.archive.org/web/20191008050814/http://home.pipel... for an implementation.
- eismcc 6y ago
  While this may seem absurdly naive, what is the CL condition system? Based on what I see (read) It seems like you need to know what it is before read the book.
  - phoe-krk 6y ago
    Let me answer by posting an introduction to the condition system by Kent M. Pitman. It is the first subchapter of the book. ------ There have been many attempts to declare the Lisp family of languages dead, and yet it continues on in many forms. There are many explanations for this, but an obvious one is that it still contains ideas and features that aren't fully appreciated outside the Lisp community, and so it continues as both a refuge and an idea factory. Gradually, other languages see the light and these important features migrate to other languages. For example, the Lisp community used to be unusual for standing steadfastly by automatic memory management and garbage collection when many said it couldn't be trusted to be efficient or responsive. In the modern world, however, many languages now presume that automatic memory management is normal and natural, as if this had never been a controversy. So times change. But proper condition handling is something which other languages still have not figured out that they need. Java's try/catch and Python's try/except have indeed shown that these language appreciate the importance of representing exceptional situations as objects. However, in adopting these concepts, they have left out restarts --- a key piece of the puzzle. When you raise an exception in Python, or throw one in Java, you are still just performing an immediate and blind transfer of control to the innermost available handler. This leaves out the rich experience that Common Lisp offers to perform actual reasoning about where to return to. The Common Lisp condition system disconnects the ability to return to a particular place in the program from the necessity to do so, and adds the ability to "look before you leap." In other languages, if you create a possible place to return to, that is what will get used. There is no ability to say "If a certain kind of error happens, this might be a good place to return to, but I don't have a strong opinion ahead of time on whether or not it is definitely the right place." The Common Lisp condition system separates out three different activities: describing a problem, describing a possible solution, and selecting the right solution for the right problem. In other languages, describing a possible solution is the same as selecting that solution, so the set of things you can describe is necessarily less expansive. This matters, because in other languages such as Python or Java, by the time your program first notices a problem, it already will have "recovered" from it. The "except" or "catch" part of your "try" statement will have received control. There will have been no intervening time. To invoke the error handling process IS to transfer control. By the time any further application code is running, a stack unwind already will have happened. The dynamic context of the problem will be gone, and with it, any potential intervening options to resume operation at other points on the stack between the raising of the condition and the handling of an error. Any such opportunities to resume operation will have lost their chance to exist. "Well, too bad", these languages would say. "If they wanted a chance, they could have handled the error." But the thing is, a lot of the business of signaling and handling conditions is about the fact that you only have partial knowledge. The more uncertain information you are forced to supply, the more your system will make bad decisions. For best results, you want to be able to defer decisions until all information is available. Simple-minded exception systems are great if you know exactly how you want to handle things ahead of time. But if you don't know, then what are you to do? Common Lisp provides much better mechanisms for navigating this uncertain space than other languages do. So in Common Lisp you can say "I got an argument of the wrong type. Moreover, I know what I would do with an argument of the right type, I just don't happen to have one or know how to make one." Or you can say "Not only do I know what to do if I'm given an argument of the right type (even at runtime), but I even know how to store such a value so they won't hit this error over and over again." In other languages, if the program doesn't know this correctly-typed value, even if you (the user) do know it at runtime, you're simply stuck. In Common Lisp, you can specify the restart mechanism separately from the mechanism of choosing among possible restarts. Having this ability means that an outer part of the program can make the choice, or the choice can fall through to a human user to make. Of course, the human user might get tired of answering, but in such a case, they can wrap the program with advice that will save them from the need to answer. This is a much more flexible division of responsibility than other languages offer.
- foobar_ 6y ago
  Congrats on the book! If you are just going to use `TAGBODY` and `GO` to implement this ... in a goto based languages like C I don't think you would be needing a condition system. In C a nested function can also jump into a parents goto label. No one probably uses it, but just saying! Also how similar is this to promises in javascript ?
  - phoe-krk 6y ago
    C's longjmp doesn't cleanly unwind the stack. Common Lisp's GO, just like all CL control flow operators, cleanly unwind the stack, allowing cleanup forms established by UNWIND-PROTECT to be executed. That's the key difference. The book contains an implementation of dynamic variables in C, though, which uses a GCC cleanup extension; the example code contains the contributed code examples for that showing various means of implementing dynavars in C. Promises are asynchronous while signaling and condition handling is fully synchronous; I don't know what kind of parallel one can draw here. If anything, a promise's .catch(...) method may act like a CL HANDLER-CASE; the promise becomes resolved, just a different code block is executed in case of a failure.
    - foobar_ 6y ago
      `UNWIND-PROTECT` thats cool! What I meant with the promises was, If you could pass three closures ... like success, error, restart could you get some kinda condition system ?
      - phoe-krk 6y ago
        That's not enough. The condition system decouples the act of signaling a condition from the act of deciding how to handle it. This means that the promise would need to reach out to its dynamic environment to figure out what it would need to do in case of errors and restarts. And that also only handles the case of "help I'm ded get me out of here". What about signals that do not expect to be handled, and instead are used to transfer information higher up the stack by invoking a handler function specified in the dynamic environment? That's a valid use of the condition system, as outlined in the book.
- nahuel0x 6y ago
  It will be very nice if you can show a big list of conditions use-cases, hypothetical and real world, besides the implementation details. That will help to popularize the concept for non-CL audiences.
  - phoe-krk 6y ago
    The main part of the book outlines multiple ways of dealing with exceptional situations, including the handler and restart subsystems of the condition system. It also includes a section with some use cases of the condition system outside standard error handling. I hope that the information stored there is, all in all, rich enough to satisfy your curiosity, and the curiosity about concrete use cases for the condition system in general.
- minerjoe 6y ago
  Wonderful topic for a book. I don't see a price for the electronic version? Will it be free such as Practical Common Lisp?
  - phoe-krk 6y ago
    Nope, it won't be free as in beer. The ebook will be purchasable when the dead-tree book arrives, but I don't yet know the price for it.
    - minerjoe 6y ago
      Curious to your logic in determining that? As far as I understand it, the freely available digital version of PCL aided in the selling of the dead-tree book?
      - phoe-krk 6y ago
        The online version of Practical Common Lisp was published fifteen years ago, well before the ebook market was as developed as it is now. Given the current market situation, I've made the decision to go the same way that Common Lisp Recipes went - selling ebooks + dead-tree books.
- graememcc 6y ago
  I'm a sucker for anything that takes the "let's learn it by building it" approach, so will definitely pick this up. Looking forward to it!
  - phoe-krk 6y ago
    Glad to hear that! The book starts off with a tutorial about dynamic variables, transfers of control, and lexical closures, and then uses these (along with a few functions and variables) to build a condition system from scratch.
- phoe-krk 6y ago
  EDIT: These were several wild hours of an AMA - thank you, everyone, who participated in the discussion. Feel free to post more comments, I'll reply to them in due time. Or just poke me via mail or on Freenode.
- mark_l_watson 6y ago
  I just tried to buy it, but it the order form wasn't active, so thanks for the timeline update. I look forward to reading your book when it is released! I have been using Common Lisp since about 1982 but to be honest, I have a lot to learn about its condition system.
  - phoe-krk 6y ago
    I just learned that preorders are available on https://www.amazon.com/Common-Lisp-Condition-System-Mechanisms/dp/148426133X https://www.amazon.com/Common-Lisp-Condition-System-Mechanis... - if preorders are your thing, that is.
- dragonwriter 6y ago
  > The book happened because I got annoyed at the fact that there is no good reference that shows how the condition system works, what are the foundations required for it to work, what are the basic building blocks that it is composed of, and what are its uses, both in the domain of handling exceptional situations and, more importantly and less obviously, outside that domain. Except lacking deep more coverage applications, I think Practical Common Lisp covers all of that. Which isn't a vote against a book focussed on the subject, though.
  - phoe-krk 6y ago
    As mentioned in one of the comments, PCL does good overall coverage of the condition system but does not go in depth enough for my taste; it is a book for teaching CL, not the condition system, after all. It does not describe any applications or use cases of the condition system that do not strictly serve exception handling, and it does not show the internal construction of the condition system, especially from the approach of starting with nothing but most basic language tools which are dynamic variables, means of transferring control, closures, functions, classes, and variables.
drcode 6y ago
For those unaware, Common Lisp exception handling is heads and shoulders above almost any other programming language: Imagine having your program crash in the middle of a function, being able to inspect the call stack to find the bug, fixing it, then completing the function call from the middle of the function, at the exact point where the exception happened, with the fix in place.
- tester34 6y ago
  During debugging, yea?
  - phoe-krk 6y ago
    Yes, all that without leaving the debugger. The debugger is left only when we want to continue execution of the program.
    - protomyth 6y ago
      I've noticed quite a few Smalltalk programmers who actually coded in the debugger. They would put a call in the program, hit it as an error, and then write new code. Does this happen with Common Lisp programmers?
      - phoe-krk 6y ago
        Smalltalk and Common Lisp are both image-based programming languages where the compiler and debugger are always available. I think debugger-oriented programming is much more common in Smalltalk than in Lisp (at least I prefer to leave the debugger back to REPL pretty often), but I think it's certainly possible to adopt. The Slime/Sly environments for emacs might accept a few patches to facilitate that programming style better.
      - p_l 6y ago
        I know of several lispers that swear by top-down "programming inside debugger", where they continuously fill in the gaps starting from top level function :-D
steveklabnik 6y ago
Fun trivia fact: Rust had conditions at one point. They weren't really used by anyone and so ended up getting dropped.
- phoe-krk 6y ago
  The book mentions that fact. The commit that removed them (along with the rationale) is available at https://github.com/rust-lang/rust/commit/454882dcb7fdb03867d695a88335e2d2c8f7561a https://github.com/rust-lang/rust/commit/454882dcb7fdb03867d...
  - gumby 6y ago
    Those reasons seem more rust-specific than being a general reason to avoid restartable conditions — as the commit message acknowledges at the end.
    - steveklabnik 6y ago
      They're also "2014 Rust" specific; Rust changed a bunch between then and 1.0. Which doesn't mean they still wouldn't apply, just it gets harder for anything earlier than may 2015.
gumby 6y ago
The condition system is often hard for non-lisp people to understand as the best they’ve had in the past is a core dump or a c++-style exception system. The main problem for me with those is that by the time you see the exception the state has already been destroyed. This is the problem with core dumps as well: all your files and network connections have been closed. The only alternative is to use a debugger which is a high overhead situation. The roots of condition system evolved from two related hacker environments: the ITS system at the MIT AI lab and the lisp machines (both CADR and its descendants and the PARC D-machines) as well as influence for the Smalltalk folks. In ITS what is called the “shell” these days was the debugger. If your program signaled an error you were immediately in the debugger and could debug, just exit, or Force a core dump. This was the routine way to interact, even for non-programmers. The lispm did the same. Debugging was a lightweight operation. So it was natural in the 80s to develop a system that formalized this mode of operation. Because an exceptional situation by definition requires greater knowledge than is available in situ. When you catch an error in java or can++ it’s too late to look at the state of the object that caused the error without standing on your head and possibly redesigning the try block say to allocate something on the heap that can be passed to the error (which means no more RAII etc etc). In addition, try/catch typically puts the hand lin too close to the site of the problem for you to be useful; if not, by the time you get the exception there’s not much you can do but terminate the calculation politely.
- phoe-krk 6y ago
  > The main problem for me with those is that by the time you see the exception the state has already been destroyed. This is the problem with core dumps as well: all your files and network connections have been closed. The only alternative is to use a debugger which is a high overhead situation. This seems like two core issues with understanding the condition system. First, the currently popular exception systems always destroy the stack by unwinding it, allowing the programmer to debug only dead remains instead of a living program; second, debuggers are bolted on top of programs instead of being their internal parts, which gives many people an impression that using them as an integral part of the whole programming lifecycle is inconvenient and unnatural. The condition system in Common Lisp solves both of these issues, which IMO, paradoxically, contributes to it not being well understood outside Lisp circles.
daly 6y ago
I tried to buy (pre-order) the book. I created an account on the Apress site as described in their help section. There is no link to buy the book. Sigh.
- phoe-krk 6y ago
  It is an upcoming book, as the title describes. It will be available later in the year.
  - MycroftJones 6y ago
    Amazon generally has a "pre-order" option. If there was a pre-order option, I also would order your book. By the time your book is in print, I will have forgotten about it. Fascinating topic; nice to see good coverage. I always felt that callbacks and exception handling left a little something to be desired. It looks like you've addressed that. When your book is in print and available for sale I hope you post again here as a reminder.
    - phoe-krk 6y ago
      The pre-order option is there! https://www.amazon.com/Common-Lisp-Condition-System-Mechanisms/dp/148426133X https://www.amazon.com/Common-Lisp-Condition-System-Mechanis... Thank you for the kind words; I hope that the book suits you well once it's out.
- phoe-krk 6y ago
  I just learned that preorders are available on https://www.amazon.com/Common-Lisp-Condition-System-Mechanisms/dp/148426133X https://www.amazon.com/Common-Lisp-Condition-System-Mechanis... - if preorders are your thing, that is.
mjbishop 6y ago
This book looks great! I tried to implement this in Ruby with a gem about 6 years ago and got really close. If anyone wants to know more about this, but is more fluent in Ruby, you might find this a good jumping-off point. There’s a good explanation in the README. https://github.com/michaeljbishop/mulligan Why did it not work in the end? Because even though I was able to implement an error-recovery mechanism that wouldn’t pop the stack, it still triggered all the `ensure` statements that were up the stack. It was still a valuable exercise. I'd welcome this addition in any exception-based language.
- phoe-krk 6y ago
  Thanks for that link! I'll include it in the book.
dan-robertson 6y ago
A problem I don’t have a good answer to is how to extend conditions (or even exceptions) to async programming models. Two problems are: 1. some code may fail multiple times (with non-async code, once an exception is caught the program can’t throw another one from the same dynamic extent of the try block; similarly once a restart is invoked and control is transferred, a condition won’t be signalled from the same dynamic extent where the restart was bound (for the purpose of this clause, we consider a retry restart to mint a new dynamic extent to bind the restart in again after retrying)) 2. Some computation may happen after you don’t care about its result because a parallel dependency has failed. E.g. suppose you have some rate limiter where you pass an async function, get a promise for it’s result, and the rate limited will queue your function to be called soon but with the condition that it will only allow say 5 calls to be in flight. Now you have some resource-downloading function which puts many separate download requests into the ratelimiter in parallel, and combines the results in such a way that the whole download fails if any request fails. But now if an early request fails, your function will continue trying to download all the other resources even though it doesn’t care about the result now that it has failed. You could imagine having a way to cancel promises and maybe you would be clever and write really solid programs and never have issues but to me this seems to be inviting very difficult to debug problems (just like expecting any thread to be killed at any random point (or at least a bunch of different points) would make correct multithreaded programming harder. Perhaps a more succinct description of problem 2 is that unwinding the “stack” isn’t well-defined with an async programming model because “stack” doesn’t really mean the call stack from the scheduler you whatever bit of async computation you happen to be doing, but rather the combined dynamic scope from everything that started the computation and is currently awaiting its completion. A related and, I think, very useful thing to have is a way to represent computations which may signal conditions or be restarted. I think CL doesn’t have a great story for this (threading isn’t really a part of the language though). An example use case would be some program which has to process some stream of events. If it gets stuck on one or some, you may want to continue processing the rest of the events while deciding what restart to invoke for the problematic ones. Perhaps an effect system could help with this.
- phoe-krk 6y ago
  I'll be frank: my book does not touch this topic in the slightest, and this is an open question. Common Lisp does not have any asynchronous features built in and, as I've said in some earlier comment, the condition system is fully synchronous. This topic is still waiting to be explored, either with an extension of Common Lisp or a completely different language that permits asynchronous programming with an adaptation of the condition system to this programming paradigm.

183 points by phoe-krk 6y ago

Author here. The book has just gone to production and will be available at the end of the year in dead-tree and electronic formats. The book happened because I got annoyed at the fact that there is no good reference that shows how the condition system works, what are the foundations required for it to work, what are the basic building blocks that it is composed of, and what are its uses, both in the domain of handling exceptional situations and, more importantly and less obviously, outside that domain. The book also contains a description of a full implementation of a condition system that can (and hopefully will!) be used outside Common Lisp. There's hope that some languages will pick up these parts of Common Lisp that fully decouple the acts and means of signaling conditions from the acts and means of handling them, therefore allowing for means of dynamically controlled control flow that are currently very hard to achieve in other languages. The accompanying code is available at https://github.com/phoe/tclcs-code https://github.com/phoe/tclcs-code and https://github.com/phoe/portable-condition-system https://github.com/phoe/portable-condition-system - it will be copied over to the Apress repository soon. Linked Reddit thread: https://www.reddit.com/r/lisp/comments/hrjzs8/ https://www.reddit.com/r/lisp/comments/hrjzs8/ AMA, I guess.

jshaqaw 6y ago
The book looks really interesting. Congrats!
- phoe-krk 6y ago
  Thanks. Hope that it indeed is interesting, and that it doesn't contain sizeable errors anymore. The Lisp community has been a great help when it came to reviewing and editing the hell out of the manuscript, both language- and content-wise.
fovc 6y ago
Looks cool! Is there a TOC and/or sample section available? Couldn't see one through the site but would like to get a sense of contents and style before buying
- phoe-krk 6y ago
  Samples should be available later; the book has only gone to production several days ago, so there's nothing properly formatted for a sample section yet. As for ToC, hold on, give me a few minutes - I'll extract it for you from the manuscript. EDIT: ToC is here, https://plaster.tymoon.eu/view/1956 https://plaster.tymoon.eu/view/1956
Archit3ch 6y ago
Is this a book for Common Lisp beginners? The description claims so, but dynamic control flow doesn't seem like a beginner topic.
- phoe-krk 6y ago
  I'll daringly say that it is. The book explicitly contains a tutorial section that describes dynamic variables, non-local transfers of control, and lexical closures - the three building blocks that allow for construction of a condition system. The part that comes afterwards constructs the condition system piece by piece, function after function - and only after we've constructed something from scratch, we draw parallels to relevant parts of the condition system. There's also a short tutorial for macro writing as an appendix, since we will write some macros as a part of the condition system. Overall, this book should be able to introduce a programming into using the Common Lisp condition system. For the very basic basics, I will still want the reader to start with Practical Common Lisp, but once the reader knows the basics of syntax, evaluation rules, and basic data types in Lisp, they should be able to follow the book from the beginning.
  - Akronymus 6y ago
    Where would you put it in relation to htdp and sicp? I personally am working my way through htdp at this time, and plan on doing sicp afterwards, and would be very interested in maybe slotting your book in there somewhere.
    - phoe-krk 6y ago
      I'd put TCLCS after SICP and HTDP; these are introductory books to programming in general and therefore are prerequisites to working with a condition system. Or, in other words, one shouldn't read a book about handling exceptional situations if one doesn't understand what are exceptional situations or why one would need to handle them.
      - Akronymus 6y ago
aidenn0 6y ago
> The book also contains a description of a full implementation of a condition system that can (and hopefully will!) be used outside Common Lisp Do you foresee any issues with other languages only having dynamic (i.e. THROW) non-local transfers of control rather than lexical as well (i.e. GO). I find it convenient for when I want to conditionally unwind the stack to have lexical transfers, and obviously it's been demonstrated you can implement GO with just THROW, so there's no loss of generality, but I wonder what the ergonomic impact will be.
- phoe-krk 6y ago
  None, other than for sheer performance. (THROW will perform worse than GO and RETURN since the list of catch tag will need to be traversed on each throw.) As you mentioned, one can implement both GO and RETURN via THROW plus gensyms - see https://web.archive.org/web/20191008050814/http://home.pipeline.com/~hbaker1/MetaCircular.html https://web.archive.org/web/20191008050814/http://home.pipel... for an implementation.
eismcc 6y ago
While this may seem absurdly naive, what is the CL condition system? Based on what I see (read) It seems like you need to know what it is before read the book.
- phoe-krk 6y ago
  Let me answer by posting an introduction to the condition system by Kent M. Pitman. It is the first subchapter of the book. ------ There have been many attempts to declare the Lisp family of languages dead, and yet it continues on in many forms. There are many explanations for this, but an obvious one is that it still contains ideas and features that aren't fully appreciated outside the Lisp community, and so it continues as both a refuge and an idea factory. Gradually, other languages see the light and these important features migrate to other languages. For example, the Lisp community used to be unusual for standing steadfastly by automatic memory management and garbage collection when many said it couldn't be trusted to be efficient or responsive. In the modern world, however, many languages now presume that automatic memory management is normal and natural, as if this had never been a controversy. So times change. But proper condition handling is something which other languages still have not figured out that they need. Java's try/catch and Python's try/except have indeed shown that these language appreciate the importance of representing exceptional situations as objects. However, in adopting these concepts, they have left out restarts --- a key piece of the puzzle. When you raise an exception in Python, or throw one in Java, you are still just performing an immediate and blind transfer of control to the innermost available handler. This leaves out the rich experience that Common Lisp offers to perform actual reasoning about where to return to. The Common Lisp condition system disconnects the ability to return to a particular place in the program from the necessity to do so, and adds the ability to "look before you leap." In other languages, if you create a possible place to return to, that is what will get used. There is no ability to say "If a certain kind of error happens, this might be a good place to return to, but I don't have a strong opinion ahead of time on whether or not it is definitely the right place." The Common Lisp condition system separates out three different activities: describing a problem, describing a possible solution, and selecting the right solution for the right problem. In other languages, describing a possible solution is the same as selecting that solution, so the set of things you can describe is necessarily less expansive. This matters, because in other languages such as Python or Java, by the time your program first notices a problem, it already will have "recovered" from it. The "except" or "catch" part of your "try" statement will have received control. There will have been no intervening time. To invoke the error handling process IS to transfer control. By the time any further application code is running, a stack unwind already will have happened. The dynamic context of the problem will be gone, and with it, any potential intervening options to resume operation at other points on the stack between the raising of the condition and the handling of an error. Any such opportunities to resume operation will have lost their chance to exist. "Well, too bad", these languages would say. "If they wanted a chance, they could have handled the error." But the thing is, a lot of the business of signaling and handling conditions is about the fact that you only have partial knowledge. The more uncertain information you are forced to supply, the more your system will make bad decisions. For best results, you want to be able to defer decisions until all information is available. Simple-minded exception systems are great if you know exactly how you want to handle things ahead of time. But if you don't know, then what are you to do? Common Lisp provides much better mechanisms for navigating this uncertain space than other languages do. So in Common Lisp you can say "I got an argument of the wrong type. Moreover, I know what I would do with an argument of the right type, I just don't happen to have one or know how to make one." Or you can say "Not only do I know what to do if I'm given an argument of the right type (even at runtime), but I even know how to store such a value so they won't hit this error over and over again." In other languages, if the program doesn't know this correctly-typed value, even if you (the user) do know it at runtime, you're simply stuck. In Common Lisp, you can specify the restart mechanism separately from the mechanism of choosing among possible restarts. Having this ability means that an outer part of the program can make the choice, or the choice can fall through to a human user to make. Of course, the human user might get tired of answering, but in such a case, they can wrap the program with advice that will save them from the need to answer. This is a much more flexible division of responsibility than other languages offer.
foobar_ 6y ago
Congrats on the book! If you are just going to use `TAGBODY` and `GO` to implement this ... in a goto based languages like C I don't think you would be needing a condition system. In C a nested function can also jump into a parents goto label. No one probably uses it, but just saying! Also how similar is this to promises in javascript ?
- phoe-krk 6y ago
  C's longjmp doesn't cleanly unwind the stack. Common Lisp's GO, just like all CL control flow operators, cleanly unwind the stack, allowing cleanup forms established by UNWIND-PROTECT to be executed. That's the key difference. The book contains an implementation of dynamic variables in C, though, which uses a GCC cleanup extension; the example code contains the contributed code examples for that showing various means of implementing dynavars in C. Promises are asynchronous while signaling and condition handling is fully synchronous; I don't know what kind of parallel one can draw here. If anything, a promise's .catch(...) method may act like a CL HANDLER-CASE; the promise becomes resolved, just a different code block is executed in case of a failure.
  - foobar_ 6y ago
    `UNWIND-PROTECT` thats cool! What I meant with the promises was, If you could pass three closures ... like success, error, restart could you get some kinda condition system ?
    - phoe-krk 6y ago
      That's not enough. The condition system decouples the act of signaling a condition from the act of deciding how to handle it. This means that the promise would need to reach out to its dynamic environment to figure out what it would need to do in case of errors and restarts. And that also only handles the case of "help I'm ded get me out of here". What about signals that do not expect to be handled, and instead are used to transfer information higher up the stack by invoking a handler function specified in the dynamic environment? That's a valid use of the condition system, as outlined in the book.
nahuel0x 6y ago
It will be very nice if you can show a big list of conditions use-cases, hypothetical and real world, besides the implementation details. That will help to popularize the concept for non-CL audiences.
- phoe-krk 6y ago
  The main part of the book outlines multiple ways of dealing with exceptional situations, including the handler and restart subsystems of the condition system. It also includes a section with some use cases of the condition system outside standard error handling. I hope that the information stored there is, all in all, rich enough to satisfy your curiosity, and the curiosity about concrete use cases for the condition system in general.
minerjoe 6y ago
Wonderful topic for a book. I don't see a price for the electronic version? Will it be free such as Practical Common Lisp?
- phoe-krk 6y ago
  Nope, it won't be free as in beer. The ebook will be purchasable when the dead-tree book arrives, but I don't yet know the price for it.
  - minerjoe 6y ago
    Curious to your logic in determining that? As far as I understand it, the freely available digital version of PCL aided in the selling of the dead-tree book?
    - phoe-krk 6y ago
      The online version of Practical Common Lisp was published fifteen years ago, well before the ebook market was as developed as it is now. Given the current market situation, I've made the decision to go the same way that Common Lisp Recipes went - selling ebooks + dead-tree books.
graememcc 6y ago
I'm a sucker for anything that takes the "let's learn it by building it" approach, so will definitely pick this up. Looking forward to it!
- phoe-krk 6y ago
  Glad to hear that! The book starts off with a tutorial about dynamic variables, transfers of control, and lexical closures, and then uses these (along with a few functions and variables) to build a condition system from scratch.
phoe-krk 6y ago
EDIT: These were several wild hours of an AMA - thank you, everyone, who participated in the discussion. Feel free to post more comments, I'll reply to them in due time. Or just poke me via mail or on Freenode.
mark_l_watson 6y ago
I just tried to buy it, but it the order form wasn't active, so thanks for the timeline update. I look forward to reading your book when it is released! I have been using Common Lisp since about 1982 but to be honest, I have a lot to learn about its condition system.
- phoe-krk 6y ago
  I just learned that preorders are available on https://www.amazon.com/Common-Lisp-Condition-System-Mechanisms/dp/148426133X https://www.amazon.com/Common-Lisp-Condition-System-Mechanis... - if preorders are your thing, that is.
dragonwriter 6y ago
> The book happened because I got annoyed at the fact that there is no good reference that shows how the condition system works, what are the foundations required for it to work, what are the basic building blocks that it is composed of, and what are its uses, both in the domain of handling exceptional situations and, more importantly and less obviously, outside that domain. Except lacking deep more coverage applications, I think Practical Common Lisp covers all of that. Which isn't a vote against a book focussed on the subject, though.
- phoe-krk 6y ago
  As mentioned in one of the comments, PCL does good overall coverage of the condition system but does not go in depth enough for my taste; it is a book for teaching CL, not the condition system, after all. It does not describe any applications or use cases of the condition system that do not strictly serve exception handling, and it does not show the internal construction of the condition system, especially from the approach of starting with nothing but most basic language tools which are dynamic variables, means of transferring control, closures, functions, classes, and variables.

drcode 6y ago

For those unaware, Common Lisp exception handling is heads and shoulders above almost any other programming language: Imagine having your program crash in the middle of a function, being able to inspect the call stack to find the bug, fixing it, then completing the function call from the middle of the function, at the exact point where the exception happened, with the fix in place.

tester34 6y ago
During debugging, yea?
- phoe-krk 6y ago
  Yes, all that without leaving the debugger. The debugger is left only when we want to continue execution of the program.
  - protomyth 6y ago
    I've noticed quite a few Smalltalk programmers who actually coded in the debugger. They would put a call in the program, hit it as an error, and then write new code. Does this happen with Common Lisp programmers?
    - phoe-krk 6y ago
      Smalltalk and Common Lisp are both image-based programming languages where the compiler and debugger are always available. I think debugger-oriented programming is much more common in Smalltalk than in Lisp (at least I prefer to leave the debugger back to REPL pretty often), but I think it's certainly possible to adopt. The Slime/Sly environments for emacs might accept a few patches to facilitate that programming style better.
    - p_l 6y ago
      I know of several lispers that swear by top-down "programming inside debugger", where they continuously fill in the gaps starting from top level function :-D

steveklabnik 6y ago

Fun trivia fact: Rust had conditions at one point. They weren't really used by anyone and so ended up getting dropped.

phoe-krk 6y ago
The book mentions that fact. The commit that removed them (along with the rationale) is available at https://github.com/rust-lang/rust/commit/454882dcb7fdb03867d695a88335e2d2c8f7561a https://github.com/rust-lang/rust/commit/454882dcb7fdb03867d...
- gumby 6y ago
  Those reasons seem more rust-specific than being a general reason to avoid restartable conditions — as the commit message acknowledges at the end.
  - steveklabnik 6y ago
    They're also "2014 Rust" specific; Rust changed a bunch between then and 1.0. Which doesn't mean they still wouldn't apply, just it gets harder for anything earlier than may 2015.

gumby 6y ago

The condition system is often hard for non-lisp people to understand as the best they’ve had in the past is a core dump or a c++-style exception system. The main problem for me with those is that by the time you see the exception the state has already been destroyed. This is the problem with core dumps as well: all your files and network connections have been closed. The only alternative is to use a debugger which is a high overhead situation. The roots of condition system evolved from two related hacker environments: the ITS system at the MIT AI lab and the lisp machines (both CADR and its descendants and the PARC D-machines) as well as influence for the Smalltalk folks. In ITS what is called the “shell” these days was the debugger. If your program signaled an error you were immediately in the debugger and could debug, just exit, or Force a core dump. This was the routine way to interact, even for non-programmers. The lispm did the same. Debugging was a lightweight operation. So it was natural in the 80s to develop a system that formalized this mode of operation. Because an exceptional situation by definition requires greater knowledge than is available in situ. When you catch an error in java or can++ it’s too late to look at the state of the object that caused the error without standing on your head and possibly redesigning the try block say to allocate something on the heap that can be passed to the error (which means no more RAII etc etc). In addition, try/catch typically puts the hand lin too close to the site of the problem for you to be useful; if not, by the time you get the exception there’s not much you can do but terminate the calculation politely.

phoe-krk 6y ago
> The main problem for me with those is that by the time you see the exception the state has already been destroyed. This is the problem with core dumps as well: all your files and network connections have been closed. The only alternative is to use a debugger which is a high overhead situation. This seems like two core issues with understanding the condition system. First, the currently popular exception systems always destroy the stack by unwinding it, allowing the programmer to debug only dead remains instead of a living program; second, debuggers are bolted on top of programs instead of being their internal parts, which gives many people an impression that using them as an integral part of the whole programming lifecycle is inconvenient and unnatural. The condition system in Common Lisp solves both of these issues, which IMO, paradoxically, contributes to it not being well understood outside Lisp circles.

daly 6y ago

I tried to buy (pre-order) the book. I created an account on the Apress site as described in their help section. There is no link to buy the book. Sigh.

phoe-krk 6y ago
It is an upcoming book, as the title describes. It will be available later in the year.
- MycroftJones 6y ago
  Amazon generally has a "pre-order" option. If there was a pre-order option, I also would order your book. By the time your book is in print, I will have forgotten about it. Fascinating topic; nice to see good coverage. I always felt that callbacks and exception handling left a little something to be desired. It looks like you've addressed that. When your book is in print and available for sale I hope you post again here as a reminder.
  - phoe-krk 6y ago
    The pre-order option is there! https://www.amazon.com/Common-Lisp-Condition-System-Mechanisms/dp/148426133X https://www.amazon.com/Common-Lisp-Condition-System-Mechanis... Thank you for the kind words; I hope that the book suits you well once it's out.
phoe-krk 6y ago
I just learned that preorders are available on https://www.amazon.com/Common-Lisp-Condition-System-Mechanisms/dp/148426133X https://www.amazon.com/Common-Lisp-Condition-System-Mechanis... - if preorders are your thing, that is.

mjbishop 6y ago

This book looks great! I tried to implement this in Ruby with a gem about 6 years ago and got really close. If anyone wants to know more about this, but is more fluent in Ruby, you might find this a good jumping-off point. There’s a good explanation in the README. https://github.com/michaeljbishop/mulligan Why did it not work in the end? Because even though I was able to implement an error-recovery mechanism that wouldn’t pop the stack, it still triggered all the `ensure` statements that were up the stack. It was still a valuable exercise. I'd welcome this addition in any exception-based language.

phoe-krk 6y ago
Thanks for that link! I'll include it in the book.

dan-robertson 6y ago

A problem I don’t have a good answer to is how to extend conditions (or even exceptions) to async programming models. Two problems are: 1. some code may fail multiple times (with non-async code, once an exception is caught the program can’t throw another one from the same dynamic extent of the try block; similarly once a restart is invoked and control is transferred, a condition won’t be signalled from the same dynamic extent where the restart was bound (for the purpose of this clause, we consider a retry restart to mint a new dynamic extent to bind the restart in again after retrying)) 2. Some computation may happen after you don’t care about its result because a parallel dependency has failed. E.g. suppose you have some rate limiter where you pass an async function, get a promise for it’s result, and the rate limited will queue your function to be called soon but with the condition that it will only allow say 5 calls to be in flight. Now you have some resource-downloading function which puts many separate download requests into the ratelimiter in parallel, and combines the results in such a way that the whole download fails if any request fails. But now if an early request fails, your function will continue trying to download all the other resources even though it doesn’t care about the result now that it has failed. You could imagine having a way to cancel promises and maybe you would be clever and write really solid programs and never have issues but to me this seems to be inviting very difficult to debug problems (just like expecting any thread to be killed at any random point (or at least a bunch of different points) would make correct multithreaded programming harder. Perhaps a more succinct description of problem 2 is that unwinding the “stack” isn’t well-defined with an async programming model because “stack” doesn’t really mean the call stack from the scheduler you whatever bit of async computation you happen to be doing, but rather the combined dynamic scope from everything that started the computation and is currently awaiting its completion. A related and, I think, very useful thing to have is a way to represent computations which may signal conditions or be restarted. I think CL doesn’t have a great story for this (threading isn’t really a part of the language though). An example use case would be some program which has to process some stream of events. If it gets stuck on one or some, you may want to continue processing the rest of the events while deciding what restart to invoke for the problematic ones. Perhaps an effect system could help with this.

phoe-krk 6y ago
I'll be frank: my book does not touch this topic in the slightest, and this is an open question. Common Lisp does not have any asynchronous features built in and, as I've said in some earlier comment, the condition system is fully synchronous. This topic is still waiting to be explored, either with an extension of Common Lisp or a completely different language that permits asynchronous programming with an adaptation of the condition system to this programming paradigm.