4 ms·

How to Define New Intrinsics in SBCL

92 points by chiachun 12y ago

eudox 12y ago
In certain ways, the difference between (Common) Lisp and other languages is the difference between Linux and Windows. When you have a fundamental problem in Windows, your only solution is to wait for a fix then reformat/reinstall. Similarly, when there's a problem in, say, a C compiler, all you can do is code around it, open an issue and wait for a fix in the next release. In Common Lisp? Well, the compiler's right there under these floorboards. Have a long-running web application that's affected by this bug, and don't want to restart it? Just run this in the REPL.
- jlas 12y ago
  Not sure exactly what you're trying to say here but there are plenty of open source C compilers [1], including GCC, which allow anyone to fix bugs as they see fit. To your second point about not restarting a long running web applications — running code in a REPL won't fix your web app since it's a separate process. You might need a remote debugging mechanism to connect a console to the app, but at that point it may be much easier to restart the app. [1] http://en.wikipedia.org/wiki/List_of_compilers#C_compilers http://en.wikipedia.org/wiki/List_of_compilers#C_compilers
  - zachbeane 12y ago
    > running code in a REPL won't fix your web app since it's a separate process Not in a Common Lisp web app.
    - jlas 12y ago
      Apparently what OP was talking about was a remote console into the webapp. You still can't open a random REPL and change the features of another process.
      - zachbeane 12y ago
        It's true that you can't open a random REPL and change the features of another process. It is the norm for a webapp in Common Lisp to have an accessible REPL or "remote console".
  - eudox 12y ago
    >running code in a REPL won't fix your web app since it's a separate process. In the server you have a script that fires up a Lisp process and starts a Swank[0] server on port whatever. Then the developers, from their own machines, fire up Emacs and SLIME[1] into that port and run something along the lines of > (myapp.server:start :port 8000) <#server process> > (... some work ...) [0] http://common-lisp.net/project/slime/doc/html/Lisp_002dside.html#Lisp_002dside http://common-lisp.net/project/slime/doc/html/Lisp_002dside.... [1] http://common-lisp.net/project/slime/ http://common-lisp.net/project/slime/
    - aardvark179 12y ago
      This is a great feature of Lisp, Smalltalk, and similar languages, in development. In production however it can be extremely bad since the source code you believe is needed to reconstruct an image may not be what is actually in there, or some bootstrapping may have been done by hand and incorrectly captured later on. If the image has been saved with this anomaly a couple of decades ago then it's really fun trying to sort it out. I've spent more time than I'd like to think doing software archeology to find out just why two systems behaved differently, so I would seriously recommend that if you do perform hot fixes in this manner then you reproduce the problem on a system built from source, apply and test the hot fix, make the change in the source and build your test system again to test, apply the hot fix to production, and schedule a redeploy. Fixing a live system can also be hazardous if you are replacing a function that may be on the stack at the moment, or if you are in any sort of multithreaded/process environment. Bugs become much harder to reason around and fix in those cases because there can be race conditions about which functions actually get called while you're applying your fix. TL;DR You can, but probably shouldn't.
      - lispm 12y ago
        Providing patches as compiled code loadable into applications is fairly common (!) in Lisp.
        kazinator 12y ago
        But usually those patches are already validated. Providing validated patches is one thing; developing them on the live system is another.
  - kazinator 12y ago
    Although you can fix a bug in GCC and submit it to the project (been there done that) you cannot deploy a gcc change as part of a C program. Then again, you can't really deploy a Linux bug fix as part of a Linux program either, so the whole linux-windows-lisp-others analogy is tenuous. The analogy holds in the sense that in Lisp we have less of a barrier between the programmer and the language, and in free software we have less of a barrier between users and the software. But the barriers are different. And, ironically, the barrier which is out of your way in Lisp is still out of your way if the Lisp is a closed-source, proprietary Lisp. That is to say, doing many things in the Lispy way does not require you to rebuild the implementation from source, or to have the source.
    - lispm 12y ago
      Proprietary Lisps don't need to be totally closed source. For example Allegro CL provides much of the source code. mocl provides a source code license. LispWorks otoh is mostly closed source - minus parts of the editor.
- lelf 12y ago
  > when there's a problem in, say, a C compiler, all you can do is code around it, open an issue and wait for a fix in the next release. Fix at as you'd (presumably) do in lisp. I don't see your point. (Yes, you have to recompile it, C is compiled language.) > long-running web application that's affected by this bug, and don't want to restart it? Just run this in the REPL. Yeah, if you it don't have long-running code, it isn't heavy threaded, that change doesn't alter any data structures, you don't care about possible downtime etc
agentultra 12y ago
TIL: a solid CL implementation can provide safe maps from high-level intrinsic types to low-level implementation types at run-time. Also, SBCL has at least two IRs with each able to provide optimization hints to the code generator and compiler. So cool. I love reading Paul's posts. His bit on Tobasco sort[0] was amazing. [0] http://www.pvk.ca/Blog/2012/08/27/tabasco-sort-super-optimal-merge-sort/ http://www.pvk.ca/Blog/2012/08/27/tabasco-sort-super-optimal... update: link
- _delirium 12y ago
  That is indeed a good post; thanks for linking it. Beyond the sort itself, there's a nice illustration at the end of dynamically scoped ("special") variables being useful on occasion for something other than the usual example of temporarily shadowing a configuration variable (e.g. the destination of stdout, or logging level).
talder 12y ago
Does the SBCL compiler use a separate third-party assembler? Otherwise, how does it know the POPCNT opcode/encoding? I didn't see the author specify it in the post.
- sedachv 12y ago
  SBCL comes with its own assembler: https://github.com/sbcl/sbcl/blob/master/src/compiler/assem.lisp https://github.com/sbcl/sbcl/blob/master/src/compiler/assem....
  - _delirium 12y ago
    And the instructions themselves are defined here (e.g. for x86-64): https://github.com/sbcl/sbcl/blob/master/src/compiler/x86-64/insts.lisp https://github.com/sbcl/sbcl/blob/master/src/compiler/x86-64... I think the question might've been how you can just write things like (instr popcnt...) and expect SBCL to know how to emit the right machine code for popcnt. It's mentioned briefly at the top, but the answer is that this post isn't adding the instruction (it would indeed not work if it were a new instruction SBCL doesn't yet know about). The example is just walking through modifying how it's generated by the VOPs, the code-generation level that sits a level above the assembler. The popcnt instruction itself had already been added to SBCL some time ago, with the define-instruction seen in the link above.

92 points by chiachun 12y ago

eudox 12y ago

In certain ways, the difference between (Common) Lisp and other languages is the difference between Linux and Windows. When you have a fundamental problem in Windows, your only solution is to wait for a fix then reformat/reinstall. Similarly, when there's a problem in, say, a C compiler, all you can do is code around it, open an issue and wait for a fix in the next release. In Common Lisp? Well, the compiler's right there under these floorboards. Have a long-running web application that's affected by this bug, and don't want to restart it? Just run this in the REPL.

jlas 12y ago
Not sure exactly what you're trying to say here but there are plenty of open source C compilers [1], including GCC, which allow anyone to fix bugs as they see fit. To your second point about not restarting a long running web applications — running code in a REPL won't fix your web app since it's a separate process. You might need a remote debugging mechanism to connect a console to the app, but at that point it may be much easier to restart the app. [1] http://en.wikipedia.org/wiki/List_of_compilers#C_compilers http://en.wikipedia.org/wiki/List_of_compilers#C_compilers
- zachbeane 12y ago
  > running code in a REPL won't fix your web app since it's a separate process Not in a Common Lisp web app.
  - jlas 12y ago
    Apparently what OP was talking about was a remote console into the webapp. You still can't open a random REPL and change the features of another process.
    - zachbeane 12y ago
      It's true that you can't open a random REPL and change the features of another process. It is the norm for a webapp in Common Lisp to have an accessible REPL or "remote console".
- eudox 12y ago
  >running code in a REPL won't fix your web app since it's a separate process. In the server you have a script that fires up a Lisp process and starts a Swank[0] server on port whatever. Then the developers, from their own machines, fire up Emacs and SLIME[1] into that port and run something along the lines of > (myapp.server:start :port 8000) <#server process> > (... some work ...) [0] http://common-lisp.net/project/slime/doc/html/Lisp_002dside.html#Lisp_002dside http://common-lisp.net/project/slime/doc/html/Lisp_002dside.... [1] http://common-lisp.net/project/slime/ http://common-lisp.net/project/slime/
  - aardvark179 12y ago
    This is a great feature of Lisp, Smalltalk, and similar languages, in development. In production however it can be extremely bad since the source code you believe is needed to reconstruct an image may not be what is actually in there, or some bootstrapping may have been done by hand and incorrectly captured later on. If the image has been saved with this anomaly a couple of decades ago then it's really fun trying to sort it out. I've spent more time than I'd like to think doing software archeology to find out just why two systems behaved differently, so I would seriously recommend that if you do perform hot fixes in this manner then you reproduce the problem on a system built from source, apply and test the hot fix, make the change in the source and build your test system again to test, apply the hot fix to production, and schedule a redeploy. Fixing a live system can also be hazardous if you are replacing a function that may be on the stack at the moment, or if you are in any sort of multithreaded/process environment. Bugs become much harder to reason around and fix in those cases because there can be race conditions about which functions actually get called while you're applying your fix. TL;DR You can, but probably shouldn't.
    - lispm 12y ago
      Providing patches as compiled code loadable into applications is fairly common (!) in Lisp.
      - kazinator 12y ago
        But usually those patches are already validated. Providing validated patches is one thing; developing them on the live system is another.
- kazinator 12y ago
  Although you can fix a bug in GCC and submit it to the project (been there done that) you cannot deploy a gcc change as part of a C program. Then again, you can't really deploy a Linux bug fix as part of a Linux program either, so the whole linux-windows-lisp-others analogy is tenuous. The analogy holds in the sense that in Lisp we have less of a barrier between the programmer and the language, and in free software we have less of a barrier between users and the software. But the barriers are different. And, ironically, the barrier which is out of your way in Lisp is still out of your way if the Lisp is a closed-source, proprietary Lisp. That is to say, doing many things in the Lispy way does not require you to rebuild the implementation from source, or to have the source.
  - lispm 12y ago
    Proprietary Lisps don't need to be totally closed source. For example Allegro CL provides much of the source code. mocl provides a source code license. LispWorks otoh is mostly closed source - minus parts of the editor.
lelf 12y ago
> when there's a problem in, say, a C compiler, all you can do is code around it, open an issue and wait for a fix in the next release. Fix at as you'd (presumably) do in lisp. I don't see your point. (Yes, you have to recompile it, C is compiled language.) > long-running web application that's affected by this bug, and don't want to restart it? Just run this in the REPL. Yeah, if you it don't have long-running code, it isn't heavy threaded, that change doesn't alter any data structures, you don't care about possible downtime etc

agentultra 12y ago

TIL: a solid CL implementation can provide safe maps from high-level intrinsic types to low-level implementation types at run-time. Also, SBCL has at least two IRs with each able to provide optimization hints to the code generator and compiler. So cool. I love reading Paul's posts. His bit on Tobasco sort[0] was amazing. [0] http://www.pvk.ca/Blog/2012/08/27/tabasco-sort-super-optimal-merge-sort/ http://www.pvk.ca/Blog/2012/08/27/tabasco-sort-super-optimal... update: link

_delirium 12y ago
That is indeed a good post; thanks for linking it. Beyond the sort itself, there's a nice illustration at the end of dynamically scoped ("special") variables being useful on occasion for something other than the usual example of temporarily shadowing a configuration variable (e.g. the destination of stdout, or logging level).

talder 12y ago

Does the SBCL compiler use a separate third-party assembler? Otherwise, how does it know the POPCNT opcode/encoding? I didn't see the author specify it in the post.

sedachv 12y ago
SBCL comes with its own assembler: https://github.com/sbcl/sbcl/blob/master/src/compiler/assem.lisp https://github.com/sbcl/sbcl/blob/master/src/compiler/assem....
- _delirium 12y ago
  And the instructions themselves are defined here (e.g. for x86-64): https://github.com/sbcl/sbcl/blob/master/src/compiler/x86-64/insts.lisp https://github.com/sbcl/sbcl/blob/master/src/compiler/x86-64... I think the question might've been how you can just write things like (instr popcnt...) and expect SBCL to know how to emit the right machine code for popcnt. It's mentioned briefly at the top, but the answer is that this post isn't adding the instruction (it would indeed not work if it were a new instruction SBCL doesn't yet know about). The example is just walking through modifying how it's generated by the VOPs, the code-generation level that sits a level above the assembler. The popcnt instruction itself had already been added to SBCL some time ago, with the define-instruction seen in the link above.