6 ms·

Emacs Lisp Readable Closures (2013)

106 points by nanna 5y ago

aidenn0 5y ago
How does this work if the closure will mutate a closed-over location? e.g. (let ((x 1)) (funcall (read (prin1-to-string (lambda () (setf x 3))))) (print x)) [edit] Realized I have Emacs installed and tried it; the above prints "1" while calling the lambda without first serializing causes the (print x) to print "3"
sitkack 5y ago
Python Dill, https://github.com/uqfoundation/dill https://github.com/uqfoundation/dill can serialize closures.
moonchild 5y ago
> I am unaware of any other programming language that has this feature S7 scheme: > (define f (let ((x 5)) (lambda (y) (+ x y)))) f > (f 4) 9 > (object->string f :readable) "(let ((x 5)) (lambda (y) (+ x y)))" There is even talk of printing continuations readably.
- bjoli 5y ago
  IIRC the python-on-guile project can probably do this using some C voodoo. I believe it can serialize continuations and make deep copies of generators with mutable state. Edit: however, I doubt the serialized format is human readable by default...
- taeric 5y ago
  This isn't really the same, though. You are printing f. Print the closure returned by f.
  - moonchild 5y ago
    This is scheme. f doesn't return a closure; it is a closure, and it returns a number.
    - taeric 5y ago
      Still feels different. For example, this (let ((x 3)) (let ((y (lambda (z) (lambda () (+ x z))))) (message "%s" (funcall y 3)) (message "hello") (setq x 5) (message "%s" (funcall y 3)))) will produce the following messages (closure ((z . 3) (x . 3) t) nil (+ x z)) hello (closure ((z . 3) (x . 5) t) nil (+ x z)) While this: (let ((x 3)) (let ((y (lambda (z) (lambda () (let ((w 0)) (+ w x z)))))) (message "%s" (funcall y 3)) (message "hello") (setq x 5) (message "%s" (funcall y 3)))) will print: (closure ((z . 3) (x . 3) t) nil (let ((w 0)) (+ w x z))) hello (closure ((z . 3) (x . 5) t) nil (let ((w 0)) (+ w x z))) Is it just that emacs is more explicit about which values were closed over? Edit: Actually, I see my confusion, you accidentally printed 'f', not 'g', which I think was your intent?
      - moonchild 5y ago
        > Edit: Actually, I see my confusion, you accidentally printed 'f', not 'g', which I think was your intent? I am not quite sure what you mean? There was no 'g' at all in my code snippet except for that in 'object->string'. Unless you are mistaking the '9' for one?
        taeric 5y ago
        Is what happens when I write so late at night. I mistook the 9 for a g, thinking it was showing the body of the lambda. Really, I was just reading it all sorts of wrong. I still feel this is somehow different, but I can't express the difference well. Note that I should say I am not at all surprised that scheme could do this, for what its worth. I'm just saying that this feels different.
      - moonchild 5y ago
        Here are your snippets in s7, and their output. (let ((x 3)) (let ((y (lambda (z) (lambda () (+ x z))))) (format #t "~a~%" (object->string (y 3) :readable)) (format #t "hello~%") (set! x 5) (format #t "~a~%" (object->string (y 3) :readable)))) Which gives: (let ((z 3) (x 3)) (lambda () (+ x z))) hello (let ((z 3) (x 5)) (lambda () (+ x z))) And: (let ((x 3)) (let ((y (lambda (z) (lambda () (let ((w 0)) (+ w x z)))))) (format #t "~a~%" (object->string (y 3) :readable)) (format #t "hello~%") (set! x 5) (format #t "~a~%" (object->string (y 3) :readable)))) Which gives: (let ((z 3) (x 3)) (lambda () (let ((w 0)) (+ w x z)))) hello (let ((z 3) (x 5)) (lambda () (let ((w 0)) (+ w x z))))
- wdkrnls 5y ago
  R can do this too if I understand correctly.
brabel 5y ago
Wow, this is a really cool feature... specially the compiled lambda form with readable bytecode. I will have a look at the "next" article from that post that explains emacs bytecode in more detail, I can imagine that opens the door to really interesting features in emacs.
throwaway81523 5y ago
From 2013, but interesting, and lots of other good Emacs-related stuff on that site.
- nanna 5y ago
  Seconded. The author is the creator of the magestic Elfeed.
  - agumonkey 5y ago
    and has a talent for short enough yet deep enough articles
sreeramvenkat 5y ago
Zenlisp by Nils Holm has readable closure too. http://t3x.org/ http://t3x.org/
- WalterGR 5y ago
  Zenlisp on Github: https://github.com/barak/zenlisp https://github.com/barak/zenlisp Wow he’s written lots of books. Can you recommend any?
raspasov 5y ago
Even though not part of the language, Clojure can do this via a library like https://github.com/technomancy/serializable-fn https://github.com/technomancy/serializable-fn (I just learned about this).
TeMPOraL 5y ago
For comparison, since I'm using Emacs with native-compilation: Interpreted result: ELISP> (hn/foo :bar :ignored) (closure ((y . :ignored) (x . :bar) t) nil x) This is equivalent to the output in the article. Now, after running compile-defun on the function and calling it, I get: ELISP> (hn/foo :bar :ignored) #f(compiled-function () #<bytecode 0x1f40000a8a1d>) This is the byte-compiled case, which sadly is not readable for some reason. That said, IELM (Elisp REPL) makes that #<bytecode ...> form clickable (it's a so-called "presentation"), which leads straight to the following disassembly: byte code: args: nil 0 varref x 1 return After native-compiling via (load (native-compile "file.el")), I get the same output as with byte-compiled case (+/- the address in #<bytecode ...> form), but the disassembly is slightly different: byte code: args: nil 0 constant :bar 1 return EDIT: turns out it's IELM that's playing games with the printer. If I write: (print (hn/foo :bar :ignored)) instead, I get the readable output from the article, in both byte and native-compiled cases. My understanding of the latter is, the closure returned from a native-compiled function is itself only byte-compiled (at least when used in REPL), and the difference in bytecode assembly come from the latter running through a more thorough optimization process.
- masklinn 5y ago
  So how does this interact with updating the captured environment? Does the printing essentially just show a snapshot of the environment at that point? I assume the (closure) is read back as a new closure such that the environment will be restored as an environment shared between invocations of the read result?
bluefox 5y ago
You can serialize closures (and continuations) in Common Lisp using Common Cold. The serialized representation of a closure contains a tag instead of the actual code, but the forms are also available in case you need to serialize them as well. CL-USER> (write-to-string (let ((y 4)) (common-cold:slambda (x) (* x y)))) "#.(S:F '1 '(4))" CL-USER> (funcall (read-from-string *) 3) ;; * here means reference to the last returned value 12
- TeMPOraL 5y ago
  > The serialized representation of a closure contains a tag instead of the actual code Was going to ask about that. For readers unfamiliar with Common Lisp, the output is essentially equivalent to "(eval '(s:f '1 '(4)))" because #. is a reader macro that means "instead of returning the s-expression, return the result of evaluating it". I.e. it's a way to run arbitrary Lisp code at parsing stage. This is not something you want to see in a serialization format. Ignoring the security implications[0], the result seems to rely on the runtime state of the Lisp image (I assume that '1 bit is a key to the actual code of the function) - so you won't be able to persist it and load after program restart, or send it to another instance of the program. -- [0] - We're talking serializing code here anyway, so adding arbitrary code execution to paring changes nothing.
  - bluefox 5y ago
    Like I said, the mapping from tags to forms is available and you can serialize that as well, if you wish. Using a tag is useful, as it makes for a compact serialization when creating many closures of the same function. With regards to security, you would want to attach an authentication tag that you can verify prior to deserialization. Common Cold comes with an example of a Hunchentoot handler that compresses and encrypts the serialized continuation representations.
- kagevf 5y ago
  Thank you for sharing this! I tried looking for this on QL and github but couldn't find it ... do you know if the common cold source code is still available somewhere?

106 points by nanna 5y ago

How does this work if the closure will mutate a closed-over location? e.g. (let ((x 1)) (funcall (read (prin1-to-string (lambda () (setf x 3))))) (print x)) [edit] Realized I have Emacs installed and tried it; the above prints "1" while calling the lambda without first serializing causes the (print x) to print "3"

sitkack 5y ago

Python Dill, https://github.com/uqfoundation/dill https://github.com/uqfoundation/dill can serialize closures.

moonchild 5y ago

> I am unaware of any other programming language that has this feature S7 scheme: > (define f (let ((x 5)) (lambda (y) (+ x y)))) f > (f 4) 9 > (object->string f :readable) "(let ((x 5)) (lambda (y) (+ x y)))" There is even talk of printing continuations readably.

bjoli 5y ago
IIRC the python-on-guile project can probably do this using some C voodoo. I believe it can serialize continuations and make deep copies of generators with mutable state. Edit: however, I doubt the serialized format is human readable by default...
taeric 5y ago
This isn't really the same, though. You are printing f. Print the closure returned by f.
- moonchild 5y ago
  This is scheme. f doesn't return a closure; it is a closure, and it returns a number.
  - taeric 5y ago
    Still feels different. For example, this (let ((x 3)) (let ((y (lambda (z) (lambda () (+ x z))))) (message "%s" (funcall y 3)) (message "hello") (setq x 5) (message "%s" (funcall y 3)))) will produce the following messages (closure ((z . 3) (x . 3) t) nil (+ x z)) hello (closure ((z . 3) (x . 5) t) nil (+ x z)) While this: (let ((x 3)) (let ((y (lambda (z) (lambda () (let ((w 0)) (+ w x z)))))) (message "%s" (funcall y 3)) (message "hello") (setq x 5) (message "%s" (funcall y 3)))) will print: (closure ((z . 3) (x . 3) t) nil (let ((w 0)) (+ w x z))) hello (closure ((z . 3) (x . 5) t) nil (let ((w 0)) (+ w x z))) Is it just that emacs is more explicit about which values were closed over? Edit: Actually, I see my confusion, you accidentally printed 'f', not 'g', which I think was your intent?
    - moonchild 5y ago
      > Edit: Actually, I see my confusion, you accidentally printed 'f', not 'g', which I think was your intent? I am not quite sure what you mean? There was no 'g' at all in my code snippet except for that in 'object->string'. Unless you are mistaking the '9' for one?
      - taeric 5y ago
        Is what happens when I write so late at night. I mistook the 9 for a g, thinking it was showing the body of the lambda. Really, I was just reading it all sorts of wrong. I still feel this is somehow different, but I can't express the difference well. Note that I should say I am not at all surprised that scheme could do this, for what its worth. I'm just saying that this feels different.
    - moonchild 5y ago
      Here are your snippets in s7, and their output. (let ((x 3)) (let ((y (lambda (z) (lambda () (+ x z))))) (format #t "~a~%" (object->string (y 3) :readable)) (format #t "hello~%") (set! x 5) (format #t "~a~%" (object->string (y 3) :readable)))) Which gives: (let ((z 3) (x 3)) (lambda () (+ x z))) hello (let ((z 3) (x 5)) (lambda () (+ x z))) And: (let ((x 3)) (let ((y (lambda (z) (lambda () (let ((w 0)) (+ w x z)))))) (format #t "~a~%" (object->string (y 3) :readable)) (format #t "hello~%") (set! x 5) (format #t "~a~%" (object->string (y 3) :readable)))) Which gives: (let ((z 3) (x 3)) (lambda () (let ((w 0)) (+ w x z)))) hello (let ((z 3) (x 5)) (lambda () (let ((w 0)) (+ w x z))))
wdkrnls 5y ago
R can do this too if I understand correctly.

brabel 5y ago

Wow, this is a really cool feature... specially the compiled lambda form with readable bytecode. I will have a look at the "next" article from that post that explains emacs bytecode in more detail, I can imagine that opens the door to really interesting features in emacs.

throwaway81523 5y ago

From 2013, but interesting, and lots of other good Emacs-related stuff on that site.

nanna 5y ago
Seconded. The author is the creator of the magestic Elfeed.
- agumonkey 5y ago
  and has a talent for short enough yet deep enough articles

sreeramvenkat 5y ago

Zenlisp by Nils Holm has readable closure too. http://t3x.org/ http://t3x.org/

WalterGR 5y ago
Zenlisp on Github: https://github.com/barak/zenlisp https://github.com/barak/zenlisp Wow he’s written lots of books. Can you recommend any?

raspasov 5y ago

Even though not part of the language, Clojure can do this via a library like https://github.com/technomancy/serializable-fn https://github.com/technomancy/serializable-fn (I just learned about this).

TeMPOraL 5y ago

For comparison, since I'm using Emacs with native-compilation: Interpreted result: ELISP> (hn/foo :bar :ignored) (closure ((y . :ignored) (x . :bar) t) nil x) This is equivalent to the output in the article. Now, after running compile-defun on the function and calling it, I get: ELISP> (hn/foo :bar :ignored) #f(compiled-function () #<bytecode 0x1f40000a8a1d>) This is the byte-compiled case, which sadly is not readable for some reason. That said, IELM (Elisp REPL) makes that #<bytecode ...> form clickable (it's a so-called "presentation"), which leads straight to the following disassembly: byte code: args: nil 0 varref x 1 return After native-compiling via (load (native-compile "file.el")), I get the same output as with byte-compiled case (+/- the address in #<bytecode ...> form), but the disassembly is slightly different: byte code: args: nil 0 constant :bar 1 return EDIT: turns out it's IELM that's playing games with the printer. If I write: (print (hn/foo :bar :ignored)) instead, I get the readable output from the article, in both byte and native-compiled cases. My understanding of the latter is, the closure returned from a native-compiled function is itself only byte-compiled (at least when used in REPL), and the difference in bytecode assembly come from the latter running through a more thorough optimization process.

masklinn 5y ago
So how does this interact with updating the captured environment? Does the printing essentially just show a snapshot of the environment at that point? I assume the (closure) is read back as a new closure such that the environment will be restored as an environment shared between invocations of the read result?

bluefox 5y ago

You can serialize closures (and continuations) in Common Lisp using Common Cold. The serialized representation of a closure contains a tag instead of the actual code, but the forms are also available in case you need to serialize them as well. CL-USER> (write-to-string (let ((y 4)) (common-cold:slambda (x) (* x y)))) "#.(S:F '1 '(4))" CL-USER> (funcall (read-from-string *) 3) ;; * here means reference to the last returned value 12

TeMPOraL 5y ago
> The serialized representation of a closure contains a tag instead of the actual code Was going to ask about that. For readers unfamiliar with Common Lisp, the output is essentially equivalent to "(eval '(s:f '1 '(4)))" because #. is a reader macro that means "instead of returning the s-expression, return the result of evaluating it". I.e. it's a way to run arbitrary Lisp code at parsing stage. This is not something you want to see in a serialization format. Ignoring the security implications[0], the result seems to rely on the runtime state of the Lisp image (I assume that '1 bit is a key to the actual code of the function) - so you won't be able to persist it and load after program restart, or send it to another instance of the program. -- [0] - We're talking serializing code here anyway, so adding arbitrary code execution to paring changes nothing.
- bluefox 5y ago
  Like I said, the mapping from tags to forms is available and you can serialize that as well, if you wish. Using a tag is useful, as it makes for a compact serialization when creating many closures of the same function. With regards to security, you would want to attach an authentication tag that you can verify prior to deserialization. Common Cold comes with an example of a Hunchentoot handler that compresses and encrypts the serialized continuation representations.
kagevf 5y ago
Thank you for sharing this! I tried looking for this on QL and github but couldn't find it ... do you know if the common cold source code is still available somewhere?