8 ms·

Scripting in Common Lisp

130 points by mr_tyzic 8y ago

codemac 8y ago
This is a great guide, and solving quick real problems is how people get started programming in general. I think seeing scsh for the first time made me realize just how good & bad shells are.
- rauhl 8y ago
  > I think seeing scsh for the first time made me realize just how good & bad shells are. Oddly enough, just last week I was trying to do some scsh-like things with Common Lisp, and I ended up banging my ahead against its case-mapping behaviour (Common Lisp upcases input by default). There’s definitely an opening for a Lisp version of something like scsh. There’s some tricky stuff to be aware of (e.g. SBCL’s RUN-PROGRAM puts processes in a different process group, which mean C-c does the wrong thing), but done well it could really be awesome.
  - aidenn0 8y ago
    Setting case to invert by default fixes a lot of those problems (you just invert the case again before passing the string onto the shell). Also, for doing many shell things fork/execv is the only thing that gives you sufficient control, so I suggest avoiding run-program and using those instead.
jstewartmobile 8y ago
Where is your piping example?
- junke 8y ago
  I do not use cl-launch, but for some tasks at job I wanted to pipe programs, and just wanted to try implementing something myself. The with-pipeline macro creates temporary fifos to connect programs and threads (random example): (let ((counter 0)) (with-pipeline () (program "ls" "-1la") (lambda-line (line) (format t "~x~%" (incf counter (length line)))) (program "sed" "-n" "s/A/_/g ; /__/ p")) counter) Outputs and return: 2__8 15399 Macroexpansion: (catch :pipeline (block nil (let* ((#:input%2195 (pipeline::ensure-stream nil :input nil)) (#:output%2196 (pipeline::ensure-stream t :output nil)) (#:error%2197 (pipeline::ensure-stream :output :error #:output%2196))) (let ((#:g2199 (program "ls" "-1la")) (#:g2200 (lambda-line (line) (format t "~x~%" (incf counter (length line))))) (#:g2201 (program "sed" "-n" "s/a/_/g ; /__/ p"))) (pipeline.pipes:with-pipes% (#:pipes2198 2) (lastcar (mapcar #'pipeline.filters:clean (list (pipeline.filters:spawn #:g2199 :input #:input%2195 :output (pipeline.pipes:pipe-out (svref #:pipes2198 0)) :error #:error%2197 :wait nil) (pipeline.filters:spawn #:g2200 :input (pipeline.pipes:pipe-in (svref #:pipes2198 0)) :output (pipeline.pipes:pipe-out (svref #:pipes2198 1)) :error #:error%2197 :wait nil) (pipeline.filters:spawn #:g2201 :input (pipeline.pipes:pipe-in (svref #:pipes2198 1)) :output #:output%2196 :error #:error%2197 :wait t))))))))) https://github.com/christophejunke/pipeline https://github.com/christophejunke/pipeline
- zhaqenl 8y ago
  Here's an example from my `pelo` project: https://github.com/zhaqenl/pelo/blob/master/pelo.lisp#L112 https://github.com/zhaqenl/pelo/blob/master/pelo.lisp#L112 (defun get-ping (host) "Get ping reply from host." (inferior-shell:run/ss `(inferior-shell:pipe (ping -c 1 ,host) (grep "time=") (sed -e "s/^.*time=//;s/ *ms$//")))) The idea is from a similar project in a different Lisp dialect: https://github.com/ebzzry/pell/blob/master/pell#L124 https://github.com/ebzzry/pell/blob/master/pell#L124
agumonkey 8y ago
Pretty nice considering the speed and capabilities of [SB]CL relative to other mainstream scripting languages.
- deleted 8y ago
  [deleted]
- TeMPOraL 8y ago
  I shy away from CL scripting due to runtime startup time, though if you compile it down to binary it's manageable (if one doesn't mind a 50MB executable). That said, I wonder why nobody tried a "hybrid" solution, in which the multi-call binary is implemented as a resident, self-restarting process, that executes each request in a lightweight sandbox. Actual shell commands would then be implemented as a request to that process, and incur only the costs of setting up IO streams on startup.
  - agumonkey 8y ago
    what CL implementation ? sbcl ? I don't know what's the leanest CL out there (even if not the most featured, since we're talking about short scripts)
    - TeMPOraL 8y ago
      SBCL mostly. I probably should give GCL a try here. SBCL is fast as hell once it starts up, and bare SBCL starts up quickly, but once you add Quicklisp and start loading external systems, the startup time gets noticeable - though again, this gets reduced quickly iff you dump a binary of your image with dependencies already loaded. The one CL script I wrote that I actually use[0] (for controlling Hue lamps) I have dumped into binary just to cut down startup time to below the point I notice. -- [0] - https://github.com/TeMPOraL/hju/ https://github.com/TeMPOraL/hju/
      - agumonkey 8y ago
        SBCL does have a tiny delay (akin to a raw JVM). CLISP is near interactive though. But that's without quicklisp.
jolmg 8y ago
This is the first time I've seen content provided in Esperanto like this. Pretty cool. Even cooler is that the Esperanto version seems to be the original, and then I suppose it was translated to English a year later. Huh, now I see the whole blog defaults to Esperanto. I wonder how much of the net takes that stand of Esperanto-first.
hobgoblin1234 8y ago
It's not common lisp, but most of my Emacs use has been as a shell scripting alternative. Workflow being based around interactive improvement of a script from a simple interactive-macro through to things that are effectively done as shell scripts. The end result is something that can be run as a shell script, although translating to another language i.e. CL, Ruby, Go, etc. is the usual path for me to make better use / performance. The old joke about Emacs being an operating system that needs a good editor is on need of a reboot. Emacs is an interactive computing and development environment with half a dozen good text editors. Factual isn't as funny though.
- anon10938750136 8y ago
  Emacs is my favourite software, but elisp is one of my least favourite lisps. The other day I stumbled upon Shen language after some years and noticed it now has BSD license and an elisp implementation. I tried to add it to spacemacs and, to my surprise, it just worked. When I have some spare time the next thing I'll do will be to learn some Shen and see if for shell scripting it is an improvement over elisp.
  - serialdev 8y ago
    Wow! thanks for pointing out there is an Elisp Shen implementation!
    - gglitch 8y ago
      Erlang too, fwiw.
aasasd 8y ago
I'm somewhat baffled that Hy didn't gain any traction. Let's review the facts: - A lot of people love Python. - Plenty of people love Lisps, afaict with a sizeable overlap with the first group. - Hy, which is a Lisp on the Python platform, has no popularity whatsoever compared to other Lisps like Clojure. The heck?
- jjtheblunt 8y ago
  Python can't use multiple cores, in contrast to the other lisps?
  - gnulinux 8y ago
    What does language have to do with "use multiple cores"? Python has both multiprocessing and threading libraries.
    - pjmlp 8y ago
      And the GIL on CPython, but yeah multiple process are also an option.
- ludston 8y ago
  It doesn't surprise me. It does not have a strong selling point. I go on to the documentation page, and see that yes, it is nice clean lisp-like language, but I can't see what problem it solves that Clojure doesn't already solve. (And clojure has the existing library/community support to give it the edge.) EDIT: That's not to say that there isn't one. I just am not immediately seeing it.
- aidenn0 8y ago
  Hy may have changed since I last looked at it, but when I had looked at it, Hy was more of an s-expression syntax bolted onto python. Some random thoughts on why this matters: Clojure is very much not a thin s-expression layer for java; that would have been terrible (Python is more like lisp than java which is why Hy is tenable whatsoever). In particular, CL style macros with out some first-class concept of symbols are a footgun[1], but I'm sure I could find other issues. Another item that can't be ignored is that Hy selected the clojure syntax, which is going to be a turn-off for anyone coming from Lisp or Scheme. Finally, there are hundreds of thin s-expression layers on top of existing languages, and most of them are terrible; Hy has done nothing to stand out from this crowd, so most lispers will not look closely at it. 1: See https://youtu.be/cPNkH-7PRTk?t=4786 https://youtu.be/cPNkH-7PRTk?t=4786 for Rich's discussion of how he managed this issue in Clojure.
kakw 8y ago
I recommend roswell https://github.com/roswell/roswell https://github.com/roswell/roswell .
kimi 8y ago
I use scripting extensively in Clojure, both on the JVM and on Planck (JS). You have the choice of slower startup times but a ton of libraries to do anything you want, versus short start-up times but less libraries, using the same language. I use https://github.com/l3nz/cli-matic https://github.com/l3nz/cli-matic as a wrapper and it works on both.
- zebraman 8y ago
  Using modern Clojure's clj is so much easier than the CL procedure.
  - eggy 8y ago
    Until you have to deal with unintelligible Java stack traces.
    - islon 8y ago
      They are intelligible if you know Java and Clojure 1.10 improved stack traces.
      - TeMPOraL 8y ago
        Yup. Common Lisp stack traces tend to be no better, anyway, at least with SBCL and CCL. Up to 90% of a trace you see in the inspector can be compiler/runtime infrastructure. Given that macros don't show up, only their expansions, the backtrace can be full of stuff like SB-KERNEL:%FUNCTION-THAT-I-DONT-RECOGNIZE-BUT-MACROEXPANDED-FROM-SOMETHING-I-KNOW. Like in Java, most of the time the top few lines are the ones related to your code, but just like in Java, there are exceptions, in which the source of the problem ends up in the middle of the backtrace, surrounded by infrastructure frames.
        deleted 8y ago
        [deleted]

130 points by mr_tyzic 8y ago

This is a great guide, and solving quick real problems is how people get started programming in general. I think seeing scsh for the first time made me realize just how good & bad shells are.

rauhl 8y ago
> I think seeing scsh for the first time made me realize just how good & bad shells are. Oddly enough, just last week I was trying to do some scsh-like things with Common Lisp, and I ended up banging my ahead against its case-mapping behaviour (Common Lisp upcases input by default). There’s definitely an opening for a Lisp version of something like scsh. There’s some tricky stuff to be aware of (e.g. SBCL’s RUN-PROGRAM puts processes in a different process group, which mean C-c does the wrong thing), but done well it could really be awesome.
- aidenn0 8y ago
  Setting case to invert by default fixes a lot of those problems (you just invert the case again before passing the string onto the shell). Also, for doing many shell things fork/execv is the only thing that gives you sufficient control, so I suggest avoiding run-program and using those instead.

jstewartmobile 8y ago

Where is your piping example?

junke 8y ago
I do not use cl-launch, but for some tasks at job I wanted to pipe programs, and just wanted to try implementing something myself. The with-pipeline macro creates temporary fifos to connect programs and threads (random example): (let ((counter 0)) (with-pipeline () (program "ls" "-1la") (lambda-line (line) (format t "~x~%" (incf counter (length line)))) (program "sed" "-n" "s/A/_/g ; /__/ p")) counter) Outputs and return: 2__8 15399 Macroexpansion: (catch :pipeline (block nil (let* ((#:input%2195 (pipeline::ensure-stream nil :input nil)) (#:output%2196 (pipeline::ensure-stream t :output nil)) (#:error%2197 (pipeline::ensure-stream :output :error #:output%2196))) (let ((#:g2199 (program "ls" "-1la")) (#:g2200 (lambda-line (line) (format t "~x~%" (incf counter (length line))))) (#:g2201 (program "sed" "-n" "s/a/_/g ; /__/ p"))) (pipeline.pipes:with-pipes% (#:pipes2198 2) (lastcar (mapcar #'pipeline.filters:clean (list (pipeline.filters:spawn #:g2199 :input #:input%2195 :output (pipeline.pipes:pipe-out (svref #:pipes2198 0)) :error #:error%2197 :wait nil) (pipeline.filters:spawn #:g2200 :input (pipeline.pipes:pipe-in (svref #:pipes2198 0)) :output (pipeline.pipes:pipe-out (svref #:pipes2198 1)) :error #:error%2197 :wait nil) (pipeline.filters:spawn #:g2201 :input (pipeline.pipes:pipe-in (svref #:pipes2198 1)) :output #:output%2196 :error #:error%2197 :wait t))))))))) https://github.com/christophejunke/pipeline https://github.com/christophejunke/pipeline
zhaqenl 8y ago
Here's an example from my `pelo` project: https://github.com/zhaqenl/pelo/blob/master/pelo.lisp#L112 https://github.com/zhaqenl/pelo/blob/master/pelo.lisp#L112 (defun get-ping (host) "Get ping reply from host." (inferior-shell:run/ss `(inferior-shell:pipe (ping -c 1 ,host) (grep "time=") (sed -e "s/^.*time=//;s/ *ms$//")))) The idea is from a similar project in a different Lisp dialect: https://github.com/ebzzry/pell/blob/master/pell#L124 https://github.com/ebzzry/pell/blob/master/pell#L124

agumonkey 8y ago

Pretty nice considering the speed and capabilities of [SB]CL relative to other mainstream scripting languages.

deleted 8y ago
[deleted]
TeMPOraL 8y ago
I shy away from CL scripting due to runtime startup time, though if you compile it down to binary it's manageable (if one doesn't mind a 50MB executable). That said, I wonder why nobody tried a "hybrid" solution, in which the multi-call binary is implemented as a resident, self-restarting process, that executes each request in a lightweight sandbox. Actual shell commands would then be implemented as a request to that process, and incur only the costs of setting up IO streams on startup.
- agumonkey 8y ago
  what CL implementation ? sbcl ? I don't know what's the leanest CL out there (even if not the most featured, since we're talking about short scripts)
  - TeMPOraL 8y ago
    SBCL mostly. I probably should give GCL a try here. SBCL is fast as hell once it starts up, and bare SBCL starts up quickly, but once you add Quicklisp and start loading external systems, the startup time gets noticeable - though again, this gets reduced quickly iff you dump a binary of your image with dependencies already loaded. The one CL script I wrote that I actually use[0] (for controlling Hue lamps) I have dumped into binary just to cut down startup time to below the point I notice. -- [0] - https://github.com/TeMPOraL/hju/ https://github.com/TeMPOraL/hju/
    - agumonkey 8y ago
      SBCL does have a tiny delay (akin to a raw JVM). CLISP is near interactive though. But that's without quicklisp.

jolmg 8y ago

This is the first time I've seen content provided in Esperanto like this. Pretty cool. Even cooler is that the Esperanto version seems to be the original, and then I suppose it was translated to English a year later. Huh, now I see the whole blog defaults to Esperanto. I wonder how much of the net takes that stand of Esperanto-first.

hobgoblin1234 8y ago

It's not common lisp, but most of my Emacs use has been as a shell scripting alternative. Workflow being based around interactive improvement of a script from a simple interactive-macro through to things that are effectively done as shell scripts. The end result is something that can be run as a shell script, although translating to another language i.e. CL, Ruby, Go, etc. is the usual path for me to make better use / performance. The old joke about Emacs being an operating system that needs a good editor is on need of a reboot. Emacs is an interactive computing and development environment with half a dozen good text editors. Factual isn't as funny though.

anon10938750136 8y ago
Emacs is my favourite software, but elisp is one of my least favourite lisps. The other day I stumbled upon Shen language after some years and noticed it now has BSD license and an elisp implementation. I tried to add it to spacemacs and, to my surprise, it just worked. When I have some spare time the next thing I'll do will be to learn some Shen and see if for shell scripting it is an improvement over elisp.
- serialdev 8y ago
  Wow! thanks for pointing out there is an Elisp Shen implementation!
  - gglitch 8y ago
    Erlang too, fwiw.

aasasd 8y ago

I'm somewhat baffled that Hy didn't gain any traction. Let's review the facts: - A lot of people love Python. - Plenty of people love Lisps, afaict with a sizeable overlap with the first group. - Hy, which is a Lisp on the Python platform, has no popularity whatsoever compared to other Lisps like Clojure. The heck?

jjtheblunt 8y ago
Python can't use multiple cores, in contrast to the other lisps?
- gnulinux 8y ago
  What does language have to do with "use multiple cores"? Python has both multiprocessing and threading libraries.
  - pjmlp 8y ago
    And the GIL on CPython, but yeah multiple process are also an option.
ludston 8y ago
It doesn't surprise me. It does not have a strong selling point. I go on to the documentation page, and see that yes, it is nice clean lisp-like language, but I can't see what problem it solves that Clojure doesn't already solve. (And clojure has the existing library/community support to give it the edge.) EDIT: That's not to say that there isn't one. I just am not immediately seeing it.
aidenn0 8y ago
Hy may have changed since I last looked at it, but when I had looked at it, Hy was more of an s-expression syntax bolted onto python. Some random thoughts on why this matters: Clojure is very much not a thin s-expression layer for java; that would have been terrible (Python is more like lisp than java which is why Hy is tenable whatsoever). In particular, CL style macros with out some first-class concept of symbols are a footgun[1], but I'm sure I could find other issues. Another item that can't be ignored is that Hy selected the clojure syntax, which is going to be a turn-off for anyone coming from Lisp or Scheme. Finally, there are hundreds of thin s-expression layers on top of existing languages, and most of them are terrible; Hy has done nothing to stand out from this crowd, so most lispers will not look closely at it. 1: See https://youtu.be/cPNkH-7PRTk?t=4786 https://youtu.be/cPNkH-7PRTk?t=4786 for Rich's discussion of how he managed this issue in Clojure.

kakw 8y ago

I recommend roswell https://github.com/roswell/roswell https://github.com/roswell/roswell .

kimi 8y ago

I use scripting extensively in Clojure, both on the JVM and on Planck (JS). You have the choice of slower startup times but a ton of libraries to do anything you want, versus short start-up times but less libraries, using the same language. I use https://github.com/l3nz/cli-matic https://github.com/l3nz/cli-matic as a wrapper and it works on both.

zebraman 8y ago
Using modern Clojure's clj is so much easier than the CL procedure.
- eggy 8y ago
  Until you have to deal with unintelligible Java stack traces.
  - islon 8y ago
    They are intelligible if you know Java and Clojure 1.10 improved stack traces.
    - TeMPOraL 8y ago
      Yup. Common Lisp stack traces tend to be no better, anyway, at least with SBCL and CCL. Up to 90% of a trace you see in the inspector can be compiler/runtime infrastructure. Given that macros don't show up, only their expansions, the backtrace can be full of stuff like SB-KERNEL:%FUNCTION-THAT-I-DONT-RECOGNIZE-BUT-MACROEXPANDED-FROM-SOMETHING-I-KNOW. Like in Java, most of the time the top few lines are the ones related to your code, but just like in Java, there are exceptions, in which the source of the problem ends up in the middle of the backtrace, surrounded by infrastructure frames.
      - deleted 8y ago
        [deleted]