6 ms·

Lisp (Common Lisp) beats all the other dynamic languages by a considerable margin. This is why I am developing Clasp - a Common Lisp implementation based on LLV

Lisp (Common Lisp) beats all the other dynamic languages by a considerable margin. This is why I am developing Clasp - a Common Lisp implementation based on LLVM that interoperates with C++/C (https://github.com/clasp-developers/clasp.git https://github.com/clasp-developers/clasp.git) for scientific programming. With Clasp, we get the best of multiple worlds. We get a dynamic language (Common Lisp) with automatic memory management and enormous expressive power that can directly use powerful C and C++ libraries. All three of these languages are "long-lived" languages in that code that was written 10 and 20 years ago still works. Performance is really important to me and I have written a lot of code over the past four decades. I won't develop meaningful code in any language that falls below Racket in table 4 because these language implementations are too inefficient. I furthermore want to keep using my code over the years and decades and so I won't develop meaningful code in any language where someone else can break my code by changing the standard. My program "leap" was written 27 years ago in C and it is still being used daily by thousands of computational chemists. But it's really hard to improve leap because the code is brittle largely because of malloc/free-style memory management (brrr). For a compiled, high-performance, standard language with proven lasting power and performance - Common Lisp is the best choice.

agumonkey 6y ago
I'm often surprised how sbcl is 'benchmark' competitive with other native langs (c, ada, pascal)
- drmeister 6y ago
  SBCL is an amazing implementation and it has an amazing compiler. I would argue that it is one of the best compilers around. Javascript compilers in browsers are pretty impressive - but given the far fewer resources that SBCL development has had, SBCL is remarkable. It's a fast compiler that generates fast code. I would be using it (and do for some applications) if I didn't also need my own large C++ computational chemistry code written over decades.
  - agumonkey 6y ago
    yes sbcl is "only" a tiny impl. Makes me wonder how far franz or other can be .. you know how chez-scheme was leagues above the other implementations while nobody knew until it was open sourced
    - pfdietz 6y ago
      The smallness of SBCL helps speed its continued development. Compiling it from scratch on my laptop and running the smoke tests takes about 1 minute.
lliamander 6y ago
Depending upon how you define "dynamic" Java wins by a small margin. If we treat "dynamic" as a spectrum though, Lisp still gets you a lot more dynamism for a relatively small increase in power consumption (and a somewhat larger gap in performance).
- drmeister 6y ago
  Hey - props to the developers of Java for bringing automatic memory management into the mainstream. But it's not quite what I'm looking for when I want to do exploratory programming.
  - lliamander 6y ago
    Agreed - on both fronts.
  - javajosh 6y ago
    Indeed, the lack of a real "REPL" experience is quite limiting with the Java language (you can kinda sorta simulate it with bytecode manipulation, but please don't). However, you can get the full REPL experience on the JVM, along with a gc, if you run Clojure! (Which is itself a bit of an oddball language, since it's dynamic code, immutable data Lisp written for a vm that likes static code and mutable data...)
    - jerven 6y ago
      Q? did you ever try the jShell that comes with java11+. Curious what a REPL power user thinks about it.
      - javajosh 6y ago
        Haven't used it, but I can't imagine it would be great. Java's issue is that you can't do anything outside of a class. So there really is no outside context in which a REPL makes conceptual sense.
7thaccount 6y ago
Didn't you do a talk on YouTube? I recall watching it when you went over the computational requirements of doing something with molecules maybe? I don't work in that field, but remember it being interesting enough to watch the full thing.
- drmeister 6y ago
  I do have a couple of talks on YouTube and thank you (the most recent one: https://www.youtube.com/watch?v=mbdXeRBbgDM https://www.youtube.com/watch?v=mbdXeRBbgDM). Yes - we are building molecules using our code. I have started a company and we are several months in to developing a new technology for warding off future pandemics (maybe even doing something about the current one). We literally started compiling the first molecules using an application I implemented in Cando (Clasp + computational chemistry) yesterday.
  - 1980phipsi 6y ago
    It would be good to make the slides available.
  - reitzensteinm 6y ago
    Your Google talk was one of my top 10 ever, it blew me away. It's not uncommon in the tech industry to build some mad scientist solution - pardon the expression - because we're trying to do something pedestrian but have painted ourselves in to a corner, eg HipHop VM. Doing it to help with cutting edge science is genuinely exciting. https://www.youtube.com/watch?v=0rSMt1pAlbE https://www.youtube.com/watch?v=0rSMt1pAlbE
    - drmeister 6y ago
      Thank you!
  - 7thaccount 6y ago
    That is really cool and I feel your pain. I've mostly been dissapointed in numerical computing stacks. You've got C/C++/Fortran that are blazing fast, but very cumbersome to use. They are nice in that you can distribute an executable. Python and Julia are good at finding a sweet spot where your scripting language is fast and user friendly, but distribution is a pain I guess unless you have a SAAS product. Matlab, Mathematica, GNU Octave, Scilab...etc have their own problems like cost or performance and also make distribution painful. I looked at SBCL, but didn't see good ways to use existing numerical libraries. I'll have to look at Clasp again.
jedbrown 6y ago
Is better C++ interop the motivation for Clasp over Julia (which has lispy roots and CLOS-like open multimethods, though not lispy syntax)? Julia is conspicuously missing in the benchmark, but should do pretty well if you don't include JIT/start-up time.
- drmeister 6y ago
  If the question is why not use Julia? The answer is - for several reasons. I started implementing Clasp before Julia was a thing. The Julia language keeps changing and it doesn't have a standard like Common Lisp, C++ and C do. I need tight C++ interoperation and Clasp does C++ interoperation like no other language I've see. Clasp uses C++ exception handling and calling conventions and this allows us to compile and link C++ code with Common Lisp - it's all LLVM under the hood. Clasp stack frames and C++ stack frames are interleaved and try/catch and RAII and Common Lisp unwind-protect and dynamic variable binding work perfectly between C++ and Common Lisp code.
6keZbCECT2uB 6y ago
Would it have been feasible to improve the FFI of existing common lisp implementations? The last talk of yours that I watched showed clasp to be slower than common lisp relative to C.
- drmeister 6y ago
  There was a time when I asked myself that every day (sigh). But I don't think so. The details involved in interoperating with C++ are so intricate and involved. I don't think I could have gotten to the same place. Meanwhile - we are improving our compiler performance and the performance of generated code. Maybe in the far future I'll start a couple of ancestor simulations and do the experiment...
  - Const-me 6y ago
    > The details involved in interoperating with C++ are so intricate and involved. Yes in general, but you can pick a subset useful enough in practice. Take a look: https://github.com/Const-me/ComLightInterop https://github.com/Const-me/ComLightInterop
sk2020 6y ago
Given that most of the computationally expensive work in Python is usually offloaded to C extensions (and JIT compilation is an option with Python), is the cost savings in electricity that significant? A desktop PC probably only costs about 10 cents an hour to run at most. Perhaps your interest in CL is because you can recall using it in its prime. Nostalgia Or just novelty is certainly a valid argument.
- Guthur 6y ago
  I remember when the OP came into the CL community and it was not long after myself, and we both came long after is hey day. I dare say that not many in the community are from some bygone time. The fact is CL has by far and away a more sophisticated runtime environment than the vast majority of dynamic languages, with small talk being a stand out exception. Some of the language aesthetics have not aged well, not talking parenthesis but rather the hideously long symbol names.
  - drmeister 6y ago
    Agreed! There are a couple of really clever xkcd comics about this (google: xkcd lisp). And I also would rather use PATH than default-pathname-defaults and logical pathnames are, uh, crufty. But I see these as minor blemishes on what I think is as close to a perfect programming language as I have seen. Clasp is an implementation of Common Lisp and any differences between what it does and what the standard says is a bug in Clasp that we need to fix. But Cando, Clasp+computational chemistry code, is a superset of Common Lisp and we are adding things to make life a bit more convenient. We even added optional infix arithmetic as a standard part of Cando (I know! I'm going to burn in hell between the ninth nested set of parentheses as a heretic).
    - scns 6y ago
      The last sentence painted a big smile on my face
- drmeister 6y ago
  We are doing computational chemistry, simulating molecular structure and designing molecules, and we want to use thousands of cores and get as much performance as possible. If my AWS bills are any measure - then yes - the cost savings in electricity and computing resources are very significant. Also, developing and maintaining Python/C++ bindings for complex libraries is very painful and frustrating. I wrote Python bindings for years using boost::python and earlier Swig and keeping bindings working and dealing with the different memory management approaches of Python and C++... bleh - it's a nightmare. At the same time Python changed from version 2 to 3.x and libraries I depended on and my own Python code was being broken and becoming outdated in ways that I had no control over. It was like trying to build a house out of sand. I've only been using Common Lisp for the past 6 years - after three decades of writing in other languages including Basic, Pascal, Smalltalk, C, Fortran, Python, PHP, Forth, Prolog... Common Lisp feels great, it feels powerful and every function I write I know will compile and run in 20 years. Common Lisp has real macros (programs that write programs! implemented in one language), dynamic variables, generic functions, the Common Lisp Object System, conditions and restarts... There are many features that haven't made it into other languages. Common Lisp makes programming interesting again.
eikenberry 6y ago
Have you had a look at Pliant? It is basically a Lisp with a modified syntax and has a really nice C FFI implementation. I wrote an Ncurses wrapper for it a decade or two back and remember it being one of the easiest FFI/wrappers I'd ever done. http://www.fullpliant.org/ http://www.fullpliant.org/
- drmeister 6y ago
  No, I just found out about Pliant from your post - but it doesn't make sense for developing large, stable codebases because it's not a standard language. (Sorry Pliant developers - I love your can-do attitude and I'd love to buy you a beer or a coffee sometime and talk about Sisyphean task management.) But Pliant is a reference implementation of a custom language. Programming language design is really, really hard - I wouldn't dare try and so I chose to go with a language that had literally hundreds of person years of design and testing underpinning it. Regarding FFI's - my approach is the same as the very clever pybind11, luabind and the older boost::python libraries. It works by using C++ template programming and letting the C++ compiler do the heavy lifting of generating wrappers at compile time. I recently updated our binding library to use C++17 std::apply with tuples. Freakin' finally! C++ has an apply construct that can be applied to a heterogenous list of objects - wow - lisp has only had it for 50 years! My point is that only recently has C++ developed the introspective capabilities to implement really powerful FFI's. Also - you have to use C++ exception handling for stack unwinding or you will break C++ RAII all the time.
  - pulse7 6y ago
    "because it's not a standard language" Every language was a "non-standard language" at it's beginning... Including C, C++ and Lisp...
    - TeMPOraL 6y ago
      What 'drmeister means is that CL as an actual standard, as in ANSI standard. Ditto for C++ (an ISO standard). This means the language has a clear target for implementations to conform to, one that isn't vulnerable to "moving fast and breaking things".
Bancakes 6y ago
Why would I use Clasp and not haskell?
- higerordermap 6y ago
  Why would I eat and not drink?

agumonkey 6y ago

I'm often surprised how sbcl is 'benchmark' competitive with other native langs (c, ada, pascal)

drmeister 6y ago
SBCL is an amazing implementation and it has an amazing compiler. I would argue that it is one of the best compilers around. Javascript compilers in browsers are pretty impressive - but given the far fewer resources that SBCL development has had, SBCL is remarkable. It's a fast compiler that generates fast code. I would be using it (and do for some applications) if I didn't also need my own large C++ computational chemistry code written over decades.
- agumonkey 6y ago
  yes sbcl is "only" a tiny impl. Makes me wonder how far franz or other can be .. you know how chez-scheme was leagues above the other implementations while nobody knew until it was open sourced
  - pfdietz 6y ago
    The smallness of SBCL helps speed its continued development. Compiling it from scratch on my laptop and running the smoke tests takes about 1 minute.

lliamander 6y ago

Depending upon how you define "dynamic" Java wins by a small margin. If we treat "dynamic" as a spectrum though, Lisp still gets you a lot more dynamism for a relatively small increase in power consumption (and a somewhat larger gap in performance).

drmeister 6y ago
Hey - props to the developers of Java for bringing automatic memory management into the mainstream. But it's not quite what I'm looking for when I want to do exploratory programming.
- lliamander 6y ago
  Agreed - on both fronts.
- javajosh 6y ago
  Indeed, the lack of a real "REPL" experience is quite limiting with the Java language (you can kinda sorta simulate it with bytecode manipulation, but please don't). However, you can get the full REPL experience on the JVM, along with a gc, if you run Clojure! (Which is itself a bit of an oddball language, since it's dynamic code, immutable data Lisp written for a vm that likes static code and mutable data...)
  - jerven 6y ago
    Q? did you ever try the jShell that comes with java11+. Curious what a REPL power user thinks about it.
    - javajosh 6y ago
      Haven't used it, but I can't imagine it would be great. Java's issue is that you can't do anything outside of a class. So there really is no outside context in which a REPL makes conceptual sense.

7thaccount 6y ago

Didn't you do a talk on YouTube? I recall watching it when you went over the computational requirements of doing something with molecules maybe? I don't work in that field, but remember it being interesting enough to watch the full thing.

drmeister 6y ago
I do have a couple of talks on YouTube and thank you (the most recent one: https://www.youtube.com/watch?v=mbdXeRBbgDM https://www.youtube.com/watch?v=mbdXeRBbgDM). Yes - we are building molecules using our code. I have started a company and we are several months in to developing a new technology for warding off future pandemics (maybe even doing something about the current one). We literally started compiling the first molecules using an application I implemented in Cando (Clasp + computational chemistry) yesterday.
- 1980phipsi 6y ago
  It would be good to make the slides available.
- reitzensteinm 6y ago
  Your Google talk was one of my top 10 ever, it blew me away. It's not uncommon in the tech industry to build some mad scientist solution - pardon the expression - because we're trying to do something pedestrian but have painted ourselves in to a corner, eg HipHop VM. Doing it to help with cutting edge science is genuinely exciting. https://www.youtube.com/watch?v=0rSMt1pAlbE https://www.youtube.com/watch?v=0rSMt1pAlbE
  - drmeister 6y ago
    Thank you!
- 7thaccount 6y ago
  That is really cool and I feel your pain. I've mostly been dissapointed in numerical computing stacks. You've got C/C++/Fortran that are blazing fast, but very cumbersome to use. They are nice in that you can distribute an executable. Python and Julia are good at finding a sweet spot where your scripting language is fast and user friendly, but distribution is a pain I guess unless you have a SAAS product. Matlab, Mathematica, GNU Octave, Scilab...etc have their own problems like cost or performance and also make distribution painful. I looked at SBCL, but didn't see good ways to use existing numerical libraries. I'll have to look at Clasp again.

jedbrown 6y ago

Is better C++ interop the motivation for Clasp over Julia (which has lispy roots and CLOS-like open multimethods, though not lispy syntax)? Julia is conspicuously missing in the benchmark, but should do pretty well if you don't include JIT/start-up time.

drmeister 6y ago
If the question is why not use Julia? The answer is - for several reasons. I started implementing Clasp before Julia was a thing. The Julia language keeps changing and it doesn't have a standard like Common Lisp, C++ and C do. I need tight C++ interoperation and Clasp does C++ interoperation like no other language I've see. Clasp uses C++ exception handling and calling conventions and this allows us to compile and link C++ code with Common Lisp - it's all LLVM under the hood. Clasp stack frames and C++ stack frames are interleaved and try/catch and RAII and Common Lisp unwind-protect and dynamic variable binding work perfectly between C++ and Common Lisp code.

6keZbCECT2uB 6y ago

Would it have been feasible to improve the FFI of existing common lisp implementations? The last talk of yours that I watched showed clasp to be slower than common lisp relative to C.

drmeister 6y ago
There was a time when I asked myself that every day (sigh). But I don't think so. The details involved in interoperating with C++ are so intricate and involved. I don't think I could have gotten to the same place. Meanwhile - we are improving our compiler performance and the performance of generated code. Maybe in the far future I'll start a couple of ancestor simulations and do the experiment...
- Const-me 6y ago
  > The details involved in interoperating with C++ are so intricate and involved. Yes in general, but you can pick a subset useful enough in practice. Take a look: https://github.com/Const-me/ComLightInterop https://github.com/Const-me/ComLightInterop

sk2020 6y ago

Given that most of the computationally expensive work in Python is usually offloaded to C extensions (and JIT compilation is an option with Python), is the cost savings in electricity that significant? A desktop PC probably only costs about 10 cents an hour to run at most. Perhaps your interest in CL is because you can recall using it in its prime. Nostalgia Or just novelty is certainly a valid argument.

Guthur 6y ago
I remember when the OP came into the CL community and it was not long after myself, and we both came long after is hey day. I dare say that not many in the community are from some bygone time. The fact is CL has by far and away a more sophisticated runtime environment than the vast majority of dynamic languages, with small talk being a stand out exception. Some of the language aesthetics have not aged well, not talking parenthesis but rather the hideously long symbol names.
- drmeister 6y ago
  Agreed! There are a couple of really clever xkcd comics about this (google: xkcd lisp). And I also would rather use PATH than default-pathname-defaults and logical pathnames are, uh, crufty. But I see these as minor blemishes on what I think is as close to a perfect programming language as I have seen. Clasp is an implementation of Common Lisp and any differences between what it does and what the standard says is a bug in Clasp that we need to fix. But Cando, Clasp+computational chemistry code, is a superset of Common Lisp and we are adding things to make life a bit more convenient. We even added optional infix arithmetic as a standard part of Cando (I know! I'm going to burn in hell between the ninth nested set of parentheses as a heretic).
  - scns 6y ago
    The last sentence painted a big smile on my face
drmeister 6y ago
We are doing computational chemistry, simulating molecular structure and designing molecules, and we want to use thousands of cores and get as much performance as possible. If my AWS bills are any measure - then yes - the cost savings in electricity and computing resources are very significant. Also, developing and maintaining Python/C++ bindings for complex libraries is very painful and frustrating. I wrote Python bindings for years using boost::python and earlier Swig and keeping bindings working and dealing with the different memory management approaches of Python and C++... bleh - it's a nightmare. At the same time Python changed from version 2 to 3.x and libraries I depended on and my own Python code was being broken and becoming outdated in ways that I had no control over. It was like trying to build a house out of sand. I've only been using Common Lisp for the past 6 years - after three decades of writing in other languages including Basic, Pascal, Smalltalk, C, Fortran, Python, PHP, Forth, Prolog... Common Lisp feels great, it feels powerful and every function I write I know will compile and run in 20 years. Common Lisp has real macros (programs that write programs! implemented in one language), dynamic variables, generic functions, the Common Lisp Object System, conditions and restarts... There are many features that haven't made it into other languages. Common Lisp makes programming interesting again.

eikenberry 6y ago

Have you had a look at Pliant? It is basically a Lisp with a modified syntax and has a really nice C FFI implementation. I wrote an Ncurses wrapper for it a decade or two back and remember it being one of the easiest FFI/wrappers I'd ever done. http://www.fullpliant.org/ http://www.fullpliant.org/

drmeister 6y ago
No, I just found out about Pliant from your post - but it doesn't make sense for developing large, stable codebases because it's not a standard language. (Sorry Pliant developers - I love your can-do attitude and I'd love to buy you a beer or a coffee sometime and talk about Sisyphean task management.) But Pliant is a reference implementation of a custom language. Programming language design is really, really hard - I wouldn't dare try and so I chose to go with a language that had literally hundreds of person years of design and testing underpinning it. Regarding FFI's - my approach is the same as the very clever pybind11, luabind and the older boost::python libraries. It works by using C++ template programming and letting the C++ compiler do the heavy lifting of generating wrappers at compile time. I recently updated our binding library to use C++17 std::apply with tuples. Freakin' finally! C++ has an apply construct that can be applied to a heterogenous list of objects - wow - lisp has only had it for 50 years! My point is that only recently has C++ developed the introspective capabilities to implement really powerful FFI's. Also - you have to use C++ exception handling for stack unwinding or you will break C++ RAII all the time.
- pulse7 6y ago
  "because it's not a standard language" Every language was a "non-standard language" at it's beginning... Including C, C++ and Lisp...
  - TeMPOraL 6y ago
    What 'drmeister means is that CL as an actual standard, as in ANSI standard. Ditto for C++ (an ISO standard). This means the language has a clear target for implementations to conform to, one that isn't vulnerable to "moving fast and breaking things".

Bancakes 6y ago

Why would I use Clasp and not haskell?

higerordermap 6y ago
Why would I eat and not drink?