3 ms·

My group is developing Clasp (github.com/clasp-developers/clasp.git). It's a Common Lisp implementation that interoperates with C++ and uses llvm as the backend

My group is developing Clasp (github.com/clasp-developers/clasp.git). It's a Common Lisp implementation that interoperates with C++ and uses llvm as the backend. It's something you might want to look at because it could change your options in the following ways: (1) Clasp interoperates with C++ - C++ libraries like graphics and physics engines can be easily exposed within Clasp Common Lisp. (See https://github.com/clasp-developers/demo-expose-bullet-physics-engine-to-clasp https://github.com/clasp-developers/demo-expose-bullet-physi...) (2) Clasp uses llvm as the backend - it generates llvm-ir from Common Lisp code and uses link time optimization to link C++ and Common Lisp generated llvm-ir. (3) Clasp uses the Memory Pool System garbage collector, which can be configured to minimize GC pause times to a programmer-defined level. (4) Clasp generates debug info and can be profiled with standard tools like dtrace and C++ functions, Common Lisp functions and GC functions are all profiled against each other. I'm a professor of Chemistry at Temple University - I'm developing Clasp to design molecules and materials - but it could be applied to a lot of other problems. Caveat: Clasp is not yet as performant as Steel Bank Common Lisp, but it uses the open source and highly configurable Cleavir compiler developed by Robert Strandh and we have incorporated inlining and are currently incorporating type inference. If you want to see a little bit more - I have a 25 min talk up on youtube that I presented a couple of weeks ago at the 2018 llvm developers meeting (https://www.youtube.com/watch?v=mbdXeRBbgDM&feature=youtu.be https://www.youtube.com/watch?v=mbdXeRBbgDM&feature=youtu.be)

mthom 8y ago
Wonderful and inspiring work! I wonder if it would be possible to create a similar interop story with the Rust programming language, which also targets LLVM -- ideally without having to redo all the hard work that went into Clasp. I have a Prolog interpreter/compiler I'm developing in Rust right now. One of my aims (eventually) is to compile Prolog to fast native code with it. It would be nice to have convenient access to it from a Common Lisp environment.
- gameswithgo 8y ago
  Rust can already talk C ABI so I think the dream is already reality.
  - armitron 8y ago
    Rust does not have native C++ interop. C ABI is different to what clasp has done (native C++ interop). All widely used Common Lisp implementations have seamless C ABI interop through their FFIs / CFFI [1]. Clasp goes above and beyond that and natively interfaces with C++. LTO works with mixed Common Lisp and C++ code. Exception handling works as expected. You can mix Common Lisp and C++ stack frames. Correct me if I'm wrong but Rust does not have anything equivalent. The language that comes closest is D I think [2]. [1] https://common-lisp.net/project/cffi/ https://common-lisp.net/project/cffi/ [2] https://www.youtube.com/watch?v=IkwaV6k6BmM https://www.youtube.com/watch?v=IkwaV6k6BmM
    - man213 8y ago
      Rust can inline cpp code via rust-cpp crate. I've been playing with it several months ago and successfully used dear-imgui and other stuff in some of my rust tech.
agumonkey 8y ago
Hi, I watch your llvm talk but, correct me if I'm wrong, it seems you didn't really have enough time to cover all you wanted. Do you think you'll have another conf to do it fully ? or maybe find time to do it on another video platform ? oh also, someone on your talk main thread suggested to use more GPU (https://news.ycombinator.com/item?id=18707771 https://news.ycombinator.com/item?id=18707771 <= get ready for some harsh adjectives). I wonder if you're already doing so or planned ? Maybe it's just a backend parameter (llvm-to-gpu-cluster..)
- drmeister 8y ago
  Yes, there is never enough time to cover all that I want. There will be more talks in the future and more as Clasp gets more exposure.
  - agumonkey 8y ago
    Well, I'm super eager to see more clasp/cando videos on frontpages.
gameswithgo 8y ago
Does clasp still support all the metaprogramming features of lisp while using llvm, and how does it do it?
- armitron 8y ago
  Given that clasp is a Common Lisp - that passes 97%+ of CL ANSI tests on par with other CL implementations like SBCL - then answer is obviously yes. You need to go into more detail about why you think llvm would make a difference. Common Lisp is usually compiled to native code and the compiler is - unlike most other languages that compile to native code - exposed and available at runtime. When you program interactively in a natively-compiled Common Lisp implementation like clasp or SBCL or ClozureCL, your code is compiled to native code on the spot. You can even customize the code generation via compiler macros. LLVM is just another compiler backend that can slot in so in that regard, conceptually it's seamless.
- cracauer 8y ago
  Well, there are a lot of those, they need to be looked at separately. What I (and the QPX school if I might use the term) use is compile-time computing, aka a programmable programming language. LLVM never gets to see this, by the time you pass it to the compiler it is dumb code. QPX after macroexpansion looked like C. When it comes to executing code that only becomes available at run time (repl, eval, from a string, from a file etc) you generally just kick off the compiler, which is that Clasp and LLVM do right now. You can also use the interpreter which again both Clasp and SBCL have. When it comes to the dynamic nature of CLOS method invocation, that is generally accelerated with special dispatch mechanism that live in the compilers. You can implement CLOS without any compiler support, but teaching the compiler about method dispatch makes it faster. The Lisp compilers emit abstract assembly code for LLVM or the last stage of SBCL. When talking about dynamic typing the same applies. The compilers that do Lisp emit abstract machine language, pretty much the same way for Clasp and SBCL. To Clarify, Clasp uses various LLVM compilation targets including LLVM's "JIT" for runtime known code. I am not sure I'd call it a JIT because it doesn't compile functions when they are used, but they are compiled into memory (as opposed to into object files). Hope this helps.

mthom 8y ago

Wonderful and inspiring work! I wonder if it would be possible to create a similar interop story with the Rust programming language, which also targets LLVM -- ideally without having to redo all the hard work that went into Clasp. I have a Prolog interpreter/compiler I'm developing in Rust right now. One of my aims (eventually) is to compile Prolog to fast native code with it. It would be nice to have convenient access to it from a Common Lisp environment.

gameswithgo 8y ago
Rust can already talk C ABI so I think the dream is already reality.
- armitron 8y ago
  Rust does not have native C++ interop. C ABI is different to what clasp has done (native C++ interop). All widely used Common Lisp implementations have seamless C ABI interop through their FFIs / CFFI [1]. Clasp goes above and beyond that and natively interfaces with C++. LTO works with mixed Common Lisp and C++ code. Exception handling works as expected. You can mix Common Lisp and C++ stack frames. Correct me if I'm wrong but Rust does not have anything equivalent. The language that comes closest is D I think [2]. [1] https://common-lisp.net/project/cffi/ https://common-lisp.net/project/cffi/ [2] https://www.youtube.com/watch?v=IkwaV6k6BmM https://www.youtube.com/watch?v=IkwaV6k6BmM
  - man213 8y ago
    Rust can inline cpp code via rust-cpp crate. I've been playing with it several months ago and successfully used dear-imgui and other stuff in some of my rust tech.

agumonkey 8y ago

Hi, I watch your llvm talk but, correct me if I'm wrong, it seems you didn't really have enough time to cover all you wanted. Do you think you'll have another conf to do it fully ? or maybe find time to do it on another video platform ? oh also, someone on your talk main thread suggested to use more GPU (https://news.ycombinator.com/item?id=18707771 https://news.ycombinator.com/item?id=18707771 <= get ready for some harsh adjectives). I wonder if you're already doing so or planned ? Maybe it's just a backend parameter (llvm-to-gpu-cluster..)

drmeister 8y ago
Yes, there is never enough time to cover all that I want. There will be more talks in the future and more as Clasp gets more exposure.
- agumonkey 8y ago
  Well, I'm super eager to see more clasp/cando videos on frontpages.

gameswithgo 8y ago

Does clasp still support all the metaprogramming features of lisp while using llvm, and how does it do it?

armitron 8y ago
Given that clasp is a Common Lisp - that passes 97%+ of CL ANSI tests on par with other CL implementations like SBCL - then answer is obviously yes. You need to go into more detail about why you think llvm would make a difference. Common Lisp is usually compiled to native code and the compiler is - unlike most other languages that compile to native code - exposed and available at runtime. When you program interactively in a natively-compiled Common Lisp implementation like clasp or SBCL or ClozureCL, your code is compiled to native code on the spot. You can even customize the code generation via compiler macros. LLVM is just another compiler backend that can slot in so in that regard, conceptually it's seamless.
cracauer 8y ago
Well, there are a lot of those, they need to be looked at separately. What I (and the QPX school if I might use the term) use is compile-time computing, aka a programmable programming language. LLVM never gets to see this, by the time you pass it to the compiler it is dumb code. QPX after macroexpansion looked like C. When it comes to executing code that only becomes available at run time (repl, eval, from a string, from a file etc) you generally just kick off the compiler, which is that Clasp and LLVM do right now. You can also use the interpreter which again both Clasp and SBCL have. When it comes to the dynamic nature of CLOS method invocation, that is generally accelerated with special dispatch mechanism that live in the compilers. You can implement CLOS without any compiler support, but teaching the compiler about method dispatch makes it faster. The Lisp compilers emit abstract assembly code for LLVM or the last stage of SBCL. When talking about dynamic typing the same applies. The compilers that do Lisp emit abstract machine language, pretty much the same way for Clasp and SBCL. To Clarify, Clasp uses various LLVM compilation targets including LLVM's "JIT" for runtime known code. I am not sure I'd call it a JIT because it doesn't compile functions when they are used, but they are compiled into memory (as opposed to into object files). Hope this helps.