10 ms·

ClojureC, a compiler for Clojure that targets C as a backend

144 points by terhechte 13y ago

pjmlp 13y ago
This is quite nice, on the other hand, one could just use one of the many Lisp compilers existing since years.
timbaldridge 13y ago
sadly, this doesn't seem to support any sort of multithreading. Even something as simple as swap! isn't thread-safe in this implementation. So that kills one of the main reasons to use Clojure in the first place.
- swannodette 13y ago
  Lack of multithreading seems like a result of the implementation being heavily based on ClojureScript ;) (it's actually pretty cool to see how reusable core.cljs is IMO). I imagine ClojureC will have its uses like ClojureScript does when the JVM is not an option.
- terhechte 13y ago
  It is quite interesting though as soon as you extend it with calls to C libraries. Like if you want to use Clojure for libavcodec, or many others.
- uvtc 13y ago
  > sadly, this doesn't seem to support any sort of multithreading Maybe multithreading is just not yet implemented? > So that kills one of the main reasons to use Clojure in the first place. I could see ClojureC being very useful for when you want: * small footprint * easy C-library interop * fast start-up time and where you don't necessarily need multithreading. (BTW, I'd be curious to hear what you think might be the differences in use cases between ClojureC and mjolnir/clojure-metal.)
densh 13y ago
Why C but not llvm? Structured code generation is always better than string-based one.
- pat_punnu 13y ago
  Can you show a proof for that claim? I believe it to be nonsense.
  - bratsche 13y ago
    I'm not sure if this is what the previous poster was talking about, but clang has some APIs that let you get access to the AST pretty easily. It's been a really long time since I've looked at it, and at the time I don't think it was exposed as a library API for general consumption. But for example: https://github.com/bratsche/clang/blob/gtkrewriter/tools/clang-cc/RewriteGtk.cpp https://github.com/bratsche/clang/blob/gtkrewriter/tools/cla...
    - pat_punnu 13y ago
      What is his definition of better? It sounds like he thinks it's entirely objective, so he should be able to express it clearly and logically. Does he think that it's technically more powerful? Again he should be able to prove that if that's the case. Otherwise he's just giving a shitty opinion, and should say that. I think the claim is nonsense because with inline assembler there is nothing that you cannot express in C that you can with LLVM. So the decision between the two is opinion.
      - bratsche 13y ago
        I don't know what his definition of anything is. I happen to agree with you. The example I pasted above was just from my only experience working with a C AST. In that case I wanted to input C code and then write out a transformed version of it.
      - coldtea 13y ago
        >Does he think that it's technically more powerful? Again he should be able to prove that if that's the case. Otherwise he's just giving a shitty opinion, and should say that. It's not like it's some controversial opinion what he said -- it's both self evident and common place. It's you who offers the more controversial opinion (and in a rude way, to top). >I think the claim is nonsense because with inline assembler there is nothing that you cannot express in C that you can with LLVM. So the decision between the two is opinion. It's not about "expression", and nobody argued that you can express more in LLVM. This is missing the point by miles! It's about having more structure and less of an ad-hoc pipeline, which helps with better tooling, error prevention, etc. (Not only what you wrote is wrong, but even if the original argument was about expression, your opinion would still be wrong. Two things offering equivalent expressive power, does not mean that they are just as good to use in practice at all. Might as well ask "why invent new languages, when assembly can express everything"). The only benefit to using C for something like this is portability, which is something else altogether.
akkartik 13y ago
"Before you can run anything make sure you have GLib 2 and the Boehm-Demers-Weiser garbage collector installed." Wow, that's a pretty skimpy list of dependencies. But.. "Make sure you're using Leiningen 2." ..argh, installing that on ubuntu that requires 110 packages. All that just for a build system?
- zhemao 13y ago
  Wait, are you trying to install it as an APT package? It's probably easier to just download the script and install it that way. If you do that, it will install all the libraries to a folder in your home directory.
- Tuna-Fish 13y ago
  It's easier to get Lein as standalone. The Ubuntu package depends on half the build infrastructure of java, using really minimal parts of it here and there. If you just get the shell script and stuff it in your path, it can self-install a 10MB jar that has all it needs.
- uvtc 13y ago
  > ..argh, installing that on ubuntu that requires 110 packages. All that just for a build system? Don't install lein using the OS packaging system (apt, rpm, yum, etc.). Instead, just grab the `lein` script (linked to from http://leiningen.org/ http://leiningen.org/ ), put it into your ~/bin, set it executable, and you're all set.
  - michaelochurch 13y ago
    This. It's a lot easier to do it that way.
iso8859-1 13y ago
How does this compare to Chicken Scheme?
mullr 13y ago
This is neat. What worries me about this and similar efforts (like https://github.com/takeoutweight/clojure-scheme https://github.com/takeoutweight/clojure-scheme) is that clojure's standard library design assumes that the underlying runtime will be do some kind of polymorphic method inlining. For example: the sequence library is all defined in terms of ISeq, which basically requires a "first" and "rest" to be defined for the data structure in question. These are polymorphic: there are different implementations of these for different data structures (list, vector, map, etc). So a dispatch step is required to choose the right one. In clojure-jvm, this is implemented using a java interface; this means the jvm will inline calls to said methods when they're being used in a tight loop. And if you use the standard library, calls to 'first' and 'rest' are going to be inside nearly all of your inner loops. Compare this to a normal lisp or scheme: 'first' and 'rest' (or 'car' and 'cdr', whatever) are monomorphic. They only work on the linked-list data structure. So compiling these directly down to C functions makes perfect sense and incurs no performance penalty. So in summary: clojure assumes theres a really smart JIT which is helping things along. This means it's not as suitable for alternate compilation targets as you might want it to be. I wonder if there's something clever you could do here. Vtables could be reordered based on expected usage, certainly. Clojure can already do some measure of type inference, so this could be used for AOT inlining when it's available. Even if it's not, perhaps several versions of a call could be speculatively generated based on what the compiler does know already. The normal polymorphic inline caching technique could perhaps be abused to apply here. But it's hard to see how any of this can work in absence of a profile or heavy hinting. (not a compiler writer, just interested in the problem)
- pjmlp 13y ago
  You can workaround it by making use of PGO (Profile Guided Optimization). This requires you to execute the application with profiling turned on. Then you compile the application a second time with the additional input of the profile results, this way the optimizer gets additional information that helps it to do decisions similar to a JIT. It is all a matter of adding support for this.
- nimrody 13y ago
  Perhaps you could overcome some of these programs with whole-program-optimization. E.g., the "Stalin" Scheme compiler. It might be interesting to translate Clojure into a subset of Scheme that is widely supported -- and use one of the mature Scheme -> C compilers (Gambit, Chicken, Bigloo) to generate the target executable.
jgalt212 13y ago
As someone who doesn't use Clojure, but watches it fairly closely, I'd say the least appealing part of Clojure is its reliance on the JVM. As such, I'd say efforts such as these are greatly welcomed.
- pjmlp 13y ago
  > As someone who doesn't use Clojure, but watches it fairly closely, I'd say the least appealing part of Clojure is its reliance on the JVM. Like it or not, this is what made Clojure successful in the enterprise, at least when compared against other Lisps.
  - lispm 13y ago
    Is that based on numbers or a guess? How many Clojure applications are there in comparison to Lisp or Scheme?
    - pjmlp 13y ago
      Based on guess. I see lots of Java shops now having Clojure code and talking about it at JUGs and how it enables their business. You just need to have a look at InfoQ, Skills That Matter, Devoxx or Jax for ongoing talks.
- cgag 13y ago
  That's how I felt about it before I started using it. If it's more than just startup time holding you back, I'd give it a shot.
saosebastiao 13y ago
So if I wanted to write CLI applications in Clojure, is this my best bet? Cause the JVM is about the least suitable platform I have ever worked with for CLI apps...which is most of what I do. I'm constantly in this pickle of wanting to use Clojure but defaulting to Ruby because the JVM is so terrible at it.
- pjmlp 13y ago
  Get one of the many AOT compilers for Java. No more JVM startup time to worry about.
- mambodog 13y ago
  You could also use node.js with Clojurescript, if you're willing to accept the compromises of Clojurescript.
  - saosebastiao 13y ago
    I'm not familiar with the compromises of clojurescript nor running node.js as a CLI platform. Got any links that can help explain? Can it be compiled to a static executable?
- bbq 13y ago
  If the startup slowness is your problem, look into Nailgun: http://www.martiansoftware.com/nailgun/ http://www.martiansoftware.com/nailgun/ If that's not your problem with the JVM, what is?
  - saosebastiao 13y ago
    Nailgun is an additional dependency. And pretty insecure at that.
    - bbq 13y ago
      Well, that would do it. =)
  - 13y ago
billsix 13y ago
For those interested in Lisp implementations which compile to C, providing cross-platform benefits and a nice FFI, gambit-c and chicken are performant, mature implementations of the Scheme programming language.
- deleted 13y ago
  [deleted]
pootch 13y ago
Im not a lisper and I likely never will be, but I am encouraged to see the potential for Clojure to move away from the JVM. Its this simple really, I want a 100GB heap with no tuning. That will never happen on the JVM, Since Im not a lisper I dont know what the implications for memory management are, but having lived with JVM GC for over a decade Im done with it and hardware has surpassed JVM's ability to keep up. If it was still a 32 bit world I would have a different opinion. But Im not going to distribute tons of code to clusters just to avoid GC pauses, give me 100GB heap or give me death.

144 points by terhechte 13y ago

pjmlp 13y ago

This is quite nice, on the other hand, one could just use one of the many Lisp compilers existing since years.

sadly, this doesn't seem to support any sort of multithreading. Even something as simple as swap! isn't thread-safe in this implementation. So that kills one of the main reasons to use Clojure in the first place.

swannodette 13y ago
Lack of multithreading seems like a result of the implementation being heavily based on ClojureScript ;) (it's actually pretty cool to see how reusable core.cljs is IMO). I imagine ClojureC will have its uses like ClojureScript does when the JVM is not an option.
terhechte 13y ago
It is quite interesting though as soon as you extend it with calls to C libraries. Like if you want to use Clojure for libavcodec, or many others.
uvtc 13y ago
> sadly, this doesn't seem to support any sort of multithreading Maybe multithreading is just not yet implemented? > So that kills one of the main reasons to use Clojure in the first place. I could see ClojureC being very useful for when you want: * small footprint * easy C-library interop * fast start-up time and where you don't necessarily need multithreading. (BTW, I'd be curious to hear what you think might be the differences in use cases between ClojureC and mjolnir/clojure-metal.)

densh 13y ago

Why C but not llvm? Structured code generation is always better than string-based one.

pat_punnu 13y ago
Can you show a proof for that claim? I believe it to be nonsense.
- bratsche 13y ago
  I'm not sure if this is what the previous poster was talking about, but clang has some APIs that let you get access to the AST pretty easily. It's been a really long time since I've looked at it, and at the time I don't think it was exposed as a library API for general consumption. But for example: https://github.com/bratsche/clang/blob/gtkrewriter/tools/clang-cc/RewriteGtk.cpp https://github.com/bratsche/clang/blob/gtkrewriter/tools/cla...
  - pat_punnu 13y ago
    What is his definition of better? It sounds like he thinks it's entirely objective, so he should be able to express it clearly and logically. Does he think that it's technically more powerful? Again he should be able to prove that if that's the case. Otherwise he's just giving a shitty opinion, and should say that. I think the claim is nonsense because with inline assembler there is nothing that you cannot express in C that you can with LLVM. So the decision between the two is opinion.
    - bratsche 13y ago
      I don't know what his definition of anything is. I happen to agree with you. The example I pasted above was just from my only experience working with a C AST. In that case I wanted to input C code and then write out a transformed version of it.
    - coldtea 13y ago
      >Does he think that it's technically more powerful? Again he should be able to prove that if that's the case. Otherwise he's just giving a shitty opinion, and should say that. It's not like it's some controversial opinion what he said -- it's both self evident and common place. It's you who offers the more controversial opinion (and in a rude way, to top). >I think the claim is nonsense because with inline assembler there is nothing that you cannot express in C that you can with LLVM. So the decision between the two is opinion. It's not about "expression", and nobody argued that you can express more in LLVM. This is missing the point by miles! It's about having more structure and less of an ad-hoc pipeline, which helps with better tooling, error prevention, etc. (Not only what you wrote is wrong, but even if the original argument was about expression, your opinion would still be wrong. Two things offering equivalent expressive power, does not mean that they are just as good to use in practice at all. Might as well ask "why invent new languages, when assembly can express everything"). The only benefit to using C for something like this is portability, which is something else altogether.

akkartik 13y ago

"Before you can run anything make sure you have GLib 2 and the Boehm-Demers-Weiser garbage collector installed." Wow, that's a pretty skimpy list of dependencies. But.. "Make sure you're using Leiningen 2." ..argh, installing that on ubuntu that requires 110 packages. All that just for a build system?

zhemao 13y ago
Wait, are you trying to install it as an APT package? It's probably easier to just download the script and install it that way. If you do that, it will install all the libraries to a folder in your home directory.
Tuna-Fish 13y ago
It's easier to get Lein as standalone. The Ubuntu package depends on half the build infrastructure of java, using really minimal parts of it here and there. If you just get the shell script and stuff it in your path, it can self-install a 10MB jar that has all it needs.
uvtc 13y ago
> ..argh, installing that on ubuntu that requires 110 packages. All that just for a build system? Don't install lein using the OS packaging system (apt, rpm, yum, etc.). Instead, just grab the `lein` script (linked to from http://leiningen.org/ http://leiningen.org/ ), put it into your ~/bin, set it executable, and you're all set.
- michaelochurch 13y ago
  This. It's a lot easier to do it that way.

iso8859-1 13y ago

How does this compare to Chicken Scheme?

mullr 13y ago

This is neat. What worries me about this and similar efforts (like https://github.com/takeoutweight/clojure-scheme https://github.com/takeoutweight/clojure-scheme) is that clojure's standard library design assumes that the underlying runtime will be do some kind of polymorphic method inlining. For example: the sequence library is all defined in terms of ISeq, which basically requires a "first" and "rest" to be defined for the data structure in question. These are polymorphic: there are different implementations of these for different data structures (list, vector, map, etc). So a dispatch step is required to choose the right one. In clojure-jvm, this is implemented using a java interface; this means the jvm will inline calls to said methods when they're being used in a tight loop. And if you use the standard library, calls to 'first' and 'rest' are going to be inside nearly all of your inner loops. Compare this to a normal lisp or scheme: 'first' and 'rest' (or 'car' and 'cdr', whatever) are monomorphic. They only work on the linked-list data structure. So compiling these directly down to C functions makes perfect sense and incurs no performance penalty. So in summary: clojure assumes theres a really smart JIT which is helping things along. This means it's not as suitable for alternate compilation targets as you might want it to be. I wonder if there's something clever you could do here. Vtables could be reordered based on expected usage, certainly. Clojure can already do some measure of type inference, so this could be used for AOT inlining when it's available. Even if it's not, perhaps several versions of a call could be speculatively generated based on what the compiler does know already. The normal polymorphic inline caching technique could perhaps be abused to apply here. But it's hard to see how any of this can work in absence of a profile or heavy hinting. (not a compiler writer, just interested in the problem)

pjmlp 13y ago
You can workaround it by making use of PGO (Profile Guided Optimization). This requires you to execute the application with profiling turned on. Then you compile the application a second time with the additional input of the profile results, this way the optimizer gets additional information that helps it to do decisions similar to a JIT. It is all a matter of adding support for this.
nimrody 13y ago
Perhaps you could overcome some of these programs with whole-program-optimization. E.g., the "Stalin" Scheme compiler. It might be interesting to translate Clojure into a subset of Scheme that is widely supported -- and use one of the mature Scheme -> C compilers (Gambit, Chicken, Bigloo) to generate the target executable.

jgalt212 13y ago

As someone who doesn't use Clojure, but watches it fairly closely, I'd say the least appealing part of Clojure is its reliance on the JVM. As such, I'd say efforts such as these are greatly welcomed.

pjmlp 13y ago
> As someone who doesn't use Clojure, but watches it fairly closely, I'd say the least appealing part of Clojure is its reliance on the JVM. Like it or not, this is what made Clojure successful in the enterprise, at least when compared against other Lisps.
- lispm 13y ago
  Is that based on numbers or a guess? How many Clojure applications are there in comparison to Lisp or Scheme?
  - pjmlp 13y ago
    Based on guess. I see lots of Java shops now having Clojure code and talking about it at JUGs and how it enables their business. You just need to have a look at InfoQ, Skills That Matter, Devoxx or Jax for ongoing talks.
cgag 13y ago
That's how I felt about it before I started using it. If it's more than just startup time holding you back, I'd give it a shot.

saosebastiao 13y ago

So if I wanted to write CLI applications in Clojure, is this my best bet? Cause the JVM is about the least suitable platform I have ever worked with for CLI apps...which is most of what I do. I'm constantly in this pickle of wanting to use Clojure but defaulting to Ruby because the JVM is so terrible at it.

pjmlp 13y ago
Get one of the many AOT compilers for Java. No more JVM startup time to worry about.
mambodog 13y ago
You could also use node.js with Clojurescript, if you're willing to accept the compromises of Clojurescript.
- saosebastiao 13y ago
  I'm not familiar with the compromises of clojurescript nor running node.js as a CLI platform. Got any links that can help explain? Can it be compiled to a static executable?
bbq 13y ago
If the startup slowness is your problem, look into Nailgun: http://www.martiansoftware.com/nailgun/ http://www.martiansoftware.com/nailgun/ If that's not your problem with the JVM, what is?
- saosebastiao 13y ago
  Nailgun is an additional dependency. And pretty insecure at that.
  - bbq 13y ago
    Well, that would do it. =)
- 13y ago

billsix 13y ago

For those interested in Lisp implementations which compile to C, providing cross-platform benefits and a nice FFI, gambit-c and chicken are performant, mature implementations of the Scheme programming language.

deleted 13y ago
[deleted]

pootch 13y ago

Im not a lisper and I likely never will be, but I am encouraged to see the potential for Clojure to move away from the JVM. Its this simple really, I want a 100GB heap with no tuning. That will never happen on the JVM, Since Im not a lisper I dont know what the implications for memory management are, but having lived with JVM GC for over a decade Im done with it and hardware has surpassed JVM's ability to keep up. If it was still a 32 bit world I would have a different opinion. But Im not going to distribute tons of code to clusters just to avoid GC pauses, give me 100GB heap or give me death.