4 ms·

Here's one idea: why don't we compile every new language into a pretty C code? This way we will be able to use every available C library with a compiler of our

by zzzcpan 12y ago

Here's one idea: why don't we compile every new language into a pretty C code? This way we will be able to use every available C library with a compiler of our choice and any extra C code, necessary for our real world application.

lmm 12y ago
Because if you use C libraries you will inherit their problems (and also their interface is rarely idiomatic in the new language). Usually if you want to use a different language it's because you think that language has advantages over C. E.g. a large part of the point of using OCaml is to avoid the safety problems of C, but that only applies if your libraries aren't written in C. See https://github.com/mirleft/ocaml-tls#why-a-new-tls-implementation https://github.com/mirleft/ocaml-tls#why-a-new-tls-implement... (Almost all languages do offer some "FFI" to call C. But in languages that don't naturally fit, there will often be significant overhead to e.g. aligning memory management so that the same memory doesn't get freed twice, as might happen when a garbage-collected language called a C library)
- chas 12y ago
  I think zzzcpan is suggesting to compile from the new toy language to C and then use a C compiler to get native code. This doesn't preclude only using libraries written in the new language. CHICKEN is a Scheme compiler that takes this approach.[0] The submitted essay addresses the problems that can result from this sort of compilation style with the question "Will native compilation of a big project result in so much code that there's a global slowdown (something actually true of mid-1990s Erlang to C translators)?" [0]http://www.call-cc.org/ http://www.call-cc.org/
- zzzcpan 12y ago
  That's why you would compile it into C instead of having hand-written C when safety is important and for most part of your program it probably is. But sometimes you absolutely need to be able to write assembly and use existing unsafe C libraries.
  - lmm 12y ago
    If you absolutely need a C library, almost all languages give you a way to do that. But why hobble your compilation strategy for all programs just to support this occasional use case better? You may well want to target a runtime (JVM, .net, Javascript) for which no good C compiler exists. Even if you're solely interested in building native executables, compiling to C would mean throwing away a lot of information (e.g. knowing which values are immutable) that an optimizer could use. Look at Haskell; performance-optimized Haskell performs a lot better than machine-generated C ever would (indeed, it often comes close to the performance of hand-tuned C). (If you're worried about duplicating effort in optimizers, do what many languages do and offer an LLVM frontend. LLVM optimizations are (mostly) language-independent, and LLVM bytecode, while imperfect, often allows a better representation of language semantics than C source would)
    - zzzcpan 12y ago
      I don't think it is so rare, that you could call it occasional. That's the author's point and mine as well. Sometimes I need sqlite, other times libjpeg, giflib, libpng and some simple image processing with at least some low-level code. And it takes a lot of effort to make these things work with other languages on different platforms. Golang tried to make it easier and it still isn't. Optimizers is also one thing I was worried about, LLVM is not the only compiler out there. And also there is friction. Most of the systems already have C compiler installed, if you target it, there is no friction in setting up building environment and so on.
      - lmm 12y ago
        > Golang tried to make it easier and it still isn't. How much of that is because Golang doesn't compile to C, and how much is because the languages just don't fit? E.g. you'd still need a way to mark whether Golang was supposed to garbage-collect a value that had been returned from a C call, or not. > Most of the systems already have C compiler installed, if you target it, there is no friction in setting up building environment and so on. Yes there is, because you still need to build the thing that interprets to C. If anything there's more friction than having a compiler that directly builds native binaries like e.g. ghc, because you need two tools - your translate-to-C tool and your C compiler. "Most systems already have a C compiler" is only really true on *nix; it will be easier for a Mac or Windows user to run your system if you distribute e.g. java bytecode.
- username223 12y ago
  > and also their interface is rarely idiomatic in the new language This. If you just want C interop, SWIG[1] has been around forever, and is pretty good at generating bindings from a C/C++ header with minimal effort. But if you've ever tried it, you'll know that you still have to do most of the work by hand to create an idiomatic interface in your language of choice. [1] http://www.swig.org/ http://www.swig.org/
_wmd 12y ago
Portable C significantly restricts the machine primitives available, e.g. no explicit SIMD, no explicit control over struct/memory layout, no obviously efficient way to represent type information (say, for implementation of a moving GC), no ability to access machine registers (again GC), ...
- zzzcpan 12y ago
  You don't have to write extra code in a portable manner, that's the point. You write it how you need it in a real world. But still benefit from refcounting, nice safe strings, higher-order functions, etc. in main part of your program.
dreadfulgoat 12y ago
If what you want is to use the C library, shoehorning the library APIs into your language is guaranteed to be nastier than just writing in C.
lomnakkus 12y ago
One potential objection I see (aside from the ones raised by others in this thread) is that C isn't actually that great as a compile target. Aliasing analysis is extremely hard in C (thus preventing obvious optimizations), you'd still have to implement your own GC on top of C, C doesn't do tail-call optimization so you'd have to do your own CPS transform anyway... at which point most of the advantages evaporate. (If you're thinking that interfacing with C would be easier: No, the FFIs to C are usually as complex as they are for exactly the right reasons, namely that C's semantics don't match very well with $OTHER_LANGUAGE.) ... at which point you're practically implementing your own VM anyway, so, y'know...
- username223 12y ago
  > C isn't actually that great as a compile target. That's not the point. C is a great common tongue, as its function call and name lookup semantics are simple, and it doesn't insist on being in control of memory, scheduling, etc. Perl and Python are pretty similar, and both talk to C, but calling between Perl and Python is enough of an unholy mess that it's usually easier to just print text over a pipe.
  - lomnakkus 12y ago
    > That's not the point. Maybe not, but did you miss the bit in my post about FFI and GC? Have you actually tried to implement a non-trivial extension to a GC'ed language which needed to interface with C code? EDIT: The point is not so much that it cannot be done -- the point is that it needs a human ("programmer") to do it. Which kind of defeats the point of "C-as-lingua-franca". Which, if you think about it C already kind of is, sadly. Any FFI currently in existence focuses on C already. What more do you want? ;)
    - username223 12y ago
      > did you miss the bit in my post about FFI and GC? No. It's a huge pain, but much less of a pain than calling between two GC'd languages. (I haven't tried calling directly between e.g. Perl and Haskell, but I'd rather gouge my eyes out with a spoon. The handles alone would bury me.) If there were one obvious GC solution, then all GC'd languages would use it, and C would not be as useful. But we've had the JVM and countless other abstractions, yet none has managed to be a higher common denominator than C, so it looks like human labor for the foreseeable future.
desdiv 12y ago
None of the primitive data structures will match up. JVM languages, CLR languages, Rust, Go, et al all have array bound checking, so their arrays won't be compatible with C libraries without a wrapper.
- kibwen 12y ago
  Rust arrays are perfectly compatible with C arrays, they're both just contiguous segments of memory with elements at fixed offsets. You can pass Rust arrays to C functions trivially, and bounds checking has nothing to do with it. I should also note that all correct C code does bounds checking somewhere. The difference with languages that are bounds-checked by default is that they pessimistically assume that you haven't checked the bound manually, and so can potentially do more bounds checking than is strictly necessary if their optimizers aren't capable of hoisting out the checks.
  - NateDad 12y ago
    Ditto for go arrays.
idlewan 12y ago
That's exactly what Nim does, and it works quite well. The bindings are easy to create (there is an automated tool that does most of the work), they look like Nim function declaration and can add better typing checks on FFI calls. You still have manual memory management and pausable, optional GC.

by zzzcpan 12y ago

lmm 12y ago

Because if you use C libraries you will inherit their problems (and also their interface is rarely idiomatic in the new language). Usually if you want to use a different language it's because you think that language has advantages over C. E.g. a large part of the point of using OCaml is to avoid the safety problems of C, but that only applies if your libraries aren't written in C. See https://github.com/mirleft/ocaml-tls#why-a-new-tls-implementation https://github.com/mirleft/ocaml-tls#why-a-new-tls-implement... (Almost all languages do offer some "FFI" to call C. But in languages that don't naturally fit, there will often be significant overhead to e.g. aligning memory management so that the same memory doesn't get freed twice, as might happen when a garbage-collected language called a C library)

chas 12y ago
I think zzzcpan is suggesting to compile from the new toy language to C and then use a C compiler to get native code. This doesn't preclude only using libraries written in the new language. CHICKEN is a Scheme compiler that takes this approach.[0] The submitted essay addresses the problems that can result from this sort of compilation style with the question "Will native compilation of a big project result in so much code that there's a global slowdown (something actually true of mid-1990s Erlang to C translators)?" [0]http://www.call-cc.org/ http://www.call-cc.org/
zzzcpan 12y ago
That's why you would compile it into C instead of having hand-written C when safety is important and for most part of your program it probably is. But sometimes you absolutely need to be able to write assembly and use existing unsafe C libraries.
- lmm 12y ago
  If you absolutely need a C library, almost all languages give you a way to do that. But why hobble your compilation strategy for all programs just to support this occasional use case better? You may well want to target a runtime (JVM, .net, Javascript) for which no good C compiler exists. Even if you're solely interested in building native executables, compiling to C would mean throwing away a lot of information (e.g. knowing which values are immutable) that an optimizer could use. Look at Haskell; performance-optimized Haskell performs a lot better than machine-generated C ever would (indeed, it often comes close to the performance of hand-tuned C). (If you're worried about duplicating effort in optimizers, do what many languages do and offer an LLVM frontend. LLVM optimizations are (mostly) language-independent, and LLVM bytecode, while imperfect, often allows a better representation of language semantics than C source would)
  - zzzcpan 12y ago
    I don't think it is so rare, that you could call it occasional. That's the author's point and mine as well. Sometimes I need sqlite, other times libjpeg, giflib, libpng and some simple image processing with at least some low-level code. And it takes a lot of effort to make these things work with other languages on different platforms. Golang tried to make it easier and it still isn't. Optimizers is also one thing I was worried about, LLVM is not the only compiler out there. And also there is friction. Most of the systems already have C compiler installed, if you target it, there is no friction in setting up building environment and so on.
    - lmm 12y ago
      > Golang tried to make it easier and it still isn't. How much of that is because Golang doesn't compile to C, and how much is because the languages just don't fit? E.g. you'd still need a way to mark whether Golang was supposed to garbage-collect a value that had been returned from a C call, or not. > Most of the systems already have C compiler installed, if you target it, there is no friction in setting up building environment and so on. Yes there is, because you still need to build the thing that interprets to C. If anything there's more friction than having a compiler that directly builds native binaries like e.g. ghc, because you need two tools - your translate-to-C tool and your C compiler. "Most systems already have a C compiler" is only really true on *nix; it will be easier for a Mac or Windows user to run your system if you distribute e.g. java bytecode.
username223 12y ago
> and also their interface is rarely idiomatic in the new language This. If you just want C interop, SWIG[1] has been around forever, and is pretty good at generating bindings from a C/C++ header with minimal effort. But if you've ever tried it, you'll know that you still have to do most of the work by hand to create an idiomatic interface in your language of choice. [1] http://www.swig.org/ http://www.swig.org/

_wmd 12y ago

Portable C significantly restricts the machine primitives available, e.g. no explicit SIMD, no explicit control over struct/memory layout, no obviously efficient way to represent type information (say, for implementation of a moving GC), no ability to access machine registers (again GC), ...

zzzcpan 12y ago
You don't have to write extra code in a portable manner, that's the point. You write it how you need it in a real world. But still benefit from refcounting, nice safe strings, higher-order functions, etc. in main part of your program.

dreadfulgoat 12y ago

If what you want is to use the C library, shoehorning the library APIs into your language is guaranteed to be nastier than just writing in C.

lomnakkus 12y ago

One potential objection I see (aside from the ones raised by others in this thread) is that C isn't actually that great as a compile target. Aliasing analysis is extremely hard in C (thus preventing obvious optimizations), you'd still have to implement your own GC on top of C, C doesn't do tail-call optimization so you'd have to do your own CPS transform anyway... at which point most of the advantages evaporate. (If you're thinking that interfacing with C would be easier: No, the FFIs to C are usually as complex as they are for exactly the right reasons, namely that C's semantics don't match very well with $OTHER_LANGUAGE.) ... at which point you're practically implementing your own VM anyway, so, y'know...

username223 12y ago
> C isn't actually that great as a compile target. That's not the point. C is a great common tongue, as its function call and name lookup semantics are simple, and it doesn't insist on being in control of memory, scheduling, etc. Perl and Python are pretty similar, and both talk to C, but calling between Perl and Python is enough of an unholy mess that it's usually easier to just print text over a pipe.
- lomnakkus 12y ago
  > That's not the point. Maybe not, but did you miss the bit in my post about FFI and GC? Have you actually tried to implement a non-trivial extension to a GC'ed language which needed to interface with C code? EDIT: The point is not so much that it cannot be done -- the point is that it needs a human ("programmer") to do it. Which kind of defeats the point of "C-as-lingua-franca". Which, if you think about it C already kind of is, sadly. Any FFI currently in existence focuses on C already. What more do you want? ;)
  - username223 12y ago
    > did you miss the bit in my post about FFI and GC? No. It's a huge pain, but much less of a pain than calling between two GC'd languages. (I haven't tried calling directly between e.g. Perl and Haskell, but I'd rather gouge my eyes out with a spoon. The handles alone would bury me.) If there were one obvious GC solution, then all GC'd languages would use it, and C would not be as useful. But we've had the JVM and countless other abstractions, yet none has managed to be a higher common denominator than C, so it looks like human labor for the foreseeable future.

desdiv 12y ago

None of the primitive data structures will match up. JVM languages, CLR languages, Rust, Go, et al all have array bound checking, so their arrays won't be compatible with C libraries without a wrapper.

kibwen 12y ago
Rust arrays are perfectly compatible with C arrays, they're both just contiguous segments of memory with elements at fixed offsets. You can pass Rust arrays to C functions trivially, and bounds checking has nothing to do with it. I should also note that all correct C code does bounds checking somewhere. The difference with languages that are bounds-checked by default is that they pessimistically assume that you haven't checked the bound manually, and so can potentially do more bounds checking than is strictly necessary if their optimizers aren't capable of hoisting out the checks.
- NateDad 12y ago
  Ditto for go arrays.

idlewan 12y ago

That's exactly what Nim does, and it works quite well. The bindings are easy to create (there is an automated tool that does most of the work), they look like Nim function declaration and can add better typing checks on FFI calls. You still have manual memory management and pausable, optional GC.