3 ms·

> academic lanaguages don't succeed because the incentives aren't aligned A much harder problem is that compilers are shockingly time-consuming to fully implem

> academic lanaguages don't succeed because the incentives aren't aligned A much harder problem is that compilers are shockingly time-consuming to fully implement and test, much less to integrate with a full set of libraries and tools that you need to use them in practice. Even if you have a new language design, implementing all of the "basic" optimizations to bring your compiler from hopeless to merely embarrasing requires person-years of heroic effort, none of which results in publications or other recognition required to keep your NSF funding coming (they're "basic"). For example, it has taken us since 2007 to get Manticore (http://manticore.cs.uchicago.edu/ http://manticore.cs.uchicago.edu/ ) to a point where our sequential performance is within a factor of 2-4 of C (depending on the benchmark), and closing that last bit is probably going to take another couple of years unless we have some magical windfall of stunning undergraduate and graduate student candidates. Further, over the _entire_ lifetime of the project, I'd doubt that we will be able to put in even half the people-hours that I saw committed to improving the template error messages in Visual C++ during my first two years working at MSFT. That said, we've been able to do some truly great things with scheduling computations on multicores, building a GC that works amazingly on NUMA computers, actually getting real speedups on > 36 core machines, etc. But I wouldn't expect to see any of the language features or implementation tricks in mainstream languages for many years. That time lag is just sort of the nature of things in PL research.

haberman 15y ago
Doesn't a modular compiler framework like LLVM significantly reduce this barrier-to-entry?
- larsberg 15y ago
  Not significantly. If you check out the sources for most functional language compilers (GHC, any of the ML family, scheme, etc.), code generation and micro-optimizations make up a tiny fraction of the code and developer time investment. LLVM is a fabulous project, but many of the optimizations that it performs are already handled at a higher level of the compiler (loop unrolling, inlining, etc.) in a type-preserving functional language. Richer type information (particularly for algebraic datatypes) enables things like loop unrolling over recursive data structures, which you really can't do anymore once it's a bunch of typecasts and tagged union discriminations. That said, we're looking to move over to it because our old code generation library (MLRISC) is beginning to show its age, and porting to LLVM is probably going to be about as much work as fixing the spill code bug we recently hit and exposing more SSE instructions. Most of the work will probably be in porting our calling convention, which does not resemble C's in any way. Like many frameworks, it's a modular compiler framework for building C compilers, not really a modular compiler framework for all types of compilers. That's not necessarily a bad thing; many people have tried and failed to make more general ones. I'd rather have to do work to shoehorn our compiler into something that works and is widely used in industry than rely on something that's an easier fit but only likely to be around as long as the group is still publishing papers on it.
  - sanxiyn 15y ago
    > Most of the work will probably be in porting our calling convention, which does not resemble C's in any way. Like many frameworks, it's a modular compiler framework for building C compilers, not really a modular compiler framework for all types of compilers. GHC folks succeeded in including GHC calling convention in LLVM, so there is hope. http://blog.llvm.org/2010/05/glasgow-haskell-compiler-and-llvm.html http://blog.llvm.org/2010/05/glasgow-haskell-compiler-and-ll...
    - larsberg 15y ago
      Yes, we're familiar with that work. One technical bit of interest is that we don't use the C stack at all (we have heap-allocated continuation records), so we're curious how that will play out with the calling convention even beyond the issue with registers. That said, we're still hopeful that we can make things work, though we're not optimistic that it will come about without some significant tweaking.

haberman 15y ago

Doesn't a modular compiler framework like LLVM significantly reduce this barrier-to-entry?

larsberg 15y ago
Not significantly. If you check out the sources for most functional language compilers (GHC, any of the ML family, scheme, etc.), code generation and micro-optimizations make up a tiny fraction of the code and developer time investment. LLVM is a fabulous project, but many of the optimizations that it performs are already handled at a higher level of the compiler (loop unrolling, inlining, etc.) in a type-preserving functional language. Richer type information (particularly for algebraic datatypes) enables things like loop unrolling over recursive data structures, which you really can't do anymore once it's a bunch of typecasts and tagged union discriminations. That said, we're looking to move over to it because our old code generation library (MLRISC) is beginning to show its age, and porting to LLVM is probably going to be about as much work as fixing the spill code bug we recently hit and exposing more SSE instructions. Most of the work will probably be in porting our calling convention, which does not resemble C's in any way. Like many frameworks, it's a modular compiler framework for building C compilers, not really a modular compiler framework for all types of compilers. That's not necessarily a bad thing; many people have tried and failed to make more general ones. I'd rather have to do work to shoehorn our compiler into something that works and is widely used in industry than rely on something that's an easier fit but only likely to be around as long as the group is still publishing papers on it.
- sanxiyn 15y ago
  > Most of the work will probably be in porting our calling convention, which does not resemble C's in any way. Like many frameworks, it's a modular compiler framework for building C compilers, not really a modular compiler framework for all types of compilers. GHC folks succeeded in including GHC calling convention in LLVM, so there is hope. http://blog.llvm.org/2010/05/glasgow-haskell-compiler-and-llvm.html http://blog.llvm.org/2010/05/glasgow-haskell-compiler-and-ll...
  - larsberg 15y ago
    Yes, we're familiar with that work. One technical bit of interest is that we don't use the C stack at all (we have heap-allocated continuation records), so we're curious how that will play out with the calling convention even beyond the issue with registers. That said, we're still hopeful that we can make things work, though we're not optimistic that it will come about without some significant tweaking.