4 ms·

Did you look at Julia[0]? It is a Lisp, semantically, with a MATLAB-like syntax (including all the linear algebra sugar), and a LLVM back end. The stated goal

by pygy_ 14y ago

Did you look at Julia[0]? It is a Lisp, semantically, with a MATLAB-like syntax (including all the linear algebra sugar), and a LLVM back end. The stated goal of the language is to give to scienctific programmers the convenience of high level languages for prototyping, yet the speed of low level ones. It has multiple dispatch, based on a powerful type system, homoiconicity and macros (optionally hygienic). The package collection is growing steadily [1]. There's also a mailing list dedicated to doing statistics with Julia[2]. I looks like a good fit for your project. [0] http://julialang.org http://julialang.org [1] http://docs.julialang.org/en/release-0.1/packages/packagelist.html http://docs.julialang.org/en/release-0.1/packages/packagelis... [2] https://groups.google.com/forum/?fromgroups=#!forum/julia-stats https://groups.google.com/forum/?fromgroups=#!forum/julia-st...

zygomega 14y ago
yes. Julia looks like a quality language but suffers the usual new kid on teh block hassles - very few packages so you have to spend too long rolling your own.
- pygy_ 14y ago
  Are there specific packages that you would currently need, but aren't there (yet), or is it a more general concern about the relative package scarcity and unforseen changes to / future needs of your code?
  - zygomega 14y ago
    It's not specific, it's just my recent experience. Hadley released his bigvis package a few weeks ago, and it contains the beginnings of a weighted average routine I sure can use. mmap is a package that let me create a fast db from scratch in a day. ggplot, lattice and all the charting routines are a joy to work with. I'm interested in random forests today, and I bet there's a package that links to the standard libraries whatever they are. This is also probably true for python, but Julia you'd have to build the API.
carterschonwald 14y ago
i've looked at how it compiles to llvm, and it basically punts any optimization to the llvm side, aside from the most basic tracing jit method monomorphization/specialization. (as in, I spent part of an afternoon end of last week skimming the entire code base). This is troubling, because when you don't design your language to make it "manageable" to do static analysis / optimization on, it be becomes DIFFICULT to add that capability later. Julia is about as dynamic as JS or Scheme or Lua, plus it has all those runtime type tag cases in code... but whereas JS has relatively unbounded manpower (relatively!) to make a fast JIT, whereas lua is a small enough language that we have the amazing impressive LuaJIT, whereas Scheme is designed with very thoughtful semantics / specifications in mind, Julia lacks. more succintly, Julia lacks a clear enough thoughtful choice in static/dynamic semantics for the pre LLVM side to have an easy optimization story given a small sized core team, such optimization engineering will either take a long time to develop, or will require some commercial entity to sink serious capital into writing a good JIT / optimizer. LLVM is a good backend, but it can't leverage the semantics of the input language, just the semantics of the LLVM bitcode you provide. You really really can't punt on optimization on the before LLVM side. also: Julia doesn't have a type system, it has a dynamic check system. (Show me the code for static type check in the julia code base as it exists on github and I'll owe you a beer :) ) Let me repeat: Julia doesn't have clear static semantics / phase distinction, and doesn't really seem to do anything beyond method specialization before passing code on to LLVM. This means it can't get compelling performance compared with any other LLVM backed language. LLVM is amazing, and i use ghc -fllvm exclusively, and while theres some really really awesome optimizations that LLVM does for haskell code (for example), a HUGE amount of high level code performance is really depending on the language (any language!) doing optimizations before passing things onto the LLVM.
- pygy_ 14y ago
  I didn't look at the implementation, and would be out of my depth if I were to. I'll relay your concerns to the authors (I share some of them, actually), but, meanwhile, here are a my thoughts: The fact is that the performance of Julia is already quite good. Julia is semantically much closer to C than Haskell is, and it needs fewer optimisations to get reasonable speed. The necessity of static analysis for performance is a myth that's been debunked for a while now. From the language semantics, I think that the runtime model of Julia must be about as complex as Lua's, including the LuaJIT FFI (swap metatables with type-based multiple dispatch). As you said, though, whereas LuaJIT has its own tailored bytecode, Julia compiles to LLVM IR, and potentially useful information is lost in the translation process. LLVM-Lua is reportedly 2-3x slower than LuaJIT. I suppose that type specialized traces and immutable types help somewhat. An area that clearly needs improvement is vectorised code. Currently, an new vector/matrix is allocated for each intermediate step. Bad. This is being worked on. > also: Julia doesn't have a type system, it has a dynamic check system. (Show me the code for static type check in the julia code base as it exists on github and I'll owe you a beer :) ) You're conflating type system with static type checking. Julia is dynamically typed, but its programming model (the UI of sorts) is built around the type system and multiple dispatch, it is one of its main features of the language, actually.

by pygy_ 14y ago

zygomega 14y ago

yes. Julia looks like a quality language but suffers the usual new kid on teh block hassles - very few packages so you have to spend too long rolling your own.

pygy_ 14y ago
Are there specific packages that you would currently need, but aren't there (yet), or is it a more general concern about the relative package scarcity and unforseen changes to / future needs of your code?
- zygomega 14y ago
  It's not specific, it's just my recent experience. Hadley released his bigvis package a few weeks ago, and it contains the beginnings of a weighted average routine I sure can use. mmap is a package that let me create a fast db from scratch in a day. ggplot, lattice and all the charting routines are a joy to work with. I'm interested in random forests today, and I bet there's a package that links to the standard libraries whatever they are. This is also probably true for python, but Julia you'd have to build the API.

carterschonwald 14y ago

i've looked at how it compiles to llvm, and it basically punts any optimization to the llvm side, aside from the most basic tracing jit method monomorphization/specialization. (as in, I spent part of an afternoon end of last week skimming the entire code base). This is troubling, because when you don't design your language to make it "manageable" to do static analysis / optimization on, it be becomes DIFFICULT to add that capability later. Julia is about as dynamic as JS or Scheme or Lua, plus it has all those runtime type tag cases in code... but whereas JS has relatively unbounded manpower (relatively!) to make a fast JIT, whereas lua is a small enough language that we have the amazing impressive LuaJIT, whereas Scheme is designed with very thoughtful semantics / specifications in mind, Julia lacks. more succintly, Julia lacks a clear enough thoughtful choice in static/dynamic semantics for the pre LLVM side to have an easy optimization story given a small sized core team, such optimization engineering will either take a long time to develop, or will require some commercial entity to sink serious capital into writing a good JIT / optimizer. LLVM is a good backend, but it can't leverage the semantics of the input language, just the semantics of the LLVM bitcode you provide. You really really can't punt on optimization on the before LLVM side. also: Julia doesn't have a type system, it has a dynamic check system. (Show me the code for static type check in the julia code base as it exists on github and I'll owe you a beer :) ) Let me repeat: Julia doesn't have clear static semantics / phase distinction, and doesn't really seem to do anything beyond method specialization before passing code on to LLVM. This means it can't get compelling performance compared with any other LLVM backed language. LLVM is amazing, and i use ghc -fllvm exclusively, and while theres some really really awesome optimizations that LLVM does for haskell code (for example), a HUGE amount of high level code performance is really depending on the language (any language!) doing optimizations before passing things onto the LLVM.

pygy_ 14y ago
I didn't look at the implementation, and would be out of my depth if I were to. I'll relay your concerns to the authors (I share some of them, actually), but, meanwhile, here are a my thoughts: The fact is that the performance of Julia is already quite good. Julia is semantically much closer to C than Haskell is, and it needs fewer optimisations to get reasonable speed. The necessity of static analysis for performance is a myth that's been debunked for a while now. From the language semantics, I think that the runtime model of Julia must be about as complex as Lua's, including the LuaJIT FFI (swap metatables with type-based multiple dispatch). As you said, though, whereas LuaJIT has its own tailored bytecode, Julia compiles to LLVM IR, and potentially useful information is lost in the translation process. LLVM-Lua is reportedly 2-3x slower than LuaJIT. I suppose that type specialized traces and immutable types help somewhat. An area that clearly needs improvement is vectorised code. Currently, an new vector/matrix is allocated for each intermediate step. Bad. This is being worked on. > also: Julia doesn't have a type system, it has a dynamic check system. (Show me the code for static type check in the julia code base as it exists on github and I'll owe you a beer :) ) You're conflating type system with static type checking. Julia is dynamically typed, but its programming model (the UI of sorts) is built around the type system and multiple dispatch, it is one of its main features of the language, actually.