8 ms·

Compiling Clojure to Javascript, pt. 2 -- Why No Eval?

57 points by JoelMcCracken 15y ago

geekam 15y ago
I had to go and look up Eval and Evil again: http://stackoverflow.com/questions/197769/when-is-javascripts-eval-not-evil http://stackoverflow.com/questions/197769/when-is-javascript...
JoelMcCracken 15y ago
I'm not sure that this was clear, but the reason this is important is for an in-browser REPL. Would it be possible to turn down the GCC compilation levels, in the "I want an REPL" use case?
- DanielRibeiro 15y ago
  Seems like a great idea. This way, developers can decide if they need eval and make the tradeoff. Note that eval can also be use for metaprogramming, and that not having the full code of clojure also affects runtime reflection. Which is more to the spirit of Clojure and LISPs in general: developers being able at any momemnt to change the language as if he was the language designer. Or, as said on Five Questions about Language Design[1]: Give the Programmer as Much Control as Possible [1] http://www.paulgraham.com/langdes.html http://www.paulgraham.com/langdes.html
  - JoelMcCracken 15y ago
    Thanks for this. I forgot about this essay.
seles 15y ago
A large advantage of lisp like languages is the data-as-code paradigm. To give up eval is to give up this. It is definitely impossible to efficiently compile code with evals for all possible programs, but lots of code using the data-as-code paradigm generates data that is completely independent of input sources (macro like). This could be computed at compile time and thus efficiently compiled. A similar technique has been used for making Haskell run faster, called super compilation, and it shouldn't be too hard to implement in any functional language. For evals that can't be compiled, well, it would be nice to still be able to do the dynamic inefficient eval, without botching the efficiency of the entire program. Do the normal dead code analysis, the eval might not work right, so provide a warning. The programmer can always do things to ensure any code they need isn't dead code eliminated by using it elsewhere. Yeah this is messy, but it leaves the choice up to the programmer and has no less power.
- takeoutweight 15y ago
  Supercompilation is the precisely the tool that should not be used for compiling a dynamic language--it relies heavily on static analysis that are simply unfeasible in a dynamic language. eg: anything that touches a dynamically bindable var cannot be specialized--this dynamicity will essentially infect the rest of the functions in the call graph. This is especially true when you're staging compilation via eval. Supercompilation is very slow and memory intensive, and it would be insane to track variable specialization information throughout runtime.
  - seles 15y ago
    Static analysis is already being done, since it is being compiled. Supercompilation taken to the extreme is very slow, but don't take it to the extreme, doing just enough to evaluate the expressions which are passed into eval would be fine (and these are often simple, macro like substitutions).
    - takeoutweight 15y ago
      I'm not sure if we share the same notion of supercompilation. The whole point of supercompilation is to speculatively evaluate terms to supply branch-level constraints that propagate through the call graph, enabling program transformations such as inlining, dead code elimination, partial evaluation and deforestation. Whole-program transformations such as these are done during compilation (i.e. in the dev environment and not in production.) How would you retro-actively undo an existing transformed piece of code when a new form is observed by eval? How would you maintain a list of already-applied transformations to allow new code to be specialized? Supercompilation is not an incremental procedure.
      - seles 15y ago
        My notion of supercompilation is just the evaluating of expressions at compile time that do not depend on things at run time. A trivial case that most compilers do is converting print 1+1 to print 2. Since 1+1 does not depend on anything that could happen at run time and does not produce any other side effects. (You need to analyze that + hasn't been overloaded to have side effects/etc.). The idea is that many of the expressions being passed into eval could be calculated in entirety at compile time, so there is no need to have eval in the compiled result at all. Just evaluate at compile time, remove the eval and the quotes. I am not talking about trying to optimize eval, if you have a variable string going in, you can't possibly do that. I guess I shouldn't have used the term supercompiling, it may mean something more powerful than in the cases I'd seen it and I was probably trying too hard to use a buzzword...
sitkack 15y ago
His argument is a total straw man, he wants us to tear it down. He also begs the question that eval is for a repl. So the ClojureScript folks decided to give up flexibility for speed? A Lisp w/o eval is not a Lisp. Strikes me as a motherly premature (de)optimization.
briggers 15y ago
I'm happy with their choices regarding the allocation of their time. I'd rather see more maturity and more core libs being made available to ClojureScript than eval/browser-repl being there immediately.
gfodor 15y ago
It seems that it should be possible to 'force' certain libraries into the compiled js, and then do a 'dumb' non-optimized eval. Yes, you get a bigger js file than if you just stuck with dead code optimization, but having access to the full library at runtime is the entire point.
- apgwoz 15y ago
  You can totally just add a flag called :eliminate-eval or :enable-eval and this is a non-issue.
guard-of-terra 15y ago
Clojure supports namespaces and namespace-public/namespace-private functions. It's relatively easy to stuff every namespace-public function into a hash of its full name -> mangled name, thus both making resolving trivial and preventing supercompilation from stripping functions you might need.
- takeoutweight 15y ago
  You've described exactly why eval is antithetical to dead code elimination: you must defensively include absolutely every public function in every referenced library, because you do not know beforehand which functions an arbitrary evalled form will call.
  - guard-of-terra 15y ago
    Well, we can drop all namespaces other than those used or explicitly specified. It's all solveable.
- lukev 15y ago
  This would prevent supercompilation from stripping any functions, because you never know what a future eval would need. This is bad because, without supercompilation, ClojureScript and its dependencies are quite large. Would you rather your compiled .js file be 500kb or 5kb? That's the difference we're talking about, here. Sure, ClojureScript might not be a "full lisp" without eval, but pragmatically, having tiny .js files is a much bigger win.
  - JoelMcCracken 15y ago
    This level of eval that we're discussing is only really important for developers. Give your super users a bookmarklet which downloads the huge lib, and pops a repl up on the page. Everyone else gets to use the library as usual. Put the standard lib on a public CDN, and this would be okay.
    - lukev 15y ago
jashkenas 15y ago
I have much respect for Mr. Fogus, but statements like this make it difficult to take this post seriously: ClojureScript is designed to solve the types of big problems that are simply too difficult to fathom, much less achieve, using raw JavaScript or any number of JavaScript frameworks. Please. ClojureScript may be a beautiful language, but it's not going to solve your problems-so-large-they're-difficult-to-fathom by magic. Better yet, I'd love to hear a real-world example of such a problem that JavaScript is incapable of addressing properly.
- swannodette 15y ago
  Most of my work in Clojure these days involves implementing logic programming and state-of-art predicate dispatch. Both have numerous practical real world use cases (I want them right now) and both would be near impossible to implement with acceptable syntax / performance in plain JavaScript. Even more importantly this functionality can be provided to end users as libraries not as an alternate version of ClojureScript. For example, while I have a deep respect for CoffeeScript, these types of innovations are "ghettoized" in CoffeeScript because they require users to fork the compiler.
  - jashkenas 15y ago
    Great -- so if logic programming and predicate dispatch allow you to tackle problems that would otherwise be intractable in JS... Do you have an example of such a problem that you've used logic programming in Clojure to tackle? Perhaps a link to your code? I'd tend to give Turing-completeness a bit more credit where it's due, especially for flexible dynamic languages.
    - apgwoz 15y ago
      I trust that you do understand the concept of compilation right? It'd be terribly difficult and tedious to implement JavaScript, for instance, in x86 assembly (but possible!), much like it would be to implement the things that David's logic programming and predicate dispatch in JavaScript directly. Having a language which supports syntactic abstraction allows you to hide away lots of the complexities and let the compiler generate them for you. It's no different than using C instead of x86, or Scheme to C to x86, or whatever other path to x86 you can think of.
ohyes 15y ago
I've thought about this a bit, and I am not sure whether I disagree. I think I do. "But honestly, if ClojureScript programs are composed of data forms directly manipulable by the language itself then why not just provide eval anyway? The answer can be found, as many answers in life, in the tradeoffs." This seems to beat around the bush a lot so here's the problem as I see it: When the code hits the browser, ClojureScript programs are not composed of Data Structures. They are composed of Java script, which acts as an assembly code. In a normal lisp, you would have the compiler right along side the generated assembly, so there would be no big deal... you want to eval something, just compile it and link it in to the run-time. In this case, the compiler is on the server, the code is in the browser... to provide an 'eval', you would have to send an RPC from the server to the browser... It is disingenuous to say that you cannot support an optimizing compiler and maintain the presence of eval in the language. This is simply not true. What you cannot do is: 1.) Separate the compiler from the runtime. 2.) Aggressively ELIMINATE code that is not being used. "You cannot effectively leverage these libraries without an optimizing compiler utilizing dead-code elimination. You cannot have the optimizing compiler if you wish to support eval. Q.E.D." Note that it is 'the optimizing compiler' not 'an optimizing compiler'. It would be possible to have an optimizing compiler without dead code elimination, or even the same optimizing compiler with dead code elimination turned off (no idea of that is possible). I think that claim number 1 is the questionable one. It seems like it would be relatively easy to decide the level of dead code elimination by indicating whether eval is available to the language. It may just mean you have to declare what functions/packages are to be made available to eval (which from a security standpoint makes a lot of sense; actually). --- Thinking about it, A possible repl implementation would actually be fairly simple: 1.) Create a hashtable 2.) Fill hash-table with desirable functions to execute 3.) Write an interpreter that recursively applies the head symbol of a list to the tail of a list (by looking up the symbol in the hash table). (Edit: There may be another wrinkle, like looking up variable symbols).This is basically what eval does. 4.) Create a loop that parses a lisp list into a javascript list, and applies the interpreter. The hole here is that there is no way to define 'new' functions. This may be possible by having a function returning a function which calls our interpreter (and adding that to the hash table as a new function). The question I have about this is being able to use arguments.. but even that may be possible. In fact, the above seems to allow for dead code elimination, as you have declared exactly which functions you want accessible to the user. edit: Sorry if I offended, but what is wrong with my post?
abecedarius 15y ago
You can easily write eval as a regular function, right? Except it needs magical access to the global environment. (Please correct me if I'm wrong; I haven't used Clojure, just checked out http://clojure.org/evaluation. http://clojure.org/evaluation.) So I'd blame the definition of Clojure's eval function more than the implementation goals here -- if users had to write (eval foo (the-global-environment)) then you could leave the-global-environment out and still supply a useful eval without any problem in compiling programs.
pnathan 15y ago
I'm not a Clojure user, but I am a bit confused about why you can't have EVAL, and also a COMPILE function, which does the appropriate dead-code elimination. That is, in Common Lisp, the REPL functions aren't usually compiled, but you can call a COMPILE function to trigger them into bytecode. I don't see the difference for your problem. I would like to be corrected. :-)

57 points by JoelMcCracken 15y ago

I had to go and look up Eval and Evil again: http://stackoverflow.com/questions/197769/when-is-javascripts-eval-not-evil http://stackoverflow.com/questions/197769/when-is-javascript...

JoelMcCracken 15y ago

I'm not sure that this was clear, but the reason this is important is for an in-browser REPL. Would it be possible to turn down the GCC compilation levels, in the "I want an REPL" use case?

DanielRibeiro 15y ago
Seems like a great idea. This way, developers can decide if they need eval and make the tradeoff. Note that eval can also be use for metaprogramming, and that not having the full code of clojure also affects runtime reflection. Which is more to the spirit of Clojure and LISPs in general: developers being able at any momemnt to change the language as if he was the language designer. Or, as said on Five Questions about Language Design[1]: Give the Programmer as Much Control as Possible [1] http://www.paulgraham.com/langdes.html http://www.paulgraham.com/langdes.html
- JoelMcCracken 15y ago
  Thanks for this. I forgot about this essay.

seles 15y ago

A large advantage of lisp like languages is the data-as-code paradigm. To give up eval is to give up this. It is definitely impossible to efficiently compile code with evals for all possible programs, but lots of code using the data-as-code paradigm generates data that is completely independent of input sources (macro like). This could be computed at compile time and thus efficiently compiled. A similar technique has been used for making Haskell run faster, called super compilation, and it shouldn't be too hard to implement in any functional language. For evals that can't be compiled, well, it would be nice to still be able to do the dynamic inefficient eval, without botching the efficiency of the entire program. Do the normal dead code analysis, the eval might not work right, so provide a warning. The programmer can always do things to ensure any code they need isn't dead code eliminated by using it elsewhere. Yeah this is messy, but it leaves the choice up to the programmer and has no less power.

takeoutweight 15y ago
Supercompilation is the precisely the tool that should not be used for compiling a dynamic language--it relies heavily on static analysis that are simply unfeasible in a dynamic language. eg: anything that touches a dynamically bindable var cannot be specialized--this dynamicity will essentially infect the rest of the functions in the call graph. This is especially true when you're staging compilation via eval. Supercompilation is very slow and memory intensive, and it would be insane to track variable specialization information throughout runtime.
- seles 15y ago
  Static analysis is already being done, since it is being compiled. Supercompilation taken to the extreme is very slow, but don't take it to the extreme, doing just enough to evaluate the expressions which are passed into eval would be fine (and these are often simple, macro like substitutions).
  - takeoutweight 15y ago
    I'm not sure if we share the same notion of supercompilation. The whole point of supercompilation is to speculatively evaluate terms to supply branch-level constraints that propagate through the call graph, enabling program transformations such as inlining, dead code elimination, partial evaluation and deforestation. Whole-program transformations such as these are done during compilation (i.e. in the dev environment and not in production.) How would you retro-actively undo an existing transformed piece of code when a new form is observed by eval? How would you maintain a list of already-applied transformations to allow new code to be specialized? Supercompilation is not an incremental procedure.
    - seles 15y ago
      My notion of supercompilation is just the evaluating of expressions at compile time that do not depend on things at run time. A trivial case that most compilers do is converting print 1+1 to print 2. Since 1+1 does not depend on anything that could happen at run time and does not produce any other side effects. (You need to analyze that + hasn't been overloaded to have side effects/etc.). The idea is that many of the expressions being passed into eval could be calculated in entirety at compile time, so there is no need to have eval in the compiled result at all. Just evaluate at compile time, remove the eval and the quotes. I am not talking about trying to optimize eval, if you have a variable string going in, you can't possibly do that. I guess I shouldn't have used the term supercompiling, it may mean something more powerful than in the cases I'd seen it and I was probably trying too hard to use a buzzword...

sitkack 15y ago

His argument is a total straw man, he wants us to tear it down. He also begs the question that eval is for a repl. So the ClojureScript folks decided to give up flexibility for speed? A Lisp w/o eval is not a Lisp. Strikes me as a motherly premature (de)optimization.

briggers 15y ago

I'm happy with their choices regarding the allocation of their time. I'd rather see more maturity and more core libs being made available to ClojureScript than eval/browser-repl being there immediately.

gfodor 15y ago

It seems that it should be possible to 'force' certain libraries into the compiled js, and then do a 'dumb' non-optimized eval. Yes, you get a bigger js file than if you just stuck with dead code optimization, but having access to the full library at runtime is the entire point.

apgwoz 15y ago
You can totally just add a flag called :eliminate-eval or :enable-eval and this is a non-issue.

guard-of-terra 15y ago

Clojure supports namespaces and namespace-public/namespace-private functions. It's relatively easy to stuff every namespace-public function into a hash of its full name -> mangled name, thus both making resolving trivial and preventing supercompilation from stripping functions you might need.

takeoutweight 15y ago
You've described exactly why eval is antithetical to dead code elimination: you must defensively include absolutely every public function in every referenced library, because you do not know beforehand which functions an arbitrary evalled form will call.
- guard-of-terra 15y ago
  Well, we can drop all namespaces other than those used or explicitly specified. It's all solveable.
lukev 15y ago
This would prevent supercompilation from stripping any functions, because you never know what a future eval would need. This is bad because, without supercompilation, ClojureScript and its dependencies are quite large. Would you rather your compiled .js file be 500kb or 5kb? That's the difference we're talking about, here. Sure, ClojureScript might not be a "full lisp" without eval, but pragmatically, having tiny .js files is a much bigger win.
- JoelMcCracken 15y ago
  This level of eval that we're discussing is only really important for developers. Give your super users a bookmarklet which downloads the huge lib, and pops a repl up on the page. Everyone else gets to use the library as usual. Put the standard lib on a public CDN, and this would be okay.
  - lukev 15y ago

jashkenas 15y ago

I have much respect for Mr. Fogus, but statements like this make it difficult to take this post seriously: ClojureScript is designed to solve the types of big problems that are simply too difficult to fathom, much less achieve, using raw JavaScript or any number of JavaScript frameworks. Please. ClojureScript may be a beautiful language, but it's not going to solve your problems-so-large-they're-difficult-to-fathom by magic. Better yet, I'd love to hear a real-world example of such a problem that JavaScript is incapable of addressing properly.

swannodette 15y ago
Most of my work in Clojure these days involves implementing logic programming and state-of-art predicate dispatch. Both have numerous practical real world use cases (I want them right now) and both would be near impossible to implement with acceptable syntax / performance in plain JavaScript. Even more importantly this functionality can be provided to end users as libraries not as an alternate version of ClojureScript. For example, while I have a deep respect for CoffeeScript, these types of innovations are "ghettoized" in CoffeeScript because they require users to fork the compiler.
- jashkenas 15y ago
  Great -- so if logic programming and predicate dispatch allow you to tackle problems that would otherwise be intractable in JS... Do you have an example of such a problem that you've used logic programming in Clojure to tackle? Perhaps a link to your code? I'd tend to give Turing-completeness a bit more credit where it's due, especially for flexible dynamic languages.
  - apgwoz 15y ago
    I trust that you do understand the concept of compilation right? It'd be terribly difficult and tedious to implement JavaScript, for instance, in x86 assembly (but possible!), much like it would be to implement the things that David's logic programming and predicate dispatch in JavaScript directly. Having a language which supports syntactic abstraction allows you to hide away lots of the complexities and let the compiler generate them for you. It's no different than using C instead of x86, or Scheme to C to x86, or whatever other path to x86 you can think of.

ohyes 15y ago

I've thought about this a bit, and I am not sure whether I disagree. I think I do. "But honestly, if ClojureScript programs are composed of data forms directly manipulable by the language itself then why not just provide eval anyway? The answer can be found, as many answers in life, in the tradeoffs." This seems to beat around the bush a lot so here's the problem as I see it: When the code hits the browser, ClojureScript programs are not composed of Data Structures. They are composed of Java script, which acts as an assembly code. In a normal lisp, you would have the compiler right along side the generated assembly, so there would be no big deal... you want to eval something, just compile it and link it in to the run-time. In this case, the compiler is on the server, the code is in the browser... to provide an 'eval', you would have to send an RPC from the server to the browser... It is disingenuous to say that you cannot support an optimizing compiler and maintain the presence of eval in the language. This is simply not true. What you cannot do is: 1.) Separate the compiler from the runtime. 2.) Aggressively ELIMINATE code that is not being used. "You cannot effectively leverage these libraries without an optimizing compiler utilizing dead-code elimination. You cannot have the optimizing compiler if you wish to support eval. Q.E.D." Note that it is 'the optimizing compiler' not 'an optimizing compiler'. It would be possible to have an optimizing compiler without dead code elimination, or even the same optimizing compiler with dead code elimination turned off (no idea of that is possible). I think that claim number 1 is the questionable one. It seems like it would be relatively easy to decide the level of dead code elimination by indicating whether eval is available to the language. It may just mean you have to declare what functions/packages are to be made available to eval (which from a security standpoint makes a lot of sense; actually). --- Thinking about it, A possible repl implementation would actually be fairly simple: 1.) Create a hashtable 2.) Fill hash-table with desirable functions to execute 3.) Write an interpreter that recursively applies the head symbol of a list to the tail of a list (by looking up the symbol in the hash table). (Edit: There may be another wrinkle, like looking up variable symbols).This is basically what eval does. 4.) Create a loop that parses a lisp list into a javascript list, and applies the interpreter. The hole here is that there is no way to define 'new' functions. This may be possible by having a function returning a function which calls our interpreter (and adding that to the hash table as a new function). The question I have about this is being able to use arguments.. but even that may be possible. In fact, the above seems to allow for dead code elimination, as you have declared exactly which functions you want accessible to the user. edit: Sorry if I offended, but what is wrong with my post?

abecedarius 15y ago

You can easily write eval as a regular function, right? Except it needs magical access to the global environment. (Please correct me if I'm wrong; I haven't used Clojure, just checked out http://clojure.org/evaluation. http://clojure.org/evaluation.) So I'd blame the definition of Clojure's eval function more than the implementation goals here -- if users had to write (eval foo (the-global-environment)) then you could leave the-global-environment out and still supply a useful eval without any problem in compiling programs.

pnathan 15y ago

I'm not a Clojure user, but I am a bit confused about why you can't have EVAL, and also a COMPILE function, which does the appropriate dead-code elimination. That is, in Common Lisp, the REPL functions aren't usually compiled, but you can call a COMPILE function to trigger them into bytecode. I don't see the difference for your problem. I would like to be corrected. :-)