6 ms·

On eval in dynamic languages generally and in Racket specifically (2011)

71 points by swatson741 1y ago

dang 1y ago
Related. Others? On eval in dynamic languages generally and in Racket specifically (2011) - https://news.ycombinator.com/item?id=8098569 https://news.ycombinator.com/item?id=8098569 - July 2014 (18 comments)
behnamoh 1y ago
Is my understanding correct that Lisp's powerful macro system stems from the ability to write the eval function in Lisp itself? From what I gather, Lisp starts with a small set of primitives and special forms—seven in the original Lisp, including lambda. I recall Paul Graham demonstrating in one of his essays that you can build an eval function using just these primitives. Those primitives are typically implemented in a host language like C, but once you have an eval function in Lisp, you can extend it with new rules. The underlying C interpreter only sees the primitives, but as a programmer, you can introduce new syntax rules via eval. This seems like a way to understand macros, where you effectively add new language rules. I know Lisp macros are typically defined using specific keywords like defmacro, but is the core idea similar—extending the language by building on the eval function with new rules?
- samth 1y ago
  No, macros and eval are quite different. You can see this for example in Python or JavaScript, which have eval but not macros.
  - Y_Y 1y ago
    You can make macros in Python: https://github.com/lihaoyi/macropy https://github.com/lihaoyi/macropy (note that that project was started for a class taught by Sussman) There's also a PEP to make them first-class: https://peps.python.org/pep-0638/ https://peps.python.org/pep-0638/
    - sparkie 1y ago
      That's a different meaning of first-class from Strachey's definition of a first-class citizen[1] - ie, one that can be passed as an argument, returned from a function, or assigned to a variable. Syntactic macros are still second-class, like Lisp macros, but an improvement over text-replacement style macros. For something macro-like which is first-class, there are fexprs[2] and operatives (from Kernel[3]) - these receive their operands verbatim, like macros, so they don't require quotation if we want to suppress evaluation. fexprs/Operatives can be passed around like any other value at runtime. [1]:https://en.wikipedia.org/wiki/First-class_citizen https://en.wikipedia.org/wiki/First-class_citizen [2]:https://en.wikipedia.org/wiki/Fexpr https://en.wikipedia.org/wiki/Fexpr [3]:https://web.cs.wpi.edu/~jshutt/kernel.html https://web.cs.wpi.edu/~jshutt/kernel.html
      - Y_Y 1y ago
        Stratchey defined "first-class objects". This was by analogy with "first-class citizens" in a legal/political sense, since they are treated just as well as any other object and have no additional limitations. If we extend the analogy to syntax then I think it's clear enough that it means that it is a piece of syntax which is treated the same as any other and does not require special treatment or impose additional restrictions. Thank you for the clarification and the additional information, I think having macros as first-class objects is a cool (but separate) idea.
  - matheusmoreira 1y ago
    They aren't that different. Fexprs are essentially additional eval cases.
- sparkie 1y ago
  > Is my understanding correct that Lisp's powerful macro system stems from the ability to write the eval function in Lisp itself? I wouldn't say this is the case. Nearly any language could implement an `eval` for itself, but obviously, it is much simpler in Lisps because there's little syntax and few rules. What makes Lisp macros different from say, C preprocessor macros, is the body of the macro is just Lisp code - so the "preprocessor" in this case has full access to the host languages facilities, which may include `eval`. The macros don't take textual input, but they take structured input in the form of S-expressions. Implementing macros is obviously simpler due to eval, because we need to run the evaluator on the macro body, but it's not a strict requirement, as macro functionality could be provided by the implementation and could encapsulate its own evaluator. Lisp macros are also simple due to the fact that Lisp code is just lists of data - you don't have to navigate a complex AST to walk through the code and emit particular syntax. You walk through the input with `car` and `cdr`, and you emit new syntax with `cons` (or `list`/`list*` which are derived from it). Macros can take code as their argument, and produce new code which is evaluated in place. Macros still have hygiene issues though, because they're based on expanding code before evaluating it, variables used in macros can accidentally shadow variables in the scope of the macro's caller. There are workarounds (gensym) to navigate these hygiene problems. > From what I gather, Lisp starts with a small set of primitives and special forms—seven in the original Lisp, including lambda. I recall Paul Graham demonstrating in one of his essays that you can build an eval function using just these primitives. This is largely a theoretical demonstration but not real-world usage. In practice, Lisps have dozens or hundreds of "primitives". Common Lisp in particular is a big language and not trivial to implement. Scheme is a bit smaller, though r6rs started to also grow quite large, but this approach was revisited in r7rs (current), which aims for a small core, with additional functionality being provided through SRFIs (Scheme requests for implementation). > Those primitives are typically implemented in a host language like C, but once you have an eval function in Lisp, you can extend it with new rules. Using Scheme as an example, some SRFIs can be implemented purely in Scheme, as libraries, but others require the implementation to provide support, which often requires writing C code to provide them. > This seems like a way to understand macros, where you effectively add new language rules. I know Lisp macros are typically defined using specific keywords like defmacro As you note, it's `defmacro`, or `macro`, or `syntax-rules`, `syntax-case`, etc, which introduce new syntax - not eval in particular. Some macros will use `eval` in their bodies, which permits control of evaluation other than the regular applicative form of lambdas. Macros are more than just `eval`. They're a multi-stage evaluation model where we first need to do some `macroexpand` (which will internally use `eval`), and afterwards the the resulting expression from the macro call is evaluated. > but is the core idea similar—extending the language by building on the eval function with new rules? There are some Lisps which still attempt this kind of minimalism. One example is Ian Piumarta's Maru[1], which support extending `eval` (and `apply`) with new functionality at runtime based on the type being evaluated. Maru basically has global maps of type->evaluator and type->applicator, where we can add new pairs to at runtime and augment the behavior of `eval`/`apply`. Kernel[2] also aims for the minimalist approach and does away with macros, quote and special-forms entirely, instead replacing them with a more general feature called an operative. The Kernel evaluator does not need to implement special rules for things like `lambda`, `cond`, `car`, `cdr` (as in Graham's "On Lips" essay) - but it just discriminates two forms - operative or applicative. Obviously, some kinds of operative are "primitive", but there's no difference from the PoV of the programmer. Which set of symbols you decide to implement as primitive is up to the implementation. The Kernel report suggests a small set of primitives and demonstrates the remaining standard features can be implemented using only the functionality provided so far. [1]:https://piumarta.com/software/maru/ https://piumarta.com/software/maru/ [2]:https://web.cs.wpi.edu/~jshutt/kernel.html https://web.cs.wpi.edu/~jshutt/kernel.html
- kazinator 1y ago
  The relationship is this: the ease of writing a meta-circular eval in Lisp, and the ease of writing macros are related by common causes.
taeric 1y ago
A giant difference between eval in most languages versus lisp, is that eval in lisp doesn't just take in a string. Sounds somewhat subtle, but it can lead to some dramatic differences. And is, in at least a small part, why lisp people claim code is data. (I had a fun time exploring this a long time ago in https://taeric.github.io/CodeAsData.html https://taeric.github.io/CodeAsData.html)
- kevin_thibedeau 1y ago
  Lisp was supposed to ingest M-expressions and convert them internally. Eval would have been more like other languages had that plan not been abandoned. It is not meaningfully different for languages that construct an AST from a string before evaluation.
  - taeric 1y ago
    I'm not sure I follow? Presuming that the M-expressions were also made of atoms, they would still be meaningfully different than taking in the string of text. Closer to the same, agreed, but still not the same.
    - spauldo 1y ago
      IIRC M-expressions broke homoiconicity, so you couldn't just spit sexps at eval and expect anything sensible. I believe that's a lot of the reason they never got implemented.
      - taeric 1y ago
        I'm still not sure I understand. It isn't like you can just throw sexps at an eval and expect something sensible, either? As a sibling was showing, calling eval on (1 2 3) will probably not be anything useful. Seems that what makes strings terrible is that they don't really encode any other structure, already. Contrast this with atoms and lists, which do encode a bit of structure. It isn't just a string of characters, it is a list of atoms and lists. Structure is already part of the thing. It does not have to be parsed out. If m-expressions would break that, I think I can see your point. But I'm confused on what would be the point of m-expressions, if they are largely just to remove structure from the content.
TOGoS 1y ago
In summary: The more layers of interpretation your `eval` function needs to do (assumption of source language, tokenization/parsing rules, symbol definitions, etc), the less well-defined the semantics are. So on the less-well-defined end, you have something like JavaScript's `eval` function, in the middle you have lisp macros, which get an AST instead of just source code, and on the actually-pretty-well-defined end, you have a lambda that's already been parsed/compiled and just needs to be evaluated.
programLyrique 1y ago
About how eval is used in other languages: - in Javascript: https://link.springer.com/chapter/10.1007/978-3-642-22655-7_4 https://link.springer.com/chapter/10.1007/978-3-642-22655-7_... - in R: https://dl.acm.org/doi/abs/10.1145/3485502 https://dl.acm.org/doi/abs/10.1145/3485502
curtisszmania 1y ago
[dead]
neilv 1y ago
I'm glad Matthew Flatt wrote this. There is a problem of newbies who read some academic textbook ("metacircular evaluator!") and then immediately try to use the eval feature when it doesn't make sense. (Also, there's a more general problem of newbies trying to use the most advanced tool they've been exposed to. Why use the standard `if` statement, when you can use a proprietary high-powered pattern-matching form, now backed by a networked Kubernetes cluster of deep learning and LLMs.) In the Racket community, one of my many attempts to discourage mistaken uses of `eval`: https://groups.google.com/g/racket-users/c/Z-IlF24RAKU/m/3h6pjUD1J-4J https://groups.google.com/g/racket-users/c/Z-IlF24RAKU/m/3h6... In the all-in-one practitioner's book I was writing, eval wouldn't be introduced until almost the end, in the "Dangerous Last Resort" part of the book. (Maybe I should've planned a marketing gimmick around it, as "free bonus DLC", and you need to do some ritual to be a Certified Certifiable Racketeer, before you can read the eval secret scroll.)
deleted 1y ago
[deleted]
kazinator 1y ago
The sandboxing of eval is something that can be left to a layer on top. You certainly don't want eval itself to be gutted of functionality; nothing less than he full language, thank you. eval can be sandboxed with a combination of multiple approaches, including validating the code, and evaluating it in a restricted environment in which only certain symbols exist. The full library package is not visible.

71 points by swatson741 1y ago

dang 1y ago

Related. Others? On eval in dynamic languages generally and in Racket specifically (2011) - https://news.ycombinator.com/item?id=8098569 https://news.ycombinator.com/item?id=8098569 - July 2014 (18 comments)

behnamoh 1y ago

Is my understanding correct that Lisp's powerful macro system stems from the ability to write the eval function in Lisp itself? From what I gather, Lisp starts with a small set of primitives and special forms—seven in the original Lisp, including lambda. I recall Paul Graham demonstrating in one of his essays that you can build an eval function using just these primitives. Those primitives are typically implemented in a host language like C, but once you have an eval function in Lisp, you can extend it with new rules. The underlying C interpreter only sees the primitives, but as a programmer, you can introduce new syntax rules via eval. This seems like a way to understand macros, where you effectively add new language rules. I know Lisp macros are typically defined using specific keywords like defmacro, but is the core idea similar—extending the language by building on the eval function with new rules?

samth 1y ago
No, macros and eval are quite different. You can see this for example in Python or JavaScript, which have eval but not macros.
- Y_Y 1y ago
  You can make macros in Python: https://github.com/lihaoyi/macropy https://github.com/lihaoyi/macropy (note that that project was started for a class taught by Sussman) There's also a PEP to make them first-class: https://peps.python.org/pep-0638/ https://peps.python.org/pep-0638/
  - sparkie 1y ago
    That's a different meaning of first-class from Strachey's definition of a first-class citizen[1] - ie, one that can be passed as an argument, returned from a function, or assigned to a variable. Syntactic macros are still second-class, like Lisp macros, but an improvement over text-replacement style macros. For something macro-like which is first-class, there are fexprs[2] and operatives (from Kernel[3]) - these receive their operands verbatim, like macros, so they don't require quotation if we want to suppress evaluation. fexprs/Operatives can be passed around like any other value at runtime. [1]:https://en.wikipedia.org/wiki/First-class_citizen https://en.wikipedia.org/wiki/First-class_citizen [2]:https://en.wikipedia.org/wiki/Fexpr https://en.wikipedia.org/wiki/Fexpr [3]:https://web.cs.wpi.edu/~jshutt/kernel.html https://web.cs.wpi.edu/~jshutt/kernel.html
    - Y_Y 1y ago
      Stratchey defined "first-class objects". This was by analogy with "first-class citizens" in a legal/political sense, since they are treated just as well as any other object and have no additional limitations. If we extend the analogy to syntax then I think it's clear enough that it means that it is a piece of syntax which is treated the same as any other and does not require special treatment or impose additional restrictions. Thank you for the clarification and the additional information, I think having macros as first-class objects is a cool (but separate) idea.
- matheusmoreira 1y ago
  They aren't that different. Fexprs are essentially additional eval cases.
sparkie 1y ago
> Is my understanding correct that Lisp's powerful macro system stems from the ability to write the eval function in Lisp itself? I wouldn't say this is the case. Nearly any language could implement an `eval` for itself, but obviously, it is much simpler in Lisps because there's little syntax and few rules. What makes Lisp macros different from say, C preprocessor macros, is the body of the macro is just Lisp code - so the "preprocessor" in this case has full access to the host languages facilities, which may include `eval`. The macros don't take textual input, but they take structured input in the form of S-expressions. Implementing macros is obviously simpler due to eval, because we need to run the evaluator on the macro body, but it's not a strict requirement, as macro functionality could be provided by the implementation and could encapsulate its own evaluator. Lisp macros are also simple due to the fact that Lisp code is just lists of data - you don't have to navigate a complex AST to walk through the code and emit particular syntax. You walk through the input with `car` and `cdr`, and you emit new syntax with `cons` (or `list`/`list*` which are derived from it). Macros can take code as their argument, and produce new code which is evaluated in place. Macros still have hygiene issues though, because they're based on expanding code before evaluating it, variables used in macros can accidentally shadow variables in the scope of the macro's caller. There are workarounds (gensym) to navigate these hygiene problems. > From what I gather, Lisp starts with a small set of primitives and special forms—seven in the original Lisp, including lambda. I recall Paul Graham demonstrating in one of his essays that you can build an eval function using just these primitives. This is largely a theoretical demonstration but not real-world usage. In practice, Lisps have dozens or hundreds of "primitives". Common Lisp in particular is a big language and not trivial to implement. Scheme is a bit smaller, though r6rs started to also grow quite large, but this approach was revisited in r7rs (current), which aims for a small core, with additional functionality being provided through SRFIs (Scheme requests for implementation). > Those primitives are typically implemented in a host language like C, but once you have an eval function in Lisp, you can extend it with new rules. Using Scheme as an example, some SRFIs can be implemented purely in Scheme, as libraries, but others require the implementation to provide support, which often requires writing C code to provide them. > This seems like a way to understand macros, where you effectively add new language rules. I know Lisp macros are typically defined using specific keywords like defmacro As you note, it's `defmacro`, or `macro`, or `syntax-rules`, `syntax-case`, etc, which introduce new syntax - not eval in particular. Some macros will use `eval` in their bodies, which permits control of evaluation other than the regular applicative form of lambdas. Macros are more than just `eval`. They're a multi-stage evaluation model where we first need to do some `macroexpand` (which will internally use `eval`), and afterwards the the resulting expression from the macro call is evaluated. > but is the core idea similar—extending the language by building on the eval function with new rules? There are some Lisps which still attempt this kind of minimalism. One example is Ian Piumarta's Maru[1], which support extending `eval` (and `apply`) with new functionality at runtime based on the type being evaluated. Maru basically has global maps of type->evaluator and type->applicator, where we can add new pairs to at runtime and augment the behavior of `eval`/`apply`. Kernel[2] also aims for the minimalist approach and does away with macros, quote and special-forms entirely, instead replacing them with a more general feature called an operative. The Kernel evaluator does not need to implement special rules for things like `lambda`, `cond`, `car`, `cdr` (as in Graham's "On Lips" essay) - but it just discriminates two forms - operative or applicative. Obviously, some kinds of operative are "primitive", but there's no difference from the PoV of the programmer. Which set of symbols you decide to implement as primitive is up to the implementation. The Kernel report suggests a small set of primitives and demonstrates the remaining standard features can be implemented using only the functionality provided so far. [1]:https://piumarta.com/software/maru/ https://piumarta.com/software/maru/ [2]:https://web.cs.wpi.edu/~jshutt/kernel.html https://web.cs.wpi.edu/~jshutt/kernel.html
kazinator 1y ago
The relationship is this: the ease of writing a meta-circular eval in Lisp, and the ease of writing macros are related by common causes.

taeric 1y ago

A giant difference between eval in most languages versus lisp, is that eval in lisp doesn't just take in a string. Sounds somewhat subtle, but it can lead to some dramatic differences. And is, in at least a small part, why lisp people claim code is data. (I had a fun time exploring this a long time ago in https://taeric.github.io/CodeAsData.html https://taeric.github.io/CodeAsData.html)

kevin_thibedeau 1y ago
Lisp was supposed to ingest M-expressions and convert them internally. Eval would have been more like other languages had that plan not been abandoned. It is not meaningfully different for languages that construct an AST from a string before evaluation.
- taeric 1y ago
  I'm not sure I follow? Presuming that the M-expressions were also made of atoms, they would still be meaningfully different than taking in the string of text. Closer to the same, agreed, but still not the same.
  - spauldo 1y ago
    IIRC M-expressions broke homoiconicity, so you couldn't just spit sexps at eval and expect anything sensible. I believe that's a lot of the reason they never got implemented.
    - taeric 1y ago
      I'm still not sure I understand. It isn't like you can just throw sexps at an eval and expect something sensible, either? As a sibling was showing, calling eval on (1 2 3) will probably not be anything useful. Seems that what makes strings terrible is that they don't really encode any other structure, already. Contrast this with atoms and lists, which do encode a bit of structure. It isn't just a string of characters, it is a list of atoms and lists. Structure is already part of the thing. It does not have to be parsed out. If m-expressions would break that, I think I can see your point. But I'm confused on what would be the point of m-expressions, if they are largely just to remove structure from the content.

TOGoS 1y ago

In summary: The more layers of interpretation your `eval` function needs to do (assumption of source language, tokenization/parsing rules, symbol definitions, etc), the less well-defined the semantics are. So on the less-well-defined end, you have something like JavaScript's `eval` function, in the middle you have lisp macros, which get an AST instead of just source code, and on the actually-pretty-well-defined end, you have a lambda that's already been parsed/compiled and just needs to be evaluated.

programLyrique 1y ago

About how eval is used in other languages: - in Javascript: https://link.springer.com/chapter/10.1007/978-3-642-22655-7_4 https://link.springer.com/chapter/10.1007/978-3-642-22655-7_... - in R: https://dl.acm.org/doi/abs/10.1145/3485502 https://dl.acm.org/doi/abs/10.1145/3485502

curtisszmania 1y ago

[dead]

neilv 1y ago

I'm glad Matthew Flatt wrote this. There is a problem of newbies who read some academic textbook ("metacircular evaluator!") and then immediately try to use the eval feature when it doesn't make sense. (Also, there's a more general problem of newbies trying to use the most advanced tool they've been exposed to. Why use the standard `if` statement, when you can use a proprietary high-powered pattern-matching form, now backed by a networked Kubernetes cluster of deep learning and LLMs.) In the Racket community, one of my many attempts to discourage mistaken uses of `eval`: https://groups.google.com/g/racket-users/c/Z-IlF24RAKU/m/3h6pjUD1J-4J https://groups.google.com/g/racket-users/c/Z-IlF24RAKU/m/3h6... In the all-in-one practitioner's book I was writing, eval wouldn't be introduced until almost the end, in the "Dangerous Last Resort" part of the book. (Maybe I should've planned a marketing gimmick around it, as "free bonus DLC", and you need to do some ritual to be a Certified Certifiable Racketeer, before you can read the eval secret scroll.)

deleted 1y ago

[deleted]

kazinator 1y ago

The sandboxing of eval is something that can be left to a layer on top. You certainly don't want eval itself to be gutted of functionality; nothing less than he full language, thank you. eval can be sandboxed with a combination of multiple approaches, including validating the code, and evaluating it in a restricted environment in which only certain symbols exist. The full library package is not visible.