7 ms·

Femtolisp: A lightweight, robust, scheme-like Lisp implementation

146 points by lelf 7y ago

arexxbifs 7y ago
I came to read about the lisp implementation, but I stayed for the "cutef8" pun among his other repos.
cultus 7y ago
The parser of the Julia compiler is written in this dialect. It just goes to show how much power you have in a small package with lisps.
- marmaduke 7y ago
  You mean, how much you can develop during a PhD on scientific computing with a known technical computing syntax and paradigm on top of a modular compiler infrastructure with an enthusiastic community? (Not to detract from Julia devs but there were plenty of Lisp based contenders ready for scientific computing before so its useful to ask why Julia why now etc ) *edit: this was kinda snarky but I actually meant a compliment on the excellent strategic choices Julia devs made as compared to other scientific computing attempts
  - forgotpwd16 7y ago
    >Lisp based contenders ready for scientific computing Any examples?
    - xiaodai 7y ago
      Incanter
    - flavio81 7y ago
      >Any examples? Common Lisp has been used in scientific computing since decades, so in some sense it's "ready". Also, see CLASP: lisp at the state of the art of computational chemistry.
    - phillc73 7y ago
      cl-ana, Common Lisp data analysis library with emphasis on modularity and conceptual clarity - https://github.com/ghollisjr/cl-ana https://github.com/ghollisjr/cl-ana numcl, a numpy clone in Common Lisp - https://github.com/numcl/numcl https://github.com/numcl/numcl Clojure kixi-stats- https://cljdoc.org/d/kixi/stats/0.5.0/doc/readme https://cljdoc.org/d/kixi/stats/0.5.0/doc/readme and the book Clojure for Data Science - http://clojuredatascience.com/ http://clojuredatascience.com/ Clojure's Incanter has been mentioned already - http://incanter.org/ http://incanter.org/ OWL, the OCaml Scientific Computing project - https://ocaml.xyz/ https://ocaml.xyz/ And probably not quite what you're looking for, but there is a Racket data frame structure which provides some data science like manipulation capabilities - https://alex-hhh.github.io/2018/08/racket-data-frame.html https://alex-hhh.github.io/2018/08/racket-data-frame.html Appreciate some of the above are Schemes and not Lisps, but they may still be interesting.
    - marmaduke 7y ago
      Lush is the one I had looked at long ago, http://lush.sourceforge.net/ http://lush.sourceforge.net/
  - ddragon 7y ago
    I'll go and say that, outside of the great work from the creators in the development and marketing of the language, the main difference between those Lisp based languages and Julia in terms of attracting people is the syntax. Julia focuses on being approachable to Python, R, Fortran and Matlab programmers (since those already "won" by the time Julia was created), and it wouldn't be able to convince any significant number of those to even check the language without looking like Python, R, Fortran and Matlab (even when S-expr is indeed more elegant and powerful). Julia base syntax is closer to how people write math, so it's more familiar even to people who are not dedicated programmers which were part of Julia's target audience. And when people discover the lispiness in Julia which is a thin layer below the surface syntax (such as the metaprogramming system and multiple dispatch) they at least already gave the language a fairer chance. Other languages that borrowed from Lisp but didn't go, to different degrees, with s-expr like Scala and Clojure (plus R) also had success in the data science/ML area. Hopefully Nim also gets a foothold, and possibly Racket with optional infix syntax.
    - flavio81 7y ago
      >the main difference between those Lisp based languages and Julia in terms of attracting people is the syntax Yes, and this is practically the only thing that I find wrong with Julia, besides the lack of the interactivity of CL. It's like populism -- an inferior syntax choice taken just to attract the Pythonistas.
      - chrispeel 7y ago
        I guess you mean you prefer prefix rather than infix notation. Do you have any specific technical reasons for this?
        jacobwilliamroy 7y ago
        I dislike operator precedence, and I also think that putting all the plus signs in between the addends is a waste time. I like being able to do this: (+ 1 2 3 4 5)
  - ihnorton 7y ago
    Is there a Lisp I can download, for Windows, that provides a linear algebra problem solving environment* out of the box or from a package manager? Ideally linked to a reasonably optimized BLAS implementation. * this is intentionally under-specified, but to me it means some nontrivial subset of matlab/numpy functionality
    - StefanKarpinski 7y ago
      I’m sure we can rig a way to call from Julia’s embedded lisp mode back into Julia for you. Then you’ll be all set :D
      - ihnorton 7y ago
        https://github.com/JuliaLang/julia/pull/17108 https://github.com/JuliaLang/julia/pull/17108 only implements the REPL part but it’s a start. “Downloadable Windows LA REPL” was a criteria Julia met within ~6-8 months of public release, and it was one of the reasons I got involved even though I only ever use(d) windows at work (...mostly for scientific instrumentation which shipped with and only supported windows). This is slowly changing, but IMO windows support is still table stakes for a scientific computing environment, and seeing windows support meant both that I could use it where I most needed it, and signaled that Julia had potential for real traction (not abandoning >70% of users out of the gate). I think windows support is also one of the reasons Python/NumPy got to be where it is — it wasn’t always perfect, but they at least cared about windows, more so than most other scripting languages in the mid-2000s. Ditto CMake.
    - dTal 7y ago
      Python + Hy + Numpy In all seriousness that's probably the quickest way, if by 'lisp' you mean 's-expressions'. For a real lisp, maybe these Numpy bindings[0] for Chez Scheme? Although the py-* names look a bit awkward. You could easily rebind those, but maybe that's not "out of the box" enough. [0] https://github.com/guenchi/NumPy https://github.com/guenchi/NumPy
- orbifold 7y ago
  Which is probably one of the reasons why Julia takes ages to do a simple plot.
  - jpfr 7y ago
    That’s LLVM as a JIT compiler. In pure byte-code interpreter mode, Julia is quite more responsive.
    - gugagore 7y ago
      There's also compilation that happens in Julia. Like the type inference. Which is to say, the time isn't just spent in LLVM.
  - gugagore 7y ago
    No, that's not why. You think there's considerable time spent in parsing? It's all code generation that takes time, which is the same thing that makes Julia powerful.
- omaranto 7y ago
  A femtolisp REPL is available from Julia by running: julia --lisp
akkartik 7y ago
"Bytecode is first-class, can be printed and read, and is human readable (the representation is a string of normal low-ASCII characters)." Anyone know where I can read more about the VM and its human-readable syntax? I'm not immediately spotting it in the repo.
- KenoFischer 7y ago
  It's not really documented, but human readable, may also not really mean what you think it means. For example, here's a snippet from the printed bytecode of the default system image: #fn("8000r1e0~|[316>0\x7fM~|[i2i343;];" [closure?])] find-in-f) #fn(";000r2c0~|}q3c1tg66I0c2e3c4c5e6g63132c73241;c8;" [#fn("8000r0c0c1~\x7fq2i2322^;" [#fn(for-each) #fn("8000r1~M|\x7f_43;" [])]) #fn("6000r1|F16B02|Mc0<16802|\x84c1<680e2|41;c3|41;" [thrown-value It's human readable in the sense that all the operations have single character ascii opcodes and strings are embedded as is and there's no raw pointers, etc, so you won't get mojibake, but it's not human readable in the sense of say a pretty-printed IR.
- kmill 7y ago
  It looks like it's done here in the print routine: https://github.com/JeffBezanson/femtolisp/blob/master/print.c#L423 https://github.com/JeffBezanson/femtolisp/blob/master/print.... It adds 48 to each byte being printed, which is ASCII '0'.
- uryga 7y ago
  (edit: clarity) i couldn't really find any docs about it either, but compiler.scm is pretty readable and has an instruction listing – their arguments aren't documented explicitly, but you can infer most of it from the code for `compiler::disassemble`, which is quite readable too. the instructions remind me of CPython, pretty standard stack-machine stuff: slots for locals (`loadv 0` where 0 is an index to func_values), loading globals by name (`loadg 0`, same as with values but the value is a symbol to be looked up), relative jumps, things like that. for no reason in particular i spent an evening[0] reverse-engineering enough of the code to be able to parse/display bytecode strings :) here's the python code [1] if you want to try it, though it'd probably be easier to just install femtolisp and use its `disassemble`... as others have mentioned, the serialized form is kind-of readable, and with some practice you could probably even read it directly! e.g "c0" is `loadv 0`, ";" is `ret`, functions always have a header like `r3` indicating 3 arguments, etc. reading it like that would be very tedious but doable in a pinch, and at the very least you can see patterns like the ones i mentioned i've only looked at the bytecode and vm, but if you were more serious about using the language, the `compiler` module seems to have a nice API for creating bytecode. (they just directly expose all the functions the compiler uses!) --- [0] would've gone much faster if i'd known they shift each byte by 48 when serializing... [1] https://gist.github.com/lubieowoce/148406b1dd61d91b1584339ffefbdcc6 https://gist.github.com/lubieowoce/148406b1dd61d91b1584339ff...
Skunkleton 7y ago
The Readme for this repo is very well written. It makes me think that the rest of the project is worth checking out.
- dnautics 7y ago
  the "rest of the project", at this juncture in programming history anyways, turned out to be "the Julia programming Language".
  - Skunkleton 7y ago
    Google seems to indicate that femtolisp is used for some portion of Julia's implementation, and didn't evolve into Julia itself?
    - StefanKarpinski 7y ago
      Yes, but the author of femtolisp is one of the co-creators of and single most prolific contributor to Julia. Spiritually, that is the successor project.
rcarmo 7y ago
Oh, goody, another tiny LISP (I've been keeping track of them at http://taoofmac.com/space/LISP http://taoofmac.com/space/LISP, will add this one ASAP).
- KenoFischer 7y ago
  Not that it matters much, but femtolisp is not a new project. The bulk of the code has been around for at least a decade.
  - rcarmo 7y ago
    Sure, but I hadn't come across it before (or if I did, I forgot), and I like to keep track of such things.
- swlkr 7y ago
  you may be interested in janet as well
  - rcarmo 7y ago
    Just found out about it. Added, thanks!
- fovc 7y ago
  You should quote the README from this one: """ Almost everybody has their own lisp implementation. Some programmers' dogs and cats probably have their own lisp implementations as well. This is great, but too often I see people omit some of the obscure but critical features that make lisp uniquely wonderful. These include read macros like #. and backreferences, gensyms, and properly escaped symbol names. If you're going to waste everybody's time with yet another lisp, at least do it right damnit. """
  - rcarmo 7y ago
    Great idea, done!
- 7y ago
papaf 7y ago
I like the "tiny" version that has a minimal lisp written in C and then builds upon that with this bit of magic: https://github.com/JeffBezanson/femtolisp/blob/master/tiny/system.lsp https://github.com/JeffBezanson/femtolisp/blob/master/tiny/s...
- phyrex 7y ago
  You may also enjoy Clojure's core.clj: https://github.com/clojure/clojure/blob/master/src/clj/clojure/core.clj https://github.com/clojure/clojure/blob/master/src/clj/cloju...
deleted 7y ago
[deleted]

146 points by lelf 7y ago

I came to read about the lisp implementation, but I stayed for the "cutef8" pun among his other repos.

The parser of the Julia compiler is written in this dialect. It just goes to show how much power you have in a small package with lisps.

marmaduke 7y ago
You mean, how much you can develop during a PhD on scientific computing with a known technical computing syntax and paradigm on top of a modular compiler infrastructure with an enthusiastic community? (Not to detract from Julia devs but there were plenty of Lisp based contenders ready for scientific computing before so its useful to ask why Julia why now etc ) *edit: this was kinda snarky but I actually meant a compliment on the excellent strategic choices Julia devs made as compared to other scientific computing attempts
- forgotpwd16 7y ago
  >Lisp based contenders ready for scientific computing Any examples?
  - xiaodai 7y ago
    Incanter
  - flavio81 7y ago
    >Any examples? Common Lisp has been used in scientific computing since decades, so in some sense it's "ready". Also, see CLASP: lisp at the state of the art of computational chemistry.
  - phillc73 7y ago
    cl-ana, Common Lisp data analysis library with emphasis on modularity and conceptual clarity - https://github.com/ghollisjr/cl-ana https://github.com/ghollisjr/cl-ana numcl, a numpy clone in Common Lisp - https://github.com/numcl/numcl https://github.com/numcl/numcl Clojure kixi-stats- https://cljdoc.org/d/kixi/stats/0.5.0/doc/readme https://cljdoc.org/d/kixi/stats/0.5.0/doc/readme and the book Clojure for Data Science - http://clojuredatascience.com/ http://clojuredatascience.com/ Clojure's Incanter has been mentioned already - http://incanter.org/ http://incanter.org/ OWL, the OCaml Scientific Computing project - https://ocaml.xyz/ https://ocaml.xyz/ And probably not quite what you're looking for, but there is a Racket data frame structure which provides some data science like manipulation capabilities - https://alex-hhh.github.io/2018/08/racket-data-frame.html https://alex-hhh.github.io/2018/08/racket-data-frame.html Appreciate some of the above are Schemes and not Lisps, but they may still be interesting.
  - marmaduke 7y ago
    Lush is the one I had looked at long ago, http://lush.sourceforge.net/ http://lush.sourceforge.net/
- ddragon 7y ago
  I'll go and say that, outside of the great work from the creators in the development and marketing of the language, the main difference between those Lisp based languages and Julia in terms of attracting people is the syntax. Julia focuses on being approachable to Python, R, Fortran and Matlab programmers (since those already "won" by the time Julia was created), and it wouldn't be able to convince any significant number of those to even check the language without looking like Python, R, Fortran and Matlab (even when S-expr is indeed more elegant and powerful). Julia base syntax is closer to how people write math, so it's more familiar even to people who are not dedicated programmers which were part of Julia's target audience. And when people discover the lispiness in Julia which is a thin layer below the surface syntax (such as the metaprogramming system and multiple dispatch) they at least already gave the language a fairer chance. Other languages that borrowed from Lisp but didn't go, to different degrees, with s-expr like Scala and Clojure (plus R) also had success in the data science/ML area. Hopefully Nim also gets a foothold, and possibly Racket with optional infix syntax.
  - flavio81 7y ago
    >the main difference between those Lisp based languages and Julia in terms of attracting people is the syntax Yes, and this is practically the only thing that I find wrong with Julia, besides the lack of the interactivity of CL. It's like populism -- an inferior syntax choice taken just to attract the Pythonistas.
    - chrispeel 7y ago
      I guess you mean you prefer prefix rather than infix notation. Do you have any specific technical reasons for this?
      - jacobwilliamroy 7y ago
        I dislike operator precedence, and I also think that putting all the plus signs in between the addends is a waste time. I like being able to do this: (+ 1 2 3 4 5)
- ihnorton 7y ago
  Is there a Lisp I can download, for Windows, that provides a linear algebra problem solving environment* out of the box or from a package manager? Ideally linked to a reasonably optimized BLAS implementation. * this is intentionally under-specified, but to me it means some nontrivial subset of matlab/numpy functionality
  - StefanKarpinski 7y ago
    I’m sure we can rig a way to call from Julia’s embedded lisp mode back into Julia for you. Then you’ll be all set :D
    - ihnorton 7y ago
      https://github.com/JuliaLang/julia/pull/17108 https://github.com/JuliaLang/julia/pull/17108 only implements the REPL part but it’s a start. “Downloadable Windows LA REPL” was a criteria Julia met within ~6-8 months of public release, and it was one of the reasons I got involved even though I only ever use(d) windows at work (...mostly for scientific instrumentation which shipped with and only supported windows). This is slowly changing, but IMO windows support is still table stakes for a scientific computing environment, and seeing windows support meant both that I could use it where I most needed it, and signaled that Julia had potential for real traction (not abandoning >70% of users out of the gate). I think windows support is also one of the reasons Python/NumPy got to be where it is — it wasn’t always perfect, but they at least cared about windows, more so than most other scripting languages in the mid-2000s. Ditto CMake.
  - dTal 7y ago
    Python + Hy + Numpy In all seriousness that's probably the quickest way, if by 'lisp' you mean 's-expressions'. For a real lisp, maybe these Numpy bindings[0] for Chez Scheme? Although the py-* names look a bit awkward. You could easily rebind those, but maybe that's not "out of the box" enough. [0] https://github.com/guenchi/NumPy https://github.com/guenchi/NumPy
orbifold 7y ago
Which is probably one of the reasons why Julia takes ages to do a simple plot.
- jpfr 7y ago
  That’s LLVM as a JIT compiler. In pure byte-code interpreter mode, Julia is quite more responsive.
  - gugagore 7y ago
    There's also compilation that happens in Julia. Like the type inference. Which is to say, the time isn't just spent in LLVM.
- gugagore 7y ago
  No, that's not why. You think there's considerable time spent in parsing? It's all code generation that takes time, which is the same thing that makes Julia powerful.
omaranto 7y ago
A femtolisp REPL is available from Julia by running: julia --lisp

akkartik 7y ago

"Bytecode is first-class, can be printed and read, and is human readable (the representation is a string of normal low-ASCII characters)." Anyone know where I can read more about the VM and its human-readable syntax? I'm not immediately spotting it in the repo.

KenoFischer 7y ago
It's not really documented, but human readable, may also not really mean what you think it means. For example, here's a snippet from the printed bytecode of the default system image: #fn("8000r1e0~|[316>0\x7fM~|[i2i343;];" [closure?])] find-in-f) #fn(";000r2c0~|}q3c1tg66I0c2e3c4c5e6g63132c73241;c8;" [#fn("8000r0c0c1~\x7fq2i2322^;" [#fn(for-each) #fn("8000r1~M|\x7f_43;" [])]) #fn("6000r1|F16B02|Mc0<16802|\x84c1<680e2|41;c3|41;" [thrown-value It's human readable in the sense that all the operations have single character ascii opcodes and strings are embedded as is and there's no raw pointers, etc, so you won't get mojibake, but it's not human readable in the sense of say a pretty-printed IR.
kmill 7y ago
It looks like it's done here in the print routine: https://github.com/JeffBezanson/femtolisp/blob/master/print.c#L423 https://github.com/JeffBezanson/femtolisp/blob/master/print.... It adds 48 to each byte being printed, which is ASCII '0'.
uryga 7y ago
(edit: clarity) i couldn't really find any docs about it either, but compiler.scm is pretty readable and has an instruction listing – their arguments aren't documented explicitly, but you can infer most of it from the code for `compiler::disassemble`, which is quite readable too. the instructions remind me of CPython, pretty standard stack-machine stuff: slots for locals (`loadv 0` where 0 is an index to func_values), loading globals by name (`loadg 0`, same as with values but the value is a symbol to be looked up), relative jumps, things like that. for no reason in particular i spent an evening[0] reverse-engineering enough of the code to be able to parse/display bytecode strings :) here's the python code [1] if you want to try it, though it'd probably be easier to just install femtolisp and use its `disassemble`... as others have mentioned, the serialized form is kind-of readable, and with some practice you could probably even read it directly! e.g "c0" is `loadv 0`, ";" is `ret`, functions always have a header like `r3` indicating 3 arguments, etc. reading it like that would be very tedious but doable in a pinch, and at the very least you can see patterns like the ones i mentioned i've only looked at the bytecode and vm, but if you were more serious about using the language, the `compiler` module seems to have a nice API for creating bytecode. (they just directly expose all the functions the compiler uses!) --- [0] would've gone much faster if i'd known they shift each byte by 48 when serializing... [1] https://gist.github.com/lubieowoce/148406b1dd61d91b1584339ffefbdcc6 https://gist.github.com/lubieowoce/148406b1dd61d91b1584339ff...

Skunkleton 7y ago

The Readme for this repo is very well written. It makes me think that the rest of the project is worth checking out.

dnautics 7y ago
the "rest of the project", at this juncture in programming history anyways, turned out to be "the Julia programming Language".
- Skunkleton 7y ago
  Google seems to indicate that femtolisp is used for some portion of Julia's implementation, and didn't evolve into Julia itself?
  - StefanKarpinski 7y ago
    Yes, but the author of femtolisp is one of the co-creators of and single most prolific contributor to Julia. Spiritually, that is the successor project.

rcarmo 7y ago

Oh, goody, another tiny LISP (I've been keeping track of them at http://taoofmac.com/space/LISP http://taoofmac.com/space/LISP, will add this one ASAP).

KenoFischer 7y ago
Not that it matters much, but femtolisp is not a new project. The bulk of the code has been around for at least a decade.
- rcarmo 7y ago
  Sure, but I hadn't come across it before (or if I did, I forgot), and I like to keep track of such things.
swlkr 7y ago
you may be interested in janet as well
- rcarmo 7y ago
  Just found out about it. Added, thanks!
fovc 7y ago
You should quote the README from this one: """ Almost everybody has their own lisp implementation. Some programmers' dogs and cats probably have their own lisp implementations as well. This is great, but too often I see people omit some of the obscure but critical features that make lisp uniquely wonderful. These include read macros like #. and backreferences, gensyms, and properly escaped symbol names. If you're going to waste everybody's time with yet another lisp, at least do it right damnit. """
- rcarmo 7y ago
  Great idea, done!
7y ago

papaf 7y ago

I like the "tiny" version that has a minimal lisp written in C and then builds upon that with this bit of magic: https://github.com/JeffBezanson/femtolisp/blob/master/tiny/system.lsp https://github.com/JeffBezanson/femtolisp/blob/master/tiny/s...

phyrex 7y ago
You may also enjoy Clojure's core.clj: https://github.com/clojure/clojure/blob/master/src/clj/clojure/core.clj https://github.com/clojure/clojure/blob/master/src/clj/cloju...

deleted 7y ago

[deleted]