9 ms·

Python vs Common Lisp, workflow and ecosystem (2019)

135 points by meistro 5y ago

heisig 5y ago
Speaking of Common Lisp - the European Lisp Symposium starts tomorrow (May 3 and May 4, https://european-lisp-symposium.org/2021/index.html https://european-lisp-symposium.org/2021/index.html). The entire conference will be broadcast on Twitch. Python programmers are invited, too :)
- ollran 5y ago
  Thanks for sharing this. Call-site optimization for Common Lisp seems quite interesting after looking at the article. http://metamodular.com/SICL/call-site-optimization.pdf http://metamodular.com/SICL/call-site-optimization.pdf
- jes 5y ago
  This is exciting. Thanks for the reference.
gustavo-fring 5y ago
This might seem slightly unrelated, but I was reading Elixir in Action and one of the statements is along the lines of a debugger being difficult to use in its naturally concurrent environment. The Elixir strategy is to kill erroring processes, capturing their exit signals with supervisor processes and then possibly recreating a replacement process. Can the common lisp condition system be adapted to Elixir? Is there an advantage to doing so? Is there some obvious tradeoff between the two I'm not expressing? Thanks. see this thread form HN for more about adapting the condition system elsewhere. https://news.ycombinator.com/item?id=26852309 https://news.ycombinator.com/item?id=26852309
- alpaca128 5y ago
  Elixir is based on the BEAM virtal machine developed for Erlang. Restarting crashed (very lightweight) processes to handle errors is the normal way of doing things in that system. LFE(Lisp-flavored Erlang) is an existing language that combines Lisp syntax with Erlang's backend, though I haven't used it myself yet.
PeterStuer 5y ago
I like Lisp, and I'm not a fan of e.g. Python's whitespace sensitivity. That said, for niches such as ML and data science, I find you just can't beat the Python ecosystem.
- rhlap 5y ago
  Are people actually doing science with Python or are they talking about doing science? There's so much buggy low quality stuff in that space that I'd write a serious application in C or C++ from scratch. It would be a custom application, sure, but not everything needs to be general. Also, I find Lisp much more natural for mathematical reasoning.
  - beforeolives 5y ago
    > Are people actually doing science with Python or are they talking about doing science? People are actually doing it. And a lot of it too. Both in terms of data science (as a broad term that can mean a bunch of different things) and in terms of computation for specific scientific fields like physics or biology.
  - gustavo-fring 5y ago
    Yeah, they're very much doing it. Pandas is huge, libraries like Spacy, NetworkX, etc exist. It's a massive and good ecosystem. Python is the goto for scientific computing in most of the sciences for newer students I'd hazard a guess over the older R and Julia. This will be blindingly obvious if you work in that area. Yes, you can do it in another language, but you're missing out on a lot of stuff that is just done and is state of the art and is fast because the speedy parts aren't in Python. The complaints about parens for lisp are superficial, but it's my experience the same same goes for whitespace in Python. They just don't matter.
    - tluyben2 5y ago
      > and is fast because the speedy parts aren't in Python. Having worked months with a slew of senior data scientists, this was a bit painful. Python is so slow and those data scientists were very good at coming up with solutions for the issues of the company, but the implementations (using Spacy, Pandas and other libs) had enough Python in them to make them not practical for the company use case. Nice prototypes which I then had to fix them or even rewrite to C/C++(we worked Rust as well to try it out) to make them usable in the company data pipeline. I think companies are burning millions (billions in total?) on depressingly slow solutions in this space by throwing massive power at it all to make them complete their computations before the sun dies out. Example: we needed a specific keyword extraction algorithm for multiple languages; my colleague used Spacy and Python to create it. It took a couple of seconds per page of text; we needed max a few ms on modern hardware. He spent quite a lot of time rewriting and changing it, but never got it under 1s per page on xlarge aws instances. My version takes a few ms on average executing the same algorithm but in optimised c/c++. Sure we could've spun up a lot more instances, but my rewrite was far cheaper than that, even in the first month.
BiteCode_dev 5y ago
He forgot: in python you read the code of somebody else and it's familiar. In lisp, you are learning a new language with every lib because each author think they are a god language designer and that their macro rock. Also they don't need a good doc cause they are obvious. Or good error message cause they never break. Also the way the author dismiss the number gap of packages available is ignoring the elephant in the room.
- diggan 5y ago
  > In lisp, you are learning a new language with every lib because each author think they are a god language designer and that their macro rock. Also they don't need a good doc cause they are obvious. Or good error message cause they never break. You have any specific examples of Common Lisp libraries you found hard to understand? I had a hard time when first learning the language, but once I got proficient to write my own code, I found it easy to understand most libraries I ended up using myself, the same as any language really. That the REPL makes it so easy to explore them with your own context, helped a lot as well. > Also the way the author dismiss the number gap of packages available is ignoring the elephant in the room. In the very same section, the author describes why the number of package don't matter as much as you think it does. Curation VS free-for-all-publishing (like APT vs NPM). Add together that Common Lisp "the language" has been stable for decades, makes it much more possible to be able to use any of the libraries you find as well, where in the Python world, we both know this not to be true (just Python2 VS Python3 makes this a whole other world of messes).
  - p_l 5y ago
    There are some famous examples[1], but to be honest, they are rare outliers for libraries published. And even for internal use, the amount of "local custom style" seems pretty minimal in my experience. [1] Off the top of my head, I can recall Cells (early reactive/dataflow system - weirdness included symbol names made to sort first in Allegro CL IDE) and hu.dwim.* stuff which had its own wrapper around CL:DEFUN and CL:DEFMETHOD, iirc. But I successfully used their stuff without caring about that.
    - 5y ago
mvanbaak 5y ago
python != lisp. It's really that simple.
- dunefox 5y ago
  What about 1 != 2?
prionassembly 5y ago
Obligatory reference to Hy / Hylang, a Lisp that compiles to the Python AST.
- gilch 5y ago
  And Hissp, a Lisp which compiles to Python expressions.
whalesalad 5y ago
I use hot reloading with an iPython repl. I write my code in such a way that I can interact with any individual part of the system via a REPL. Lisp excels here, but you can have a decent approximation of a real-time evaluation loop going.
agumonkey 5y ago
Not to start a flamewar, every time I see a python talk (pycon or else), with fancy tricks like metaclasses.. all I can think is that, well, CLOS would have been perfectly fit for this too. I know people are tired of the "lisp/smalltalk did it better" but what features of python are not possible (or hard) in CL[OS] ? ps: how many CL shops are out there ? I'd work near free just to try a CL team once.
- dasyatidprime 5y ago
  CLOS (as you presumably know) models method application as calls to generic functions, instead of the now-more-mainstream Smalltalk-like message-dispatch approach which Python uses. The latter allows for things like overriding __getattr__ to intercept ‘all’ method calls and property accesses, for which I don't think there's any equivalent in CLOS. The way methods are ‘attached’ to classes gives you a natural form of type-directed name lookup. CL generics have the advantage that you can define your own methods on existing classes while naming the methods in your own package so they don't conflict, but also have a curse of inconvenience along the way where importing a class doesn't naturally pull in everything associated with it, and you wind up writing the class name again and again when dealing with fields of an object. with-slots et al. are poor substitutes. (In an experimental sublanguage at one point I actually had local variables with object type declarations implicitly look up the class using the MOP and symbol-macrolet every available var.slot combination within the scope as a brute hack around the most common desirable case.) Python's short infix/prefix operators are naturally generic, since they're implemented as method calls. In CL there's the generic-cl extension, but I haven't seen it have that much uptake… in particular, any library code that isn't explicitly aware of it won't use it ‘naturally’ on foreign objects, which could be good or bad. That shades into the very-concrete type system that CL starts out with, where any attempt at ad-hoc polymorphic interop is a disaster unless everyone already agrees on what methods to use. I can't make a thing that acts like a hash table but uses a different implementation underneath, then pass it to something that expects to be able to gethash on it. I especially seem to get bitten by this in cases where alists are the expected way of representing key-value maps: there's no way to extricate yourself from the linear search without rewriting every piece of code that touches it, there's often an implicit contract that you don't want duplicate keys but it's easy to violate by accident and create bad behavior down the line, and so on. By comparison, Java collections in particular got this very right in terms of decoupling intention from implementation, and Python does basically the same thing but with a looser set of ‘expected’ methods. By default, Python objects have a ‘purely’ dynamic set of properties, rather than the fixed slots CLOS imputes on an object via its class. Indeed the class-level property one can set in Python to constrain this for possible performance gains is called __slots__.
mark_l_watson 5y ago
Nice article. One thing, re “In Python we typically restart everything at each code change“: I sometimes run Python in Emacs with a REPL. I evaluate region to pick up edits. Not bad. The big win for the Common Lisp REPL is being able to modify data, do restarts, etc. I usually use Common Lisp, but for right now I am heavily using Clojure to write examples for a new Clojure AI book that I am writing. I miss the Common Lisp REPL!
- lukashrb 5y ago
  The first time I encountered "interactive development" was when I had to use python with jupiter notebook. I really liked this style of development. The tdd approach I practiced before was somewhat similar but by far not as visual. Clojure was an eye opener for me and I think it offers a great developer experience (e.g. I'm addicted to C-c C-p) It seems my journey hasn't ended and I definitely have to check out CL! But atm it's hard for me to give up the things clojure offers to me: persistent datastructures, access to a great ecosystem and a very good designed standard library.
wexq 5y ago
I'm not a particularly experienced or good Lisp programmer, I can customize Emacs, but that's pretty much it. However, I think this article is a bit skewed and not highlighting things Python has. For instance, the standard library means that Python is more usable out of the box. Also when you've got things like iPython or Jupyter it means you can get off the ground easily. So, in the end, they're two different languages, and I do not think either is better of the two. Right tool for the job and all that.
- mplanchard 5y ago
  This point seems to be addressed in the “State of the Libraries” section[0] [0]: https://lisp-journey.gitlab.io/pythonvslisp/#state-of-the-libraries https://lisp-journey.gitlab.io/pythonvslisp/#state-of-the-li...
  - wexq 5y ago
    Not really, the basic Python install has a lot of things ready to use without any external libraries.
rahimiali 5y ago
There is popular python environment where you do not restart the environment after an edit: Jupyter notebooks. This comes with its own set of problems, which CL also has, but I still find notebooks a worthwhile mindset for writing applications. There’s no need to equate python with the script mindset.
slifin 5y ago
If you like lisp and you like python you might like libpython-clj

135 points by meistro 5y ago

Speaking of Common Lisp - the European Lisp Symposium starts tomorrow (May 3 and May 4, https://european-lisp-symposium.org/2021/index.html https://european-lisp-symposium.org/2021/index.html). The entire conference will be broadcast on Twitch. Python programmers are invited, too :)

ollran 5y ago
Thanks for sharing this. Call-site optimization for Common Lisp seems quite interesting after looking at the article. http://metamodular.com/SICL/call-site-optimization.pdf http://metamodular.com/SICL/call-site-optimization.pdf
jes 5y ago
This is exciting. Thanks for the reference.

gustavo-fring 5y ago

This might seem slightly unrelated, but I was reading Elixir in Action and one of the statements is along the lines of a debugger being difficult to use in its naturally concurrent environment. The Elixir strategy is to kill erroring processes, capturing their exit signals with supervisor processes and then possibly recreating a replacement process. Can the common lisp condition system be adapted to Elixir? Is there an advantage to doing so? Is there some obvious tradeoff between the two I'm not expressing? Thanks. see this thread form HN for more about adapting the condition system elsewhere. https://news.ycombinator.com/item?id=26852309 https://news.ycombinator.com/item?id=26852309

alpaca128 5y ago
Elixir is based on the BEAM virtal machine developed for Erlang. Restarting crashed (very lightweight) processes to handle errors is the normal way of doing things in that system. LFE(Lisp-flavored Erlang) is an existing language that combines Lisp syntax with Erlang's backend, though I haven't used it myself yet.

PeterStuer 5y ago

I like Lisp, and I'm not a fan of e.g. Python's whitespace sensitivity. That said, for niches such as ML and data science, I find you just can't beat the Python ecosystem.

rhlap 5y ago
Are people actually doing science with Python or are they talking about doing science? There's so much buggy low quality stuff in that space that I'd write a serious application in C or C++ from scratch. It would be a custom application, sure, but not everything needs to be general. Also, I find Lisp much more natural for mathematical reasoning.
- beforeolives 5y ago
  > Are people actually doing science with Python or are they talking about doing science? People are actually doing it. And a lot of it too. Both in terms of data science (as a broad term that can mean a bunch of different things) and in terms of computation for specific scientific fields like physics or biology.
- gustavo-fring 5y ago
  Yeah, they're very much doing it. Pandas is huge, libraries like Spacy, NetworkX, etc exist. It's a massive and good ecosystem. Python is the goto for scientific computing in most of the sciences for newer students I'd hazard a guess over the older R and Julia. This will be blindingly obvious if you work in that area. Yes, you can do it in another language, but you're missing out on a lot of stuff that is just done and is state of the art and is fast because the speedy parts aren't in Python. The complaints about parens for lisp are superficial, but it's my experience the same same goes for whitespace in Python. They just don't matter.
  - tluyben2 5y ago
    > and is fast because the speedy parts aren't in Python. Having worked months with a slew of senior data scientists, this was a bit painful. Python is so slow and those data scientists were very good at coming up with solutions for the issues of the company, but the implementations (using Spacy, Pandas and other libs) had enough Python in them to make them not practical for the company use case. Nice prototypes which I then had to fix them or even rewrite to C/C++(we worked Rust as well to try it out) to make them usable in the company data pipeline. I think companies are burning millions (billions in total?) on depressingly slow solutions in this space by throwing massive power at it all to make them complete their computations before the sun dies out. Example: we needed a specific keyword extraction algorithm for multiple languages; my colleague used Spacy and Python to create it. It took a couple of seconds per page of text; we needed max a few ms on modern hardware. He spent quite a lot of time rewriting and changing it, but never got it under 1s per page on xlarge aws instances. My version takes a few ms on average executing the same algorithm but in optimised c/c++. Sure we could've spun up a lot more instances, but my rewrite was far cheaper than that, even in the first month.

BiteCode_dev 5y ago

He forgot: in python you read the code of somebody else and it's familiar. In lisp, you are learning a new language with every lib because each author think they are a god language designer and that their macro rock. Also they don't need a good doc cause they are obvious. Or good error message cause they never break. Also the way the author dismiss the number gap of packages available is ignoring the elephant in the room.

diggan 5y ago
> In lisp, you are learning a new language with every lib because each author think they are a god language designer and that their macro rock. Also they don't need a good doc cause they are obvious. Or good error message cause they never break. You have any specific examples of Common Lisp libraries you found hard to understand? I had a hard time when first learning the language, but once I got proficient to write my own code, I found it easy to understand most libraries I ended up using myself, the same as any language really. That the REPL makes it so easy to explore them with your own context, helped a lot as well. > Also the way the author dismiss the number gap of packages available is ignoring the elephant in the room. In the very same section, the author describes why the number of package don't matter as much as you think it does. Curation VS free-for-all-publishing (like APT vs NPM). Add together that Common Lisp "the language" has been stable for decades, makes it much more possible to be able to use any of the libraries you find as well, where in the Python world, we both know this not to be true (just Python2 VS Python3 makes this a whole other world of messes).
- p_l 5y ago
  There are some famous examples[1], but to be honest, they are rare outliers for libraries published. And even for internal use, the amount of "local custom style" seems pretty minimal in my experience. [1] Off the top of my head, I can recall Cells (early reactive/dataflow system - weirdness included symbol names made to sort first in Allegro CL IDE) and hu.dwim.* stuff which had its own wrapper around CL:DEFUN and CL:DEFMETHOD, iirc. But I successfully used their stuff without caring about that.
  - 5y ago

mvanbaak 5y ago

python != lisp. It's really that simple.

dunefox 5y ago
What about 1 != 2?

prionassembly 5y ago

Obligatory reference to Hy / Hylang, a Lisp that compiles to the Python AST.

gilch 5y ago
And Hissp, a Lisp which compiles to Python expressions.

whalesalad 5y ago

I use hot reloading with an iPython repl. I write my code in such a way that I can interact with any individual part of the system via a REPL. Lisp excels here, but you can have a decent approximation of a real-time evaluation loop going.

agumonkey 5y ago

Not to start a flamewar, every time I see a python talk (pycon or else), with fancy tricks like metaclasses.. all I can think is that, well, CLOS would have been perfectly fit for this too. I know people are tired of the "lisp/smalltalk did it better" but what features of python are not possible (or hard) in CL[OS] ? ps: how many CL shops are out there ? I'd work near free just to try a CL team once.

dasyatidprime 5y ago
CLOS (as you presumably know) models method application as calls to generic functions, instead of the now-more-mainstream Smalltalk-like message-dispatch approach which Python uses. The latter allows for things like overriding __getattr__ to intercept ‘all’ method calls and property accesses, for which I don't think there's any equivalent in CLOS. The way methods are ‘attached’ to classes gives you a natural form of type-directed name lookup. CL generics have the advantage that you can define your own methods on existing classes while naming the methods in your own package so they don't conflict, but also have a curse of inconvenience along the way where importing a class doesn't naturally pull in everything associated with it, and you wind up writing the class name again and again when dealing with fields of an object. with-slots et al. are poor substitutes. (In an experimental sublanguage at one point I actually had local variables with object type declarations implicitly look up the class using the MOP and symbol-macrolet every available var.slot combination within the scope as a brute hack around the most common desirable case.) Python's short infix/prefix operators are naturally generic, since they're implemented as method calls. In CL there's the generic-cl extension, but I haven't seen it have that much uptake… in particular, any library code that isn't explicitly aware of it won't use it ‘naturally’ on foreign objects, which could be good or bad. That shades into the very-concrete type system that CL starts out with, where any attempt at ad-hoc polymorphic interop is a disaster unless everyone already agrees on what methods to use. I can't make a thing that acts like a hash table but uses a different implementation underneath, then pass it to something that expects to be able to gethash on it. I especially seem to get bitten by this in cases where alists are the expected way of representing key-value maps: there's no way to extricate yourself from the linear search without rewriting every piece of code that touches it, there's often an implicit contract that you don't want duplicate keys but it's easy to violate by accident and create bad behavior down the line, and so on. By comparison, Java collections in particular got this very right in terms of decoupling intention from implementation, and Python does basically the same thing but with a looser set of ‘expected’ methods. By default, Python objects have a ‘purely’ dynamic set of properties, rather than the fixed slots CLOS imputes on an object via its class. Indeed the class-level property one can set in Python to constrain this for possible performance gains is called __slots__.

mark_l_watson 5y ago

Nice article. One thing, re “In Python we typically restart everything at each code change“: I sometimes run Python in Emacs with a REPL. I evaluate region to pick up edits. Not bad. The big win for the Common Lisp REPL is being able to modify data, do restarts, etc. I usually use Common Lisp, but for right now I am heavily using Clojure to write examples for a new Clojure AI book that I am writing. I miss the Common Lisp REPL!

lukashrb 5y ago
The first time I encountered "interactive development" was when I had to use python with jupiter notebook. I really liked this style of development. The tdd approach I practiced before was somewhat similar but by far not as visual. Clojure was an eye opener for me and I think it offers a great developer experience (e.g. I'm addicted to C-c C-p) It seems my journey hasn't ended and I definitely have to check out CL! But atm it's hard for me to give up the things clojure offers to me: persistent datastructures, access to a great ecosystem and a very good designed standard library.

wexq 5y ago

I'm not a particularly experienced or good Lisp programmer, I can customize Emacs, but that's pretty much it. However, I think this article is a bit skewed and not highlighting things Python has. For instance, the standard library means that Python is more usable out of the box. Also when you've got things like iPython or Jupyter it means you can get off the ground easily. So, in the end, they're two different languages, and I do not think either is better of the two. Right tool for the job and all that.

mplanchard 5y ago
This point seems to be addressed in the “State of the Libraries” section[0] [0]: https://lisp-journey.gitlab.io/pythonvslisp/#state-of-the-libraries https://lisp-journey.gitlab.io/pythonvslisp/#state-of-the-li...
- wexq 5y ago
  Not really, the basic Python install has a lot of things ready to use without any external libraries.

rahimiali 5y ago

There is popular python environment where you do not restart the environment after an edit: Jupyter notebooks. This comes with its own set of problems, which CL also has, but I still find notebooks a worthwhile mindset for writing applications. There’s no need to equate python with the script mindset.

slifin 5y ago

If you like lisp and you like python you might like libpython-clj