9 ms·

Much like the sibling comment I have heard variations of this proposition before. But instead of wondering why don’t more groups take this approach, I want to k

Much like the sibling comment I have heard variations of this proposition before. But instead of wondering why don’t more groups take this approach, I want to know what make it possible in the first place. So, in your experience, or even just via some passes down stories, etc., what is it about Lisp that allows for this efficiency? I have heard, although it is always just a quick statement with no explanation that it is the macros. And then one of siblings children comments seems to support the idea that Lisp programmers as a whole may be more experienced/better/more well rounded developers in general, so maybe that’s it. I am genuinely interested in peoples thoughts on this matter and am not being flippant or sarcastic.

mmmm2 5y ago
I can't speak from a professional level about this, but I'll tell you why I started using Common Lisp for my personal development. One day I was thinking it would be awesome if someone created a really powerful language with great facilities for abstraction: first class functions, objects, meta objects, etc. It should use dynamic typing during development with an excellent REPL and a built in debugger available everywhere, but it should have the ability to add type declarations when the code got more settled. Even better, this language should include ways of hinting to the compiler how to make a chunk of code faster, and if that wasn't enough, you should be able to replace chunks of code seamlessly with highly optimized C code. I thought about how to implement a language like this for a while, but when I looked into it I discovered Common Lisp does most of this already. It's static typing isn't perfect, though now there is https://coalton-lang.github.io/ https://coalton-lang.github.io/ if that's your thing. So that did it for me, but really this is only part of what you can do with Common Lisp. I think it still has a niche in places where the "right thing" isn't know yet. For example, I read a Reddit post saying that Common Lisp would be a poor choice now for a web startup, and that Paul Graham overstates Common Lisp's case as the reason for Viaweb's success. This is probably true in 2022, but I think it's instructive to look at why Paul claimed Lisp was so helpful back in 1996. There was no Rails or Django, no React or single page web apps. Viaweb built Scheme style continuations onto Common Lisp to simulate an SPA before AJAX was a thing, and on top of that created a slick HTML templating language based off lisp macros. To do that in Perl, or C, would have required something like writing a scheme interpreter, and would have taken much longer for a 2 person team. Is that a 100x advantage? Hard to say. Ultimately, I do think it still has advantages, you just have to find the areas where nothing else will really do the job.
evdubs 5y ago
It's really not that much more efficient when you compare Lisps to modern languages that share features like interactive (REPL) development, garbage collection, large standard libraries, concurrency primitives and abstractions, object/functional programming styles, integrated package managers, etc. When you compare modern languages, including Lisps, to something like C99, I think it's easier to see how you can be much more efficient/productive. If you take away the similarities of Lisps and modern languages, you're mostly left with the syntax and how the syntax enables macros that set Lisps apart from other languages. But, most of your Lisp code looks very much like your other code where you're writing functions, defining structures, calling object methods, etc. so it seems a stretch that the syntax and macros alone make you much more efficient/productive.
- duped 5y ago
  > It's really not that much more efficient when you compare Lisps to modern languages that share features like interactive (REPL) development, garbage collection, large standard libraries, concurrency primitives and abstractions, object/functional programming styles, integrated package managers, etc. pick 2! But seriously, what modern languages have all these features?
  - evdubs 5y ago
    Java/JVM languages, C#/CLR languages, Python and Ruby (minus real threads, but compare to Racket with not-the-best real threading story), NodeJS (minus the "large" standard library, but compare to Scheme with a small standard library), can probably include Go
    - reikonomusha 5y ago
      I don't think any of these have REPL-driven development. They might have a "REPL" but not interactive code reloading as a part of the development cycle as is meant by a Lisp REPL.
      - School-Cotton 5y ago
jrumbut 5y ago
Lisp projects also tend to keep the dependency list short. In the best case scenario, once the foundation is laid and the project has momentum, you end up with what feels like a language finely tuned to express solutions to problems in your domain. In other languages, you may reuse useful routines but it's not as powerful. Additionally, as the dependencies are added the interactions between them can become a real headache. What's interesting to me is that for some problems Lisp feels like a cheat code and it goes exactly the way I just described. For other problems it feels like a weird old language with a lot of parens. Perhaps I'm just not good enough at Lisp though.
Jach 5y ago
How many languages do you know? You might find some inspiration in answering your own question by comparing what makes your efficiency different in one or the other. If you find your efficiencies are roughly the same, I might suggest you aren't using each language to its fullest potential and are writing in a lowest-common-denominator style, but that really might not be the case and perhaps the efficiency gains and tradeoffs have just been balanced well for you, so you'll need to decompose individual (in)efficiencies. For example, in one language you might not need to wait for a lengthy compilation cycle before you can run the code after changes, but with a beefy enough computer (or a networked supercomputer constantly doing compilation for all developers) perhaps the other language's lengthy compilation cycle is not in fact so lengthy for you. Or in another language, you might have strings with built-in pattern matching facilities that don't even need an import statement, if you need such things you'll be at an advantage over languages without them -- until you discover that for some other language someone else has already written a nice regex library with syntax that isn't too annoying even if it's not quite as elegant as something that's part of the language, and so the advantage narrows or disappears. (In regards to the GP's comment, you might very well get by fine with a Java team of 10 just like the Lispers, because 90+ other Java programmers have already written what you need in the broader Java ecosystem which the Lispers take for granted as part of the language, allowing you both to focus on the problem without distraction.) I'm a fan of JRebel for improving efficiencies in Java development, in one of their marketing materials (https://www.jrebel.com/sites/rebel/files/pdfs/rw-mythbusters-jrebel.pdf https://www.jrebel.com/sites/rebel/files/pdfs/rw-mythbusters...) they claim that waiting on redeploys takes up about 20% of a Java developer's salaried time. So with naive assumptions if you had a team of 100 developers, you could fire 20 of them and maintain the same pace just by forcing the remaining 80 to use JRebel. Or just keep all 100 but now everyone's got a 20% efficiency boost and can do 20% more than before! (Yadda yadda real world caveats.) For Common Lisp, this "no waiting for redeploys" is part of the language, you've been able to enjoy such a feature for decades without having to wait for JRebel to come into existence/figure out how to add it to a language yourself (some game studios accomplish approximations with their custom C++ engines), so it's a possible efficiency gain over languages that don't have a way to balance it -- like with JRebel which comes close, or to a lesser degree by more disciplined unit testing that lets you confidently and accurately make changes without needing to redeploy the whole thing for verification, or writing in a more purely functional style where you don't additionally suffer the burden of having to rebuild up tons of state after a redeploy, or to another degree by solving everything with an advanced type system, or just by having simple problems where you need integer-factor efficiency differences for them to matter for the duration you'll be dealing with the problem... CL has a bunch of other stuff built-in that can lead to efficiency gains too; macros of course are a big part of that but not the only other part. I'll list a few examples of macros being useful. There's a macro that's been around in the ecosystem since the 90s that lets you write things in infix notation, so you can write your #I((-b + sqrt(b^^2 - 4*a*c))/(2*a)) and not have to live with (/ (+ (- b) (sqrt (- (expt b 2) (* 4 a c)))) (* 2 a)). (In any case negative sqrt will give you a complex number automatically, which is nice.) The built in "loop" macro for looping has some very advanced features on top of the more common for-i-.. style or for-each.. style of other langs, e.g. from Practical Common Lisp: (defparameter *random* (loop repeat 100 collect (random 10000))) (loop for i in *random* counting (evenp i) into evens counting (oddp i) into odds summing i into total maximizing i into max minimizing i into min finally (return (list min max total evens odds))) (Symbols being first-class (that is, "maximizing" or "minimizing" aren't function names or strings) enables a lot of nice features apart from macros themselves.) It's not particularly hard to get such similar statistics from a list of numbers in another language, but having it be this easy and terse can give you a marginal efficiency boost if you happen to need to do something similar. The string formatting function "format" (not macro) has similar flexibility and any time I have to output a comma-delimited (except the last item) string of things in other-lang I get sad I don't have Lisp's format. Marginal efficiency boosts add up. The final macro example is that in Java we eventually got try-with-resources syntax (https://docs.oracle.com/javase/tutorial/essential/exceptions/tryResourceClose.html https://docs.oracle.com/javase/tutorial/essential/exceptions...), or in Python we eventually got the "with" statement, meanwhile in CL there's always been a style of "with-" macros that address the same issues. Some "with-" things are built-in, but it's trivial to write your own just like it's trivial in Java to implement AutoCloseable. Macros in Lisp can be about as easy to write as C preprocessor macros, though instead of string substitution into a template you're technically doing expression substitution, but Lisp macros can also manipulate the incoming data and run any other Lisp code before it produces some expansion, making them capable of far more when you need them to be. e.g. the boolean operator "or" is implemented as a macro that expands into a conditional evaluation (second+ operands won't evaluate if the first is true). Historically this has helped Lisp programmers "steal" any neat features from other languages without having to wait for an update to the Lisp implementation or standard. That has more to do with how Lisp has continued to be relevant though. Hopefully at least I've given you an idea of how it's at least possible for there to be any efficiency gains using Lisp, either in the past before other languages and their ecosystems caught up, or even now for certain classes of problems (e.g. if you believe 20% efficiency gain is possible by eliminating waiting on redeploys while developing). Though I think marginal efficiencies that sum up to 10x are probably pretty rare nowadays. Such a general claim was more tenable in the past when even garbage collection wasn't really mainstream...
mbrodersen 5y ago
I have heard the same kind of claims about Ruby, Python, JavaScript, non-typed languages, functional languages etc. as well. I still have seen zero evidence that any of those claims are true. I think those claims are often made when somebody feel super productive using language/technology X, and then jump to conclusions about X based on that feeling. Seriously, if Lisp was indeed 10x more productive than competing languages, I would have switched to it a long time ago. It would be silly not to!

mmmm2 5y ago

I can't speak from a professional level about this, but I'll tell you why I started using Common Lisp for my personal development. One day I was thinking it would be awesome if someone created a really powerful language with great facilities for abstraction: first class functions, objects, meta objects, etc. It should use dynamic typing during development with an excellent REPL and a built in debugger available everywhere, but it should have the ability to add type declarations when the code got more settled. Even better, this language should include ways of hinting to the compiler how to make a chunk of code faster, and if that wasn't enough, you should be able to replace chunks of code seamlessly with highly optimized C code. I thought about how to implement a language like this for a while, but when I looked into it I discovered Common Lisp does most of this already. It's static typing isn't perfect, though now there is https://coalton-lang.github.io/ https://coalton-lang.github.io/ if that's your thing. So that did it for me, but really this is only part of what you can do with Common Lisp. I think it still has a niche in places where the "right thing" isn't know yet. For example, I read a Reddit post saying that Common Lisp would be a poor choice now for a web startup, and that Paul Graham overstates Common Lisp's case as the reason for Viaweb's success. This is probably true in 2022, but I think it's instructive to look at why Paul claimed Lisp was so helpful back in 1996. There was no Rails or Django, no React or single page web apps. Viaweb built Scheme style continuations onto Common Lisp to simulate an SPA before AJAX was a thing, and on top of that created a slick HTML templating language based off lisp macros. To do that in Perl, or C, would have required something like writing a scheme interpreter, and would have taken much longer for a 2 person team. Is that a 100x advantage? Hard to say. Ultimately, I do think it still has advantages, you just have to find the areas where nothing else will really do the job.

evdubs 5y ago

It's really not that much more efficient when you compare Lisps to modern languages that share features like interactive (REPL) development, garbage collection, large standard libraries, concurrency primitives and abstractions, object/functional programming styles, integrated package managers, etc. When you compare modern languages, including Lisps, to something like C99, I think it's easier to see how you can be much more efficient/productive. If you take away the similarities of Lisps and modern languages, you're mostly left with the syntax and how the syntax enables macros that set Lisps apart from other languages. But, most of your Lisp code looks very much like your other code where you're writing functions, defining structures, calling object methods, etc. so it seems a stretch that the syntax and macros alone make you much more efficient/productive.

duped 5y ago
> It's really not that much more efficient when you compare Lisps to modern languages that share features like interactive (REPL) development, garbage collection, large standard libraries, concurrency primitives and abstractions, object/functional programming styles, integrated package managers, etc. pick 2! But seriously, what modern languages have all these features?
- evdubs 5y ago
  Java/JVM languages, C#/CLR languages, Python and Ruby (minus real threads, but compare to Racket with not-the-best real threading story), NodeJS (minus the "large" standard library, but compare to Scheme with a small standard library), can probably include Go
  - reikonomusha 5y ago
    I don't think any of these have REPL-driven development. They might have a "REPL" but not interactive code reloading as a part of the development cycle as is meant by a Lisp REPL.
    - School-Cotton 5y ago

jrumbut 5y ago

Lisp projects also tend to keep the dependency list short. In the best case scenario, once the foundation is laid and the project has momentum, you end up with what feels like a language finely tuned to express solutions to problems in your domain. In other languages, you may reuse useful routines but it's not as powerful. Additionally, as the dependencies are added the interactions between them can become a real headache. What's interesting to me is that for some problems Lisp feels like a cheat code and it goes exactly the way I just described. For other problems it feels like a weird old language with a lot of parens. Perhaps I'm just not good enough at Lisp though.

Jach 5y ago

How many languages do you know? You might find some inspiration in answering your own question by comparing what makes your efficiency different in one or the other. If you find your efficiencies are roughly the same, I might suggest you aren't using each language to its fullest potential and are writing in a lowest-common-denominator style, but that really might not be the case and perhaps the efficiency gains and tradeoffs have just been balanced well for you, so you'll need to decompose individual (in)efficiencies. For example, in one language you might not need to wait for a lengthy compilation cycle before you can run the code after changes, but with a beefy enough computer (or a networked supercomputer constantly doing compilation for all developers) perhaps the other language's lengthy compilation cycle is not in fact so lengthy for you. Or in another language, you might have strings with built-in pattern matching facilities that don't even need an import statement, if you need such things you'll be at an advantage over languages without them -- until you discover that for some other language someone else has already written a nice regex library with syntax that isn't too annoying even if it's not quite as elegant as something that's part of the language, and so the advantage narrows or disappears. (In regards to the GP's comment, you might very well get by fine with a Java team of 10 just like the Lispers, because 90+ other Java programmers have already written what you need in the broader Java ecosystem which the Lispers take for granted as part of the language, allowing you both to focus on the problem without distraction.) I'm a fan of JRebel for improving efficiencies in Java development, in one of their marketing materials (https://www.jrebel.com/sites/rebel/files/pdfs/rw-mythbusters-jrebel.pdf https://www.jrebel.com/sites/rebel/files/pdfs/rw-mythbusters...) they claim that waiting on redeploys takes up about 20% of a Java developer's salaried time. So with naive assumptions if you had a team of 100 developers, you could fire 20 of them and maintain the same pace just by forcing the remaining 80 to use JRebel. Or just keep all 100 but now everyone's got a 20% efficiency boost and can do 20% more than before! (Yadda yadda real world caveats.) For Common Lisp, this "no waiting for redeploys" is part of the language, you've been able to enjoy such a feature for decades without having to wait for JRebel to come into existence/figure out how to add it to a language yourself (some game studios accomplish approximations with their custom C++ engines), so it's a possible efficiency gain over languages that don't have a way to balance it -- like with JRebel which comes close, or to a lesser degree by more disciplined unit testing that lets you confidently and accurately make changes without needing to redeploy the whole thing for verification, or writing in a more purely functional style where you don't additionally suffer the burden of having to rebuild up tons of state after a redeploy, or to another degree by solving everything with an advanced type system, or just by having simple problems where you need integer-factor efficiency differences for them to matter for the duration you'll be dealing with the problem... CL has a bunch of other stuff built-in that can lead to efficiency gains too; macros of course are a big part of that but not the only other part. I'll list a few examples of macros being useful. There's a macro that's been around in the ecosystem since the 90s that lets you write things in infix notation, so you can write your #I((-b + sqrt(b^^2 - 4*a*c))/(2*a)) and not have to live with (/ (+ (- b) (sqrt (- (expt b 2) (* 4 a c)))) (* 2 a)). (In any case negative sqrt will give you a complex number automatically, which is nice.) The built in "loop" macro for looping has some very advanced features on top of the more common for-i-.. style or for-each.. style of other langs, e.g. from Practical Common Lisp: (defparameter *random* (loop repeat 100 collect (random 10000))) (loop for i in *random* counting (evenp i) into evens counting (oddp i) into odds summing i into total maximizing i into max minimizing i into min finally (return (list min max total evens odds))) (Symbols being first-class (that is, "maximizing" or "minimizing" aren't function names or strings) enables a lot of nice features apart from macros themselves.) It's not particularly hard to get such similar statistics from a list of numbers in another language, but having it be this easy and terse can give you a marginal efficiency boost if you happen to need to do something similar. The string formatting function "format" (not macro) has similar flexibility and any time I have to output a comma-delimited (except the last item) string of things in other-lang I get sad I don't have Lisp's format. Marginal efficiency boosts add up. The final macro example is that in Java we eventually got try-with-resources syntax (https://docs.oracle.com/javase/tutorial/essential/exceptions/tryResourceClose.html https://docs.oracle.com/javase/tutorial/essential/exceptions...), or in Python we eventually got the "with" statement, meanwhile in CL there's always been a style of "with-" macros that address the same issues. Some "with-" things are built-in, but it's trivial to write your own just like it's trivial in Java to implement AutoCloseable. Macros in Lisp can be about as easy to write as C preprocessor macros, though instead of string substitution into a template you're technically doing expression substitution, but Lisp macros can also manipulate the incoming data and run any other Lisp code before it produces some expansion, making them capable of far more when you need them to be. e.g. the boolean operator "or" is implemented as a macro that expands into a conditional evaluation (second+ operands won't evaluate if the first is true). Historically this has helped Lisp programmers "steal" any neat features from other languages without having to wait for an update to the Lisp implementation or standard. That has more to do with how Lisp has continued to be relevant though. Hopefully at least I've given you an idea of how it's at least possible for there to be any efficiency gains using Lisp, either in the past before other languages and their ecosystems caught up, or even now for certain classes of problems (e.g. if you believe 20% efficiency gain is possible by eliminating waiting on redeploys while developing). Though I think marginal efficiencies that sum up to 10x are probably pretty rare nowadays. Such a general claim was more tenable in the past when even garbage collection wasn't really mainstream...

mbrodersen 5y ago

I have heard the same kind of claims about Ruby, Python, JavaScript, non-typed languages, functional languages etc. as well. I still have seen zero evidence that any of those claims are true. I think those claims are often made when somebody feel super productive using language/technology X, and then jump to conclusions about X based on that feeling. Seriously, if Lisp was indeed 10x more productive than competing languages, I would have switched to it a long time ago. It would be silly not to!