23 ms·

Learning Common Lisp to beat Java and Rust on a phone encoding problem

337 points by medo-bear 5y ago

kjgkjhfkjf 5y ago
It's not quicker to write code in Lisp when you don't know Lisp, and you have to learn it to make a change to some Lisp code that someone randomly added to the codebase. This is true of all niche languages that are supposedly quicker to use than regular mainstream languages.
- Jtsummers 5y ago
  Your first part is tautological but worthless. It's never faster to write in a language you don't know versus one you do know. EDIT: To amend the preceding sentence, it's almost never faster. There are probably languages which people know that are sufficiently in conflict with the problem domain that they could be faster in some other new-to-them language versus the one they know for specific problems.
  - fsckboy 5y ago
    APL comes to mind as a language that could be faster to learn and use for a particular task, if your particular task involves a lot of pure matrix combination and manipulation. though, the fact that it wants its own keyboard is a bit of a hurdle :)
- deleted 5y ago
  [deleted]
wedesoft 5y ago
If you're coming from Java, you might want to look at Clojure. It has immutable datastructures and Java interop.
- afandian 5y ago
  I just used a Clojure library from a Kotlin project via Java interop. And it wasn't even weird.
- medo-bear 5y ago
  From the author: > Even though Clojure might be a more obvious choice for someone, like me, who is used to working with the JVM, I actually wanted to try using something else with a lighter runtime and great performance.
- facelessuser 5y ago
  Unless you go out of your way in writing non-idiomatic Clojure like in Java with parenthesis Clojure, Clojure will the slowest of the three by far.
- dunefox 5y ago
  Or Armed Bear Common Lisp.
kubb 5y ago
I found Common Lisp to be surprisingly ahead of its time in many regards (debugging, repl, compilation and execution speed, metaprogramming), but unfortunately it doesn't have a large community, and it's showing its age (no standard package management, threading not built into the language). It's also dynamically typed which disqualifies it for large collaborative projects.
- gibsonf1 5y ago
  It has great package management with https://www.quicklisp.org/beta/ https://www.quicklisp.org/beta/ and some truly great and high quality libraries, especially Fukamachi's suite of web libraries and so many others. Woo, for example, is the fastest web server. https://github.com/fukamachi/woo https://github.com/fukamachi/woo (Faster then the runner up Go by quite a bit) For parallel computing, we use: https://lparallel.org/ https://lparallel.org/ Its been great at handling massive loads accross all processors elegantly. And then for locking against overwrites on highly parallel database transactions we use mutex locks that are built into the http://sbcl.org/ http://sbcl.org/ compiler with very handy macros.
  - Mikeb85 5y ago
    Slightly off-topic but I'm in awe of Fukamachi's repos. That one person has built so much incredible stuff is amazing to me. Not sure it's enough to get me using CL all the time, but it's very impressive.
    - gibsonf1 5y ago
      The math library we use is incredibly fast with quaternion matrix transformations: https://github.com/cbaggers/rtg-math/ https://github.com/cbaggers/rtg-math/ The only gaps we've had with our production code and lisp is PDF (we use the java pdfbox), translating between RDF formats (also a java lib) and encrypting JWP tokens for PKCE dPop authentication (also java) The complete conceputal AI system and space/time causal systems digital twin technology is all in common lisp (sbcl) Also fantastic is the sb-profile library in sbcl that lets you profile any number of functions and see number of iterations and time used as well as consing all ordered by slowest cummulative time. That feature has been key on finding those functions that are slow and optimizing leading to orders of magnitude speed improvements.
eggy 5y ago
Before Python became ML's darling, there was Lush: Lisp Universal SHell by none other than Yann LeCun and Leon Bottouk[1], and I thought my Lisp knowledge was going to finally pay off ;) I've since moved on to J and APL. SBCL is amazing and aside from calling C directly, it is still very fast. Python is slow without its specialized libraries from the C-world, but SBCL produces short, readable code that is pretty efficient. There ECL[2] for an embedded Lisp, or uLisp[32] for really small processors like the ESP chips. [1] http://lush.sourceforge.net/ http://lush.sourceforge.net/ [2] https://common-lisp.net/project/ecl/ https://common-lisp.net/project/ecl/ [3] http://www.ulisp.com/ http://www.ulisp.com/
- pvitz 5y ago
  I am using J on and off, but I am not aware of a ML package for it. Are you writing all algorithms yourself or could you recommend a package?
  - ruste 5y ago
    I'd also like to know. I'm considering writing some for fun in GNU APL right now.
  - liveranga 5y ago
    There's a handful of resources around that look fun though I haven't dug into them yet. There's this on the J wiki: https://code.jsoftware.com/wiki/User:Brian_Schott/code/feedforwardNN/code/feedforwardNN https://code.jsoftware.com/wiki/User:Brian_Schott/code/feedf... And there's this YouTube series for APL: https://youtube.com/playlist?list=PLgTqamKi1MS3p-O0QAgjv5vt4NY5OgpiM https://youtube.com/playlist?list=PLgTqamKi1MS3p-O0QAgjv5vt4...
  - eggy 5y ago
    I have a bunch of links to ML material for either APL or J. I don't know of any particular library for J. J is interpreted, so it is not as fast as other implementations. I am mainly using it to experiment on concepts and teach myself more ML in J because of the iterative nature of the REPL, and the succinct code. I can keep what's going on in my head, and glance at less than 100 lines, usually 15 lines, of code to refresh it. There is a series of videos of learning neural networks in APL cited by others here on this thread. Pandas author, Wes McKinney, cited J as an influence in his work on Pandas. Extreme Learning Machine in J (code and PDF are here too): https://github.com/peportier/jelm https://github.com/peportier/jelm Convolutional neural networks in APL (PDF and video on page): https://dl.acm.org/doi/10.1145/3315454.3329960 https://dl.acm.org/doi/10.1145/3315454.3329960 A DSL to implement MENACE (Matchbox Educable Noughts And Crosses Engine) in APL (Noughts and Crosses or Tic-tac-toe): https://romilly.github.io/o-x-o/an-introduction.html https://romilly.github.io/o-x-o/an-introduction.html
get52 5y ago
Common Lisp? Yuu mean the language that got horsefucked like 30 years ago and barely anyone uses? I'll pass lol
tyingq 5y ago
Not the same phone encoding challenge, but an interesting feature of the bsd globbing built into bash. The map is the typical map of letters on a touch-tone phone..where "2" can be a, b, or c. All letter combos for the number "6397": #!/usr/bin/bash m[0]=0;m[1]=1;m[2]="{a,b,c}";m[3]="{d,e,f}";m[4]="{g,h,i}";m[5]="{j,k,l}"; m[6]="{m,n,o}";m[7]="{p,q,r,s}";m[8]="{t,u,v}";m[9]="{w,x,y,z}" var=$(echo ${m[6]}${m[3]}${m[9]}${m[7]}) eval echo $var
xondono 5y ago
Some weeks ago I tried to start learning Lisp too. I found it odd, but I’ve seen weirder. Then I tried the simple example of writing a function to compute the nth fibonacci number (it’s almost the first example). After testing some numbers, I thought it seemed slow, so I quickly implemented the same function in Rust (with recursion too). Rust took less time in compiling, and computing fib(n) for n 1..50 than Lisp to compute fib(50). Maybe I did something very wrong, but for now I’d rather learn Rust better.
- Jtsummers 5y ago
  The only way anyone but you can know if you did anything wrong is if you include your code, and your common lisp implementation. They don’t all have the same performance.
  - xondono 5y ago
    It’s not like I did something weird, I pretty much copy pasted from https://lisp-lang.org/learn/functions https://lisp-lang.org/learn/functions I know the reason for the time taken is because it’s not tail recursive, and that making it tail recursive would make it almost immediate, but for me that was not the point. The thing is, I don’t really know how to write performant rust code, and yet my naive implementation in rust works better than the example in learn-lisp. To me that means performant Lisp is non trivial (meaning you need deep understanding to achieve it). If you show a slow but easier to understand example, it’s because fast examples are way harder to understand.
    - aidenn0 5y ago
      > I know the reason for the time taken is because it’s not tail recursive, and that making it tail recursive would make it almost immediate, but for me that was not the point. Assuming your rust version is not tail-recursive, this is not true. Of course a tail-recursive (or iterative for that matter) version would be way faster, but that is independent of the language used[1]. The naive algorithm in both Rust and Lisp should be relatively similar in runtimes, and for me (after I properly enabled optimiations) they were. Also, tail recursion is a red-herring here for another reason. For both Rust and Lisp, I would use an iterative approach, not a tail-recursive approach, like the example in the Rust num_bigint docs[2]. 1: A theoretical language that automatically memoized pure functions would make this false, but that doesn't apply to the current discussion. 2: https://docs.rs/num-bigint/0.4.2/num_bigint/ https://docs.rs/num-bigint/0.4.2/num_bigint/ The equivalent Lisp would be: CL-USER> (defun fib (n) (let ((f0 0) (f1 1)) (loop repeat n do (shiftf f0 f1 (+ f0 f1))) f0)) FIB CL-USER> (time (fib 1000)) Evaluation took: 0.000 seconds of real time 0.000171 seconds of total run time (0.000136 user, 0.000035 system) 100.00% CPU 471,103 processor cycles 65,456 bytes consed 43466557686937456435688527675040625802564660517371780402481729089536555417949051890403879840079255169295922593080322634775209689623239873322471161642996440906533187938298969649928516003704476137795166849228875 CL-USER> For small values the difference is lost in the noise. For larger values (e.g. 1000000) Rust is about 2x faster than SBCL, which is about what I would expect for a test like this.
Decabytes 5y ago
Is it weird that I don’t like Common Lisp at all but I like Scheme a lot? I just never liked Lisp 2s and separate name spaces for functions and variables. But really that is the biggest issue for me. I’m sure if only Common Lisp existed it wouldn’t bother me at all. That being said, I think CL is a fantastic language and there is a lot more libraries out there to do useful things than in scheme. My C programming is really weak so I find it challenging whenever I come across a library in c that isn’t already wrapped
- bitwize 5y ago
  Not really. I'm a diehard Schemer as well, for the same reasons: the Lisp-1 nature helps you treat procedures as true first-class values and fits in with Scheme's more functional style. Something you can do in CL also, just with a bit more ceremony. And CL programmers are more "haha, setf go brrrr" anyway. That said, I'd rather use CL by far than any other language except Scheme, and there are cases where CL is the right thing and Scheme is not. The most brilliant, beautiful, joy to maintain enterprise system I've ever seen was written in CL.
- sleibrock 5y ago
  I'm a bit the same. I've been writing Racket for a number of years now and looking back at Lisp I see a lot of ugliness that I don't really think I enjoy. Racket has a nice package manager and module system that kind of works for me, and the documentation is honestly some of the best I've ever used, if not my favorite. Comparatively, I've tried using GNU Guile and found the online documentation to be horrendous, and trying to find online documentation for what's considered to be the "standard library" in Common Lisp still confuses me. I love seeing people use CL and other Lisp-likes in the wild, and Norvig was a big inspiration for me.
  - bollu 5y ago
    What IDE do you use for racket? In emacs, I've found SLIME and its associated debugger to be more powerful than GEISER. I never could come to like Dr. Racket due to its lack of autocomplete and things like parinfer / paredit.
kragen 5y ago
This is a pretty introductory CL article, mostly a commentary on Norvig's solution to the problem. Still, I learned about the #. readmacro from it. The conclusion: "[The Lisp implementation] was the fastest implementation for all input sizes except the largest one, where it performed just slightly worse than my best Java implementation." GH repo at https://github.com/renatoathaydes/prechelt-phone-number-encoding https://github.com/renatoathaydes/prechelt-phone-number-enco.... Sounds like he was mostly measuring the performance of the SBCL bignum implementation.
- xxs 5y ago
  The tests are just few seconds, it's measuring bootstap, compilation time for the most of the part.
  - kragen 5y ago
    It's true that they're just a few seconds, but if he were measuring compilation time then none of the Java tests would come in under 10 000 ms, and (although the graph isn't usefully labeled) it looks like they're around 100 ms.
  - brabel 5y ago
    In one of the articles, it shows SBCL performance does not increase by first compiling it to a binary executable. Compilation overhead in SBCL is therefore negligible.
lkey 5y ago
As with all of these forays, I applaud the author for learning new things, but these benchmarks are primarily testing reading from stdin and format printing to stdout, followed by testing bignum impls as mentioned elsewhere. For this reason, the title and benchmarks are a bit misleading.
- xxs 5y ago
  ...also testing half-compiled code. JIT compiled tests that run few seconds just discredit their authors. It's yet another example of how not to perform microbenchmarks.
  - twicetwice 5y ago
    Worth pointing out for those who don't know that the Java runtime does JIT optimizations of bytecode based on observed runtime patterns. Blew my mind when engineers were talking about "warming up the code paths" during startup of a large Java service and I found out it wasn't joke/superstition.
    - mhh__ 5y ago
      I just tentatively found a 10% speedup in the D compiler frontend by enabling profile guided optimization.
      - xxs 5y ago
        Java effectively always have guided compilation with perf. counters and all.
    - xxs 5y ago
      Warming is a tiny part of microbenchmarking actually. Knowing how and what the compiler optimizes, like purpose lattice checks to ensure no extra array boundaries in the code. Call site optimizations (class hierarchy analysis) - single site (direct calls, inclines), dual site - guarded checks, vs 3-5 - inline caches vs. full virtual calls. Card-marking of stores for concurrent GC. L2 and L3 CPU cache sizes awareness (certain dataset may fit and the results are misleading)... There are a lot other techniques to consider, including looking directly at the assembly code. Microbenchmarking requires rather deep knowledge how the JIT works, how the hardware - CPU/cache/memory operates, the costs of calls certain system calls and what not.
eigenhombre 5y ago
I write Clojure for food, and Common Lisp for fun. One reason for the latter is CL's speed -- awhile back I compared a bit of (non-optimized) Clojure code I wrote for a blog post with a rough equivalent in CL, and was stunned that the Common Lisp code ran about 10x faster. This made me curious as to how fast it could be made if I really tried, and was able to get nearly 30x more[1] by optimizing it. Clojure is definitely fast enough for everything I've done professionally for six years. But Common Lisp, while having plenty of rough edges, intrigues on the basis of performance alone. (This is on SBCL -- I have yet to play with a commercial implementation.) [1] http://johnj.com/from-elegance-to-speed.html http://johnj.com/from-elegance-to-speed.html
- lmilcin 5y ago
  I have implemented real algotrading framework in Common Lisp (SBCL) that connected directly to Warsaw Stock Exchange. The application was receiving binary messages from the exchange over multicast, rebuilding state of the market, running various (simple) algorithms and responding with orders within 5 microseconds of the original message, at up to 10k messages per second. With SBCL you can write a DSL and have ability to fully control the resulting instructions (through vops). It is just the question how dedicated you are to writing macros upon macros upon macros. I used this to the fullest extent and the result was as good as any hand optimized C/assembly. For example, the binary message parser would receive a stack of complicated specifications for the messages in the form of XML files (see here if you are curious: https://www.gpw.pl/pub/GPW/files/WSE_CDE_XDP_Message_Specifications_-_v5.9b.pdf https://www.gpw.pl/pub/GPW/files/WSE_CDE_XDP_Message_Specifi...), converted XML to DSL and then, through magic of macros, the DSL was converted to accessors that allowed optimal access to the fields. Optimal here mans the assembly could not be improved upon any further. Large parts of the application (especially any communication and device control as it was done with full kernel bypass) was written in ANSI C but the Foreign Function Interface makes integrating it into the rest of application a breeze. I write all of this, because I frequently meet complete disbelief from people that Lisp can be used in production. I personally think it is exactly the opposite. Lisps offer fantastic development environment. The problem are developers who are mostly unable to use Lisp for work effectively, I guess due to too much power, freedom and complete lack of direction on how to structure your application.
tehjoker 5y ago
I tried experimenting with Common LISP but never understood how one was supposed to deploy a finished binary to the end user. It seemed like you were supposed to load up an environment? It made me leery of continuing.
- aidenn0 5y ago
  For most common-lisp implementations you can just save the current state as an executable, along with specifying an entry-point. Obviously doing this from your dev environment is a bad idea, so most people write a script that loads the system and dumps the image. Or you can use one that someone else has already written. Not all lisps work this way (most notably ECL does not), and you can use ASDF's program-op to create an executable directly from a package definition, which should work on any implemntation supported by ASDF.
  - tehjoker 5y ago
    Thanks, the saving the dev environment part was the part that scared me. It certainly helps to have someone tell you you're not crazy. Maybe I'll give it another try sometime!
    - aidenn0 5y ago
      One of the things I've learned with powerful tools is that, by their nature, they empower you to do bad things almost as much as they empower you to do good things. It's even alluded to in the ruby bare-words example in this famous talk[1] 1: https://www.youtube.com/watch?v=3se2-thqf-A https://www.youtube.com/watch?v=3se2-thqf-A
    - foobarbaz33 5y ago
      You can just run make when you are ready to make a clean build and executable. Same as any other language.
    - _ph_ 5y ago
      Nothing to worry about. Yes, the minimal executable you get with SBCL is a bit fat, but for any real-world application which do some actual work, the size is a non-issue. Especially, as the executables are starting up very fast, as no dynamic linking is necessary, most of the code are static references. For any sizable Lisp project you would have an ASDF-file to load the system, with that you can create an easy build-script or even a make-file, which loads your program via ASDF and then builds the executable, it is described well here: https://stackoverflow.com/questions/14171849/compiling-common-lisp-to-an-executable https://stackoverflow.com/questions/14171849/compiling-commo... For consistancy, I would recommend to build the executable from the freshly loaded source, not a lisp image which has been used to develop in.
orthecreedence 5y ago
I miss my common lisp days, and I think rust being able to export C ABIs makes it a really great companion language for CL. I also think common lisp (or really, any fast lisp) is a really great tool for game development. The fact that you can redefine a program as it's running really helps iterate without having to set up your state over and over. Pair that with a fast, low-level language that can handle a lot of the lower level stuff (been trying to find time to look at Bevy on rust), and you have a killer combo. The main reason I stopped using lisp was because of the community. There were some amazing people that helped out with a lot of the stuff I was building, but not a critical mass of them and things just kind of stagnated. Then it seemed like for every project I poured my soul into, someone would write a one-off copy of it "just because" instead of contributing. It's definitely a culture of lone-wolf programmers. CL is a decent language with some really good implementations, and everywhere I go I miss the macros. I definitely want to pick it up again sometime, but probably will just let the async stuff I used to work on rest in peace.
- a-dub 5y ago
  i assume you're familiar with GOAL? https://en.wikipedia.org/wiki/Game_Oriented_Assembly_Lisp https://en.wikipedia.org/wiki/Game_Oriented_Assembly_Lisp
  - orthecreedence 5y ago
    I was not! Thanks for the tip.
- sahil-kang 5y ago
  I can understand the failure to build a critical mass of contributors, but can you share some examples of where your work was duplicated instead of built upon? I’ve read through some of your async work in the past and from an initial glance, it seemed like you had the right idea by wrapping existing event libs and exposing familiar event loop idioms. At the very least, it seemed uncontroversial so I’m interested to see why others would choose not to build upon it.
  - 5y ago
codr7 5y ago
I've been playing around with designing programming languages in Common Lisp lately. I'm curious how far it's possible to push performance by generating Lisp code in SBCL compared to the classical C interpreter goto loop. https://github.com/codr7/snabl https://github.com/codr7/snabl
temporallobe 5y ago
I regularly work with Clojure. This is probably an unpopular opinion in this thread, but despite having worked with it for years, I still don’t much like it, mostly because it’s far too terse and the syntax is so far removed from that of C-based languages. The other day I wrote a Java-based DTO and it was refreshing how clear and verbose everything was, despite the almost comical volume of code it took in comparison to what similar functionality would look look like in Clojure. Plus, the state of tooling in Clojure is not the best. I would also add that while you might initially do well with something like Clojure, it may be difficult to maintain, especially if you plan to make it a legacy product with O&M support in the future.
- Guthur 5y ago
  Well this article is about Common Lisp :). You could easily use CLOS or Structs in Common Lisp to do DTO. If you were using CLOS for DTOs you'd also have generic functions to dispatch on said DTOs.
- jhhh 5y ago
  The hard to maintain issue seems to follow most dynamic languages around. What kind of those issues have you run in to in your time working with Clojure professionally?
  - Guthur 5y ago
    To be honest; "hard to maintain", "technical debt", "total rewrite required", "monolith to micro service", probably "micro service to monolith" at some point, all of these appear no matter what the language, I really don't think static analysis is the problem. Though I will give you that pretty much every popular dynamic language has terrible runtime environments, which doesn't help at all. If I don't have comprehensive static analysis I want good introspection and reflection, I'd ideally want static manipulation and continuation.
- deleted 5y ago
  [deleted]
- deleted 5y ago
  [deleted]
peatmoss 5y ago
I’ve been increasingly coming around to the opinion that computers have gotten kind of boring. I miss how weird and different computers used to be, both in terms of hardware and software. For lisps, I think Racket and Clojure feel the most modern. But I’m starting to come around to the idea that there are enough modern, hyper modern, post-modern, languages and programmers out there. Some of the value in learning Common Lisp might be in its value for living software archaeology. You can find actual code written at the time people were figuring out Big Ideas that we take for granted today. And usually the process of synthesis means that Big Ideas lose a lot of associated commentary and thinking that fed in. There’s a place for modern languages that conform to the same general set of sensibilities. But there’s also joy in gettin’ weird with some of the old stuff. It’s smugly satisfying to beat the new kid on the block with old tricks on occasion.

337 points by medo-bear 5y ago

It's not quicker to write code in Lisp when you don't know Lisp, and you have to learn it to make a change to some Lisp code that someone randomly added to the codebase. This is true of all niche languages that are supposedly quicker to use than regular mainstream languages.

Jtsummers 5y ago
Your first part is tautological but worthless. It's never faster to write in a language you don't know versus one you do know. EDIT: To amend the preceding sentence, it's almost never faster. There are probably languages which people know that are sufficiently in conflict with the problem domain that they could be faster in some other new-to-them language versus the one they know for specific problems.
- fsckboy 5y ago
  APL comes to mind as a language that could be faster to learn and use for a particular task, if your particular task involves a lot of pure matrix combination and manipulation. though, the fact that it wants its own keyboard is a bit of a hurdle :)
deleted 5y ago
[deleted]

wedesoft 5y ago

If you're coming from Java, you might want to look at Clojure. It has immutable datastructures and Java interop.

afandian 5y ago
I just used a Clojure library from a Kotlin project via Java interop. And it wasn't even weird.
medo-bear 5y ago
From the author: > Even though Clojure might be a more obvious choice for someone, like me, who is used to working with the JVM, I actually wanted to try using something else with a lighter runtime and great performance.
facelessuser 5y ago
Unless you go out of your way in writing non-idiomatic Clojure like in Java with parenthesis Clojure, Clojure will the slowest of the three by far.
dunefox 5y ago
Or Armed Bear Common Lisp.

kubb 5y ago

I found Common Lisp to be surprisingly ahead of its time in many regards (debugging, repl, compilation and execution speed, metaprogramming), but unfortunately it doesn't have a large community, and it's showing its age (no standard package management, threading not built into the language). It's also dynamically typed which disqualifies it for large collaborative projects.

gibsonf1 5y ago
It has great package management with https://www.quicklisp.org/beta/ https://www.quicklisp.org/beta/ and some truly great and high quality libraries, especially Fukamachi's suite of web libraries and so many others. Woo, for example, is the fastest web server. https://github.com/fukamachi/woo https://github.com/fukamachi/woo (Faster then the runner up Go by quite a bit) For parallel computing, we use: https://lparallel.org/ https://lparallel.org/ Its been great at handling massive loads accross all processors elegantly. And then for locking against overwrites on highly parallel database transactions we use mutex locks that are built into the http://sbcl.org/ http://sbcl.org/ compiler with very handy macros.
- Mikeb85 5y ago
  Slightly off-topic but I'm in awe of Fukamachi's repos. That one person has built so much incredible stuff is amazing to me. Not sure it's enough to get me using CL all the time, but it's very impressive.
  - gibsonf1 5y ago
    The math library we use is incredibly fast with quaternion matrix transformations: https://github.com/cbaggers/rtg-math/ https://github.com/cbaggers/rtg-math/ The only gaps we've had with our production code and lisp is PDF (we use the java pdfbox), translating between RDF formats (also a java lib) and encrypting JWP tokens for PKCE dPop authentication (also java) The complete conceputal AI system and space/time causal systems digital twin technology is all in common lisp (sbcl) Also fantastic is the sb-profile library in sbcl that lets you profile any number of functions and see number of iterations and time used as well as consing all ordered by slowest cummulative time. That feature has been key on finding those functions that are slow and optimizing leading to orders of magnitude speed improvements.

eggy 5y ago

Before Python became ML's darling, there was Lush: Lisp Universal SHell by none other than Yann LeCun and Leon Bottouk[1], and I thought my Lisp knowledge was going to finally pay off ;) I've since moved on to J and APL. SBCL is amazing and aside from calling C directly, it is still very fast. Python is slow without its specialized libraries from the C-world, but SBCL produces short, readable code that is pretty efficient. There ECL[2] for an embedded Lisp, or uLisp[32] for really small processors like the ESP chips. [1] http://lush.sourceforge.net/ http://lush.sourceforge.net/ [2] https://common-lisp.net/project/ecl/ https://common-lisp.net/project/ecl/ [3] http://www.ulisp.com/ http://www.ulisp.com/

pvitz 5y ago
I am using J on and off, but I am not aware of a ML package for it. Are you writing all algorithms yourself or could you recommend a package?
- ruste 5y ago
  I'd also like to know. I'm considering writing some for fun in GNU APL right now.
- liveranga 5y ago
  There's a handful of resources around that look fun though I haven't dug into them yet. There's this on the J wiki: https://code.jsoftware.com/wiki/User:Brian_Schott/code/feedforwardNN/code/feedforwardNN https://code.jsoftware.com/wiki/User:Brian_Schott/code/feedf... And there's this YouTube series for APL: https://youtube.com/playlist?list=PLgTqamKi1MS3p-O0QAgjv5vt4NY5OgpiM https://youtube.com/playlist?list=PLgTqamKi1MS3p-O0QAgjv5vt4...
- eggy 5y ago
  I have a bunch of links to ML material for either APL or J. I don't know of any particular library for J. J is interpreted, so it is not as fast as other implementations. I am mainly using it to experiment on concepts and teach myself more ML in J because of the iterative nature of the REPL, and the succinct code. I can keep what's going on in my head, and glance at less than 100 lines, usually 15 lines, of code to refresh it. There is a series of videos of learning neural networks in APL cited by others here on this thread. Pandas author, Wes McKinney, cited J as an influence in his work on Pandas. Extreme Learning Machine in J (code and PDF are here too): https://github.com/peportier/jelm https://github.com/peportier/jelm Convolutional neural networks in APL (PDF and video on page): https://dl.acm.org/doi/10.1145/3315454.3329960 https://dl.acm.org/doi/10.1145/3315454.3329960 A DSL to implement MENACE (Matchbox Educable Noughts And Crosses Engine) in APL (Noughts and Crosses or Tic-tac-toe): https://romilly.github.io/o-x-o/an-introduction.html https://romilly.github.io/o-x-o/an-introduction.html

get52 5y ago

Common Lisp? Yuu mean the language that got horsefucked like 30 years ago and barely anyone uses? I'll pass lol

tyingq 5y ago

Not the same phone encoding challenge, but an interesting feature of the bsd globbing built into bash. The map is the typical map of letters on a touch-tone phone..where "2" can be a, b, or c. All letter combos for the number "6397": #!/usr/bin/bash m[0]=0;m[1]=1;m[2]="{a,b,c}";m[3]="{d,e,f}";m[4]="{g,h,i}";m[5]="{j,k,l}"; m[6]="{m,n,o}";m[7]="{p,q,r,s}";m[8]="{t,u,v}";m[9]="{w,x,y,z}" var=$(echo ${m[6]}${m[3]}${m[9]}${m[7]}) eval echo $var

xondono 5y ago

Some weeks ago I tried to start learning Lisp too. I found it odd, but I’ve seen weirder. Then I tried the simple example of writing a function to compute the nth fibonacci number (it’s almost the first example). After testing some numbers, I thought it seemed slow, so I quickly implemented the same function in Rust (with recursion too). Rust took less time in compiling, and computing fib(n) for n 1..50 than Lisp to compute fib(50). Maybe I did something very wrong, but for now I’d rather learn Rust better.

Jtsummers 5y ago
The only way anyone but you can know if you did anything wrong is if you include your code, and your common lisp implementation. They don’t all have the same performance.
- xondono 5y ago
  It’s not like I did something weird, I pretty much copy pasted from https://lisp-lang.org/learn/functions https://lisp-lang.org/learn/functions I know the reason for the time taken is because it’s not tail recursive, and that making it tail recursive would make it almost immediate, but for me that was not the point. The thing is, I don’t really know how to write performant rust code, and yet my naive implementation in rust works better than the example in learn-lisp. To me that means performant Lisp is non trivial (meaning you need deep understanding to achieve it). If you show a slow but easier to understand example, it’s because fast examples are way harder to understand.
  - aidenn0 5y ago
    > I know the reason for the time taken is because it’s not tail recursive, and that making it tail recursive would make it almost immediate, but for me that was not the point. Assuming your rust version is not tail-recursive, this is not true. Of course a tail-recursive (or iterative for that matter) version would be way faster, but that is independent of the language used[1]. The naive algorithm in both Rust and Lisp should be relatively similar in runtimes, and for me (after I properly enabled optimiations) they were. Also, tail recursion is a red-herring here for another reason. For both Rust and Lisp, I would use an iterative approach, not a tail-recursive approach, like the example in the Rust num_bigint docs[2]. 1: A theoretical language that automatically memoized pure functions would make this false, but that doesn't apply to the current discussion. 2: https://docs.rs/num-bigint/0.4.2/num_bigint/ https://docs.rs/num-bigint/0.4.2/num_bigint/ The equivalent Lisp would be: CL-USER> (defun fib (n) (let ((f0 0) (f1 1)) (loop repeat n do (shiftf f0 f1 (+ f0 f1))) f0)) FIB CL-USER> (time (fib 1000)) Evaluation took: 0.000 seconds of real time 0.000171 seconds of total run time (0.000136 user, 0.000035 system) 100.00% CPU 471,103 processor cycles 65,456 bytes consed 43466557686937456435688527675040625802564660517371780402481729089536555417949051890403879840079255169295922593080322634775209689623239873322471161642996440906533187938298969649928516003704476137795166849228875 CL-USER> For small values the difference is lost in the noise. For larger values (e.g. 1000000) Rust is about 2x faster than SBCL, which is about what I would expect for a test like this.

Decabytes 5y ago

Is it weird that I don’t like Common Lisp at all but I like Scheme a lot? I just never liked Lisp 2s and separate name spaces for functions and variables. But really that is the biggest issue for me. I’m sure if only Common Lisp existed it wouldn’t bother me at all. That being said, I think CL is a fantastic language and there is a lot more libraries out there to do useful things than in scheme. My C programming is really weak so I find it challenging whenever I come across a library in c that isn’t already wrapped

bitwize 5y ago
Not really. I'm a diehard Schemer as well, for the same reasons: the Lisp-1 nature helps you treat procedures as true first-class values and fits in with Scheme's more functional style. Something you can do in CL also, just with a bit more ceremony. And CL programmers are more "haha, setf go brrrr" anyway. That said, I'd rather use CL by far than any other language except Scheme, and there are cases where CL is the right thing and Scheme is not. The most brilliant, beautiful, joy to maintain enterprise system I've ever seen was written in CL.
sleibrock 5y ago
I'm a bit the same. I've been writing Racket for a number of years now and looking back at Lisp I see a lot of ugliness that I don't really think I enjoy. Racket has a nice package manager and module system that kind of works for me, and the documentation is honestly some of the best I've ever used, if not my favorite. Comparatively, I've tried using GNU Guile and found the online documentation to be horrendous, and trying to find online documentation for what's considered to be the "standard library" in Common Lisp still confuses me. I love seeing people use CL and other Lisp-likes in the wild, and Norvig was a big inspiration for me.
- bollu 5y ago
  What IDE do you use for racket? In emacs, I've found SLIME and its associated debugger to be more powerful than GEISER. I never could come to like Dr. Racket due to its lack of autocomplete and things like parinfer / paredit.

kragen 5y ago

This is a pretty introductory CL article, mostly a commentary on Norvig's solution to the problem. Still, I learned about the #. readmacro from it. The conclusion: "[The Lisp implementation] was the fastest implementation for all input sizes except the largest one, where it performed just slightly worse than my best Java implementation." GH repo at https://github.com/renatoathaydes/prechelt-phone-number-encoding https://github.com/renatoathaydes/prechelt-phone-number-enco.... Sounds like he was mostly measuring the performance of the SBCL bignum implementation.

xxs 5y ago
The tests are just few seconds, it's measuring bootstap, compilation time for the most of the part.
- kragen 5y ago
  It's true that they're just a few seconds, but if he were measuring compilation time then none of the Java tests would come in under 10 000 ms, and (although the graph isn't usefully labeled) it looks like they're around 100 ms.
- brabel 5y ago
  In one of the articles, it shows SBCL performance does not increase by first compiling it to a binary executable. Compilation overhead in SBCL is therefore negligible.

lkey 5y ago

As with all of these forays, I applaud the author for learning new things, but these benchmarks are primarily testing reading from stdin and format printing to stdout, followed by testing bignum impls as mentioned elsewhere. For this reason, the title and benchmarks are a bit misleading.

xxs 5y ago
...also testing half-compiled code. JIT compiled tests that run few seconds just discredit their authors. It's yet another example of how not to perform microbenchmarks.
- twicetwice 5y ago
  Worth pointing out for those who don't know that the Java runtime does JIT optimizations of bytecode based on observed runtime patterns. Blew my mind when engineers were talking about "warming up the code paths" during startup of a large Java service and I found out it wasn't joke/superstition.
  - mhh__ 5y ago
    I just tentatively found a 10% speedup in the D compiler frontend by enabling profile guided optimization.
    - xxs 5y ago
      Java effectively always have guided compilation with perf. counters and all.
  - xxs 5y ago
    Warming is a tiny part of microbenchmarking actually. Knowing how and what the compiler optimizes, like purpose lattice checks to ensure no extra array boundaries in the code. Call site optimizations (class hierarchy analysis) - single site (direct calls, inclines), dual site - guarded checks, vs 3-5 - inline caches vs. full virtual calls. Card-marking of stores for concurrent GC. L2 and L3 CPU cache sizes awareness (certain dataset may fit and the results are misleading)... There are a lot other techniques to consider, including looking directly at the assembly code. Microbenchmarking requires rather deep knowledge how the JIT works, how the hardware - CPU/cache/memory operates, the costs of calls certain system calls and what not.

eigenhombre 5y ago

I write Clojure for food, and Common Lisp for fun. One reason for the latter is CL's speed -- awhile back I compared a bit of (non-optimized) Clojure code I wrote for a blog post with a rough equivalent in CL, and was stunned that the Common Lisp code ran about 10x faster. This made me curious as to how fast it could be made if I really tried, and was able to get nearly 30x more[1] by optimizing it. Clojure is definitely fast enough for everything I've done professionally for six years. But Common Lisp, while having plenty of rough edges, intrigues on the basis of performance alone. (This is on SBCL -- I have yet to play with a commercial implementation.) [1] http://johnj.com/from-elegance-to-speed.html http://johnj.com/from-elegance-to-speed.html

lmilcin 5y ago
I have implemented real algotrading framework in Common Lisp (SBCL) that connected directly to Warsaw Stock Exchange. The application was receiving binary messages from the exchange over multicast, rebuilding state of the market, running various (simple) algorithms and responding with orders within 5 microseconds of the original message, at up to 10k messages per second. With SBCL you can write a DSL and have ability to fully control the resulting instructions (through vops). It is just the question how dedicated you are to writing macros upon macros upon macros. I used this to the fullest extent and the result was as good as any hand optimized C/assembly. For example, the binary message parser would receive a stack of complicated specifications for the messages in the form of XML files (see here if you are curious: https://www.gpw.pl/pub/GPW/files/WSE_CDE_XDP_Message_Specifications_-_v5.9b.pdf https://www.gpw.pl/pub/GPW/files/WSE_CDE_XDP_Message_Specifi...), converted XML to DSL and then, through magic of macros, the DSL was converted to accessors that allowed optimal access to the fields. Optimal here mans the assembly could not be improved upon any further. Large parts of the application (especially any communication and device control as it was done with full kernel bypass) was written in ANSI C but the Foreign Function Interface makes integrating it into the rest of application a breeze. I write all of this, because I frequently meet complete disbelief from people that Lisp can be used in production. I personally think it is exactly the opposite. Lisps offer fantastic development environment. The problem are developers who are mostly unable to use Lisp for work effectively, I guess due to too much power, freedom and complete lack of direction on how to structure your application.

tehjoker 5y ago

I tried experimenting with Common LISP but never understood how one was supposed to deploy a finished binary to the end user. It seemed like you were supposed to load up an environment? It made me leery of continuing.

aidenn0 5y ago
For most common-lisp implementations you can just save the current state as an executable, along with specifying an entry-point. Obviously doing this from your dev environment is a bad idea, so most people write a script that loads the system and dumps the image. Or you can use one that someone else has already written. Not all lisps work this way (most notably ECL does not), and you can use ASDF's program-op to create an executable directly from a package definition, which should work on any implemntation supported by ASDF.
- tehjoker 5y ago
  Thanks, the saving the dev environment part was the part that scared me. It certainly helps to have someone tell you you're not crazy. Maybe I'll give it another try sometime!
  - aidenn0 5y ago
    One of the things I've learned with powerful tools is that, by their nature, they empower you to do bad things almost as much as they empower you to do good things. It's even alluded to in the ruby bare-words example in this famous talk[1] 1: https://www.youtube.com/watch?v=3se2-thqf-A https://www.youtube.com/watch?v=3se2-thqf-A
  - foobarbaz33 5y ago
    You can just run make when you are ready to make a clean build and executable. Same as any other language.
  - _ph_ 5y ago
    Nothing to worry about. Yes, the minimal executable you get with SBCL is a bit fat, but for any real-world application which do some actual work, the size is a non-issue. Especially, as the executables are starting up very fast, as no dynamic linking is necessary, most of the code are static references. For any sizable Lisp project you would have an ASDF-file to load the system, with that you can create an easy build-script or even a make-file, which loads your program via ASDF and then builds the executable, it is described well here: https://stackoverflow.com/questions/14171849/compiling-common-lisp-to-an-executable https://stackoverflow.com/questions/14171849/compiling-commo... For consistancy, I would recommend to build the executable from the freshly loaded source, not a lisp image which has been used to develop in.

orthecreedence 5y ago

I miss my common lisp days, and I think rust being able to export C ABIs makes it a really great companion language for CL. I also think common lisp (or really, any fast lisp) is a really great tool for game development. The fact that you can redefine a program as it's running really helps iterate without having to set up your state over and over. Pair that with a fast, low-level language that can handle a lot of the lower level stuff (been trying to find time to look at Bevy on rust), and you have a killer combo. The main reason I stopped using lisp was because of the community. There were some amazing people that helped out with a lot of the stuff I was building, but not a critical mass of them and things just kind of stagnated. Then it seemed like for every project I poured my soul into, someone would write a one-off copy of it "just because" instead of contributing. It's definitely a culture of lone-wolf programmers. CL is a decent language with some really good implementations, and everywhere I go I miss the macros. I definitely want to pick it up again sometime, but probably will just let the async stuff I used to work on rest in peace.

a-dub 5y ago
i assume you're familiar with GOAL? https://en.wikipedia.org/wiki/Game_Oriented_Assembly_Lisp https://en.wikipedia.org/wiki/Game_Oriented_Assembly_Lisp
- orthecreedence 5y ago
  I was not! Thanks for the tip.
sahil-kang 5y ago
I can understand the failure to build a critical mass of contributors, but can you share some examples of where your work was duplicated instead of built upon? I’ve read through some of your async work in the past and from an initial glance, it seemed like you had the right idea by wrapping existing event libs and exposing familiar event loop idioms. At the very least, it seemed uncontroversial so I’m interested to see why others would choose not to build upon it.
- 5y ago

codr7 5y ago

I've been playing around with designing programming languages in Common Lisp lately. I'm curious how far it's possible to push performance by generating Lisp code in SBCL compared to the classical C interpreter goto loop. https://github.com/codr7/snabl https://github.com/codr7/snabl

temporallobe 5y ago

I regularly work with Clojure. This is probably an unpopular opinion in this thread, but despite having worked with it for years, I still don’t much like it, mostly because it’s far too terse and the syntax is so far removed from that of C-based languages. The other day I wrote a Java-based DTO and it was refreshing how clear and verbose everything was, despite the almost comical volume of code it took in comparison to what similar functionality would look look like in Clojure. Plus, the state of tooling in Clojure is not the best. I would also add that while you might initially do well with something like Clojure, it may be difficult to maintain, especially if you plan to make it a legacy product with O&M support in the future.

Guthur 5y ago
Well this article is about Common Lisp :). You could easily use CLOS or Structs in Common Lisp to do DTO. If you were using CLOS for DTOs you'd also have generic functions to dispatch on said DTOs.
jhhh 5y ago
The hard to maintain issue seems to follow most dynamic languages around. What kind of those issues have you run in to in your time working with Clojure professionally?
- Guthur 5y ago
  To be honest; "hard to maintain", "technical debt", "total rewrite required", "monolith to micro service", probably "micro service to monolith" at some point, all of these appear no matter what the language, I really don't think static analysis is the problem. Though I will give you that pretty much every popular dynamic language has terrible runtime environments, which doesn't help at all. If I don't have comprehensive static analysis I want good introspection and reflection, I'd ideally want static manipulation and continuation.
deleted 5y ago
[deleted]
deleted 5y ago
[deleted]

peatmoss 5y ago

I’ve been increasingly coming around to the opinion that computers have gotten kind of boring. I miss how weird and different computers used to be, both in terms of hardware and software. For lisps, I think Racket and Clojure feel the most modern. But I’m starting to come around to the idea that there are enough modern, hyper modern, post-modern, languages and programmers out there. Some of the value in learning Common Lisp might be in its value for living software archaeology. You can find actual code written at the time people were figuring out Big Ideas that we take for granted today. And usually the process of synthesis means that Big Ideas lose a lot of associated commentary and thinking that fed in. There’s a place for modern languages that conform to the same general set of sensibilities. But there’s also joy in gettin’ weird with some of the old stuff. It’s smugly satisfying to beat the new kid on the block with old tricks on occasion.