7 ms·

Statistical Analysis with Lisp-Stat

101 points by ngcc_hk 5y ago

WalterGR 5y ago
“Statistical Analysis with Lisp-Stat”
- ngcc_hk 5y ago
  It is more Common Lisp with an extension into my area, even though my first degree and jobs are all stat. Thanks for the suggestion. Just not how I thinking of it. In fact if you just want to do stat, I recommend spss, R, python … not really sure you will start with this and learn the lisp underlying. Clips-stat guy moved to develop R for a reason. Still I think it is lovely they restart this path. Love Common Lisp.
openfuture 5y ago
One thing to keep in mind is that common lisp is actually a super new and exciting language... All the implementations used to be proprietary for a long time (but there's been widespread acknowledgement of the merits of the language from the very beginning). Now that we've got super awesome FOSS implementations the party has gotten started and immediately there are so many crazy cool things coming out. I think common lisp is basically a better rust for a lot of the things rust is being used for.. Although of course rust is way better for many things also. It's not really an apples-to-apples comparison but the point is that common lisp is quite fast and good for application development (which rust is too but often with more complexity than necessary). Another thing to remember is this rule of thumb where if you want to make a neutral prior for how long a technology will be around you should basically just look at how long it has already existed. Something exists for a week, probably it will be around for another week. Something exists for a decade, probably it will be around for another decade. Of course this is not a very detailed assessment but SICL and CIEL are respectively efforts at minimizing the bootstrap and modernizing the final product. There will be more of these efforts and eventually lisp will become a totally different language (as we approach the energy minimum for the vocabulary)... Essentially what I am trying to say is that right now; lisp code is the code that I have the most confidence in w.r.t. longevity.
- aidenn0 5y ago
  I don't have exact dates, but I definitely used cmucl on linux/x86 before 2000 (though not much before; 1999 maybe?).
- neutronicus 5y ago
  IMO Common Lisp is extremely unlike Rust, which is basically (C++)++, and I would not use them for the same things
  - jhgb 5y ago
    CL is quite unlike Rust, but I've actually found lots of similarities between CL and C++: religiously stressing backwards compatibility and the ability to run very old software, providing the ability to hook and customize as many things as possible while providing default implementations, being unopiniated with regards to programming paradigms...
  - pjmlp 5y ago
    Symbolics and Xerox PARC beg to differ, and their products failed to establish themselves more due to their mismanagement than technical. Rust still needs to grow up quite a bit before it can assert to be (C++)++ specially in anything that involves a GPU.
- gnufx 5y ago
  Static for 25+ years, but "super new"?? Of implementations, CMUCL even predates Common Lisp, but I don't know when it implemented CL. GNU Common Lisp was available GPL'd around the time of the standard and, I think Clisp.
- perihelions 5y ago
  - "if you want to make a neutral prior for how long a technology will be around you should basically just look at how long it has already existed" AKA Lindy's effect. https://en.wikipedia.org/wiki/Lindy_effect https://en.wikipedia.org/wiki/Lindy_effect
- pjmlp 5y ago
  While FOSS implementations are great, their tooling is still missing the whole stack experience as offered by Allegro and Lispworks.
na85 5y ago
Not sure why OP didn't use TFA's actual title, but oh well. Lisp-Stat is maybe not as full-featured as its analogues in the python ecosystem, but it's still quite good. I recently developed an algorithmic trading bot in Common Lisp using statistical arbitrage methods and I used lisp-stat quite a bit in the development process, and while I enjoyed vega-lite (the plotting library that lisp-stat uses) I will say that I despite JSON as a format and found exporting to gnuplot scripts more flexible and useful.
- ngcc_hk 5y ago
  Deliberate. I think the original title is good for the writer. But for those are statisticians, I am not sure you should go there. R, pathos, SAS and SPSS (plus one I am not sure but is around) would be more common. Hence, the change to the emphasis. You know Common Lisp or Lisp, a good tool back to life.
aidenn0 5y ago
Not to be confused with XLispStat[1] 1: https://en.wikipedia.org/wiki/XLispStat https://en.wikipedia.org/wiki/XLispStat
- civilized 5y ago
  This was going to be my question. "Didn't statisticians abandon this for R like 25 years ago?"
  - ngcc_hk 5y ago
    Yes they did to my regrets. I did admit whilst xlisp-stat is much cleaner implementation for stat. and is my entry to Common Lisp (via simulation of social scientist 1st edition). But lisp is in the way, sort of, for many. It is nice to have a common lisp version to use in statistical area, for those who does not hinder by it.
- kilbuz 5y ago
  Ah I did get confused. I have Luke's book from 1990 still!
- kwccoin 5y ago
  There is some migration aid for this, as it is really common lisp based statistical package like xlisp. https://lisp-stat.dev/docs/resources/xlisp/ https://lisp-stat.dev/docs/resources/xlisp/
wodenokoto 5y ago
People who promote the virtues of lisp online usually talk about the powerful repl. Here they advertise jupyter notebooks. Does that give that lisp repl experience I’ve heard so much about or is it just like my familiar r and Python repl experience?
- kazinator 5y ago
  Lisp is excellent even if you write every line of code in a file, and compile every time you test, using a fresh image (using the REPL only for some exploration of the code committed in files, and debugging).
  - ngcc_hk 5y ago
    I do not like emacs and is a vim guy. With that https://news.ycombinator.com/item?id=30812912 https://news.ycombinator.com/item?id=30812912 for using Slimv and vim (the source talked about slime as well, but I am not sure I like that). I can do the expression by expression and buffer by buffer kind of building an image, not just file by file. In this regards, the notebook approach is very different. I am still learning from this site about this under common lisp. Based on python notebook (and R), I think it helps after you build your system you can use this to help others to understand what you have build.
- pjmlp 5y ago
  For that you need to dive into Lisp commercial offerings. Jupiter notebooks are the poor man's experience of what using a Symbolics or Interlisp-D workstation used to be like.
- jhgb 5y ago
  The REPL experience in Lisp stems from how Lisp works; notebooks on top of Python don't remove Python's fundamental limitations. You can put a better interface on top of Python and it will still have Python's limitations, such as the necessity to create new instances if you redefine a class as opposed to old instances adapting to redefined structure and behavior.
piokoch 5y ago
Wow, I didn't know this is still alive. I used this project for some strange reason good 15 years ago. The most funky feature it had (still has?) is data visualization in 6 dimensions (yes, six), it was done as a kind of rotating plot that is switching between variables to show three each time. Very cool, although rather useless for average human.

101 points by ngcc_hk 5y ago

WalterGR 5y ago

“Statistical Analysis with Lisp-Stat”

ngcc_hk 5y ago
It is more Common Lisp with an extension into my area, even though my first degree and jobs are all stat. Thanks for the suggestion. Just not how I thinking of it. In fact if you just want to do stat, I recommend spss, R, python … not really sure you will start with this and learn the lisp underlying. Clips-stat guy moved to develop R for a reason. Still I think it is lovely they restart this path. Love Common Lisp.

openfuture 5y ago

One thing to keep in mind is that common lisp is actually a super new and exciting language... All the implementations used to be proprietary for a long time (but there's been widespread acknowledgement of the merits of the language from the very beginning). Now that we've got super awesome FOSS implementations the party has gotten started and immediately there are so many crazy cool things coming out. I think common lisp is basically a better rust for a lot of the things rust is being used for.. Although of course rust is way better for many things also. It's not really an apples-to-apples comparison but the point is that common lisp is quite fast and good for application development (which rust is too but often with more complexity than necessary). Another thing to remember is this rule of thumb where if you want to make a neutral prior for how long a technology will be around you should basically just look at how long it has already existed. Something exists for a week, probably it will be around for another week. Something exists for a decade, probably it will be around for another decade. Of course this is not a very detailed assessment but SICL and CIEL are respectively efforts at minimizing the bootstrap and modernizing the final product. There will be more of these efforts and eventually lisp will become a totally different language (as we approach the energy minimum for the vocabulary)... Essentially what I am trying to say is that right now; lisp code is the code that I have the most confidence in w.r.t. longevity.

aidenn0 5y ago
I don't have exact dates, but I definitely used cmucl on linux/x86 before 2000 (though not much before; 1999 maybe?).
neutronicus 5y ago
IMO Common Lisp is extremely unlike Rust, which is basically (C++)++, and I would not use them for the same things
- jhgb 5y ago
  CL is quite unlike Rust, but I've actually found lots of similarities between CL and C++: religiously stressing backwards compatibility and the ability to run very old software, providing the ability to hook and customize as many things as possible while providing default implementations, being unopiniated with regards to programming paradigms...
- pjmlp 5y ago
  Symbolics and Xerox PARC beg to differ, and their products failed to establish themselves more due to their mismanagement than technical. Rust still needs to grow up quite a bit before it can assert to be (C++)++ specially in anything that involves a GPU.
gnufx 5y ago
Static for 25+ years, but "super new"?? Of implementations, CMUCL even predates Common Lisp, but I don't know when it implemented CL. GNU Common Lisp was available GPL'd around the time of the standard and, I think Clisp.
perihelions 5y ago
- "if you want to make a neutral prior for how long a technology will be around you should basically just look at how long it has already existed" AKA Lindy's effect. https://en.wikipedia.org/wiki/Lindy_effect https://en.wikipedia.org/wiki/Lindy_effect
pjmlp 5y ago
While FOSS implementations are great, their tooling is still missing the whole stack experience as offered by Allegro and Lispworks.

na85 5y ago

Not sure why OP didn't use TFA's actual title, but oh well. Lisp-Stat is maybe not as full-featured as its analogues in the python ecosystem, but it's still quite good. I recently developed an algorithmic trading bot in Common Lisp using statistical arbitrage methods and I used lisp-stat quite a bit in the development process, and while I enjoyed vega-lite (the plotting library that lisp-stat uses) I will say that I despite JSON as a format and found exporting to gnuplot scripts more flexible and useful.

ngcc_hk 5y ago
Deliberate. I think the original title is good for the writer. But for those are statisticians, I am not sure you should go there. R, pathos, SAS and SPSS (plus one I am not sure but is around) would be more common. Hence, the change to the emphasis. You know Common Lisp or Lisp, a good tool back to life.

aidenn0 5y ago

Not to be confused with XLispStat[1] 1: https://en.wikipedia.org/wiki/XLispStat https://en.wikipedia.org/wiki/XLispStat

civilized 5y ago
This was going to be my question. "Didn't statisticians abandon this for R like 25 years ago?"
- ngcc_hk 5y ago
  Yes they did to my regrets. I did admit whilst xlisp-stat is much cleaner implementation for stat. and is my entry to Common Lisp (via simulation of social scientist 1st edition). But lisp is in the way, sort of, for many. It is nice to have a common lisp version to use in statistical area, for those who does not hinder by it.
kilbuz 5y ago
Ah I did get confused. I have Luke's book from 1990 still!
kwccoin 5y ago
There is some migration aid for this, as it is really common lisp based statistical package like xlisp. https://lisp-stat.dev/docs/resources/xlisp/ https://lisp-stat.dev/docs/resources/xlisp/

wodenokoto 5y ago

People who promote the virtues of lisp online usually talk about the powerful repl. Here they advertise jupyter notebooks. Does that give that lisp repl experience I’ve heard so much about or is it just like my familiar r and Python repl experience?

kazinator 5y ago
Lisp is excellent even if you write every line of code in a file, and compile every time you test, using a fresh image (using the REPL only for some exploration of the code committed in files, and debugging).
- ngcc_hk 5y ago
  I do not like emacs and is a vim guy. With that https://news.ycombinator.com/item?id=30812912 https://news.ycombinator.com/item?id=30812912 for using Slimv and vim (the source talked about slime as well, but I am not sure I like that). I can do the expression by expression and buffer by buffer kind of building an image, not just file by file. In this regards, the notebook approach is very different. I am still learning from this site about this under common lisp. Based on python notebook (and R), I think it helps after you build your system you can use this to help others to understand what you have build.
pjmlp 5y ago
For that you need to dive into Lisp commercial offerings. Jupiter notebooks are the poor man's experience of what using a Symbolics or Interlisp-D workstation used to be like.
jhgb 5y ago
The REPL experience in Lisp stems from how Lisp works; notebooks on top of Python don't remove Python's fundamental limitations. You can put a better interface on top of Python and it will still have Python's limitations, such as the necessity to create new instances if you redefine a class as opposed to old instances adapting to redefined structure and behavior.

piokoch 5y ago

Wow, I didn't know this is still alive. I used this project for some strange reason good 15 years ago. The most funky feature it had (still has?) is data visualization in 6 dimensions (yes, six), it was done as a kind of rotating plot that is switching between variables to show three each time. Very cool, although rather useless for average human.