3 ms·

I found his remarks about how (modern, ML-based) AI is the winter for Lisp interesting and congruent with how I see things currently. At the end of the day, man

I found his remarks about how (modern, ML-based) AI is the winter for Lisp interesting and congruent with how I see things currently. At the end of the day, many of the hardest problems in computing are solved best by simply throwing more compute (esp. matrix and array), memory, and data at them. Today's Lisp machines are GPGPU.

mepian 4y ago
In my view, GPGPU is much closer to contemporary Connection Machines, which used Lisp machines for their front-ends. Now the front-end is x86-64 machines running Python instead of Lisp. Guy Steele wrote the first edition of Common Lisp the Language, the original de facto standard, while working for the makers of Connection Machines.
- larve 4y ago
  Similar. So much of llm is about language design and leveraging being able to quickly manipulate semi formal dsls that I think lisp is a brilliant “driver”, so much more so than then current “string templates and regex parsing” approaches.
zackmorris 4y ago
I've been surviving a winter of my own since 1997 when video cards took over and Moore's Law for CPUs started slowing down, finally ending around 2007 when smartphones arrived and research dollars got redirected to cost and power reduction. We also got stuck with stopgap SIMD measures like SSE/AltiVec instead of a truly scalable multicore/multiprocessor MIMD architecture. Which made it much more difficult to experiment with stuff like genetic algorithms or any other machine learning algorithm that don't map well to vectors. Because a 10 billion transistor GPU that reports thousands of shading units may only have 64 physical cores. Whereas something like a 1 GHz DEC Alpha with 15 million transistors would give us more on the order of 100-1000 cores on the same die, especially if the cache was removed and it used a distributed content-addressable memory for data locality and a near-linear bandwidth increase for workloads like map-reduce. Anyway, maybe something like MIMD could be built on GPGPU? I just want to get info on my system and see 1024+ threads and 10 TB/sec memory bandwidth on a chip that costs no more than $1000. The fact that this doesn't exist today after all these years, and that people are perfectly happy to keep repurchasing the same desktop computer the same speed as ones from a decade ago, and on top of that go to the ends of the earth to get the simplest shader code working (rather than just coding directly in stuff like Rust/D/C#/Java/C++ and Docker and/or Lisp) just absolutely blows my mind. https://en.wikipedia.org/wiki/Multiple_instruction,_multiple_data https://en.wikipedia.org/wiki/Multiple_instruction,_multiple... https://en.wikipedia.org/wiki/Transputer https://en.wikipedia.org/wiki/Transputer http://aggregate.org/MOG/ http://aggregate.org/MOG/ https://www.microsoft.com/en-us/research/video/mimd-on-gpu/ https://www.microsoft.com/en-us/research/video/mimd-on-gpu/
- mepian 4y ago
  >We also got stuck with stopgap SIMD measures like SSE/AltiVec instead of a truly scalable multicore/multiprocessor MIMD architecture. We did have the Xeon Phi until recently: https://en.wikipedia.org/wiki/Xeon_Phi https://en.wikipedia.org/wiki/Xeon_Phi It was a lot of simple (derived from the original Pentium) x86 cores on the same die, as you want.
- LargoLasskhyfv 4y ago
  Have you heard of https://en.wikipedia.org/wiki/Libre-SOC https://en.wikipedia.org/wiki/Libre-SOC ? Would be nice if scaled up, I think. Give it a few years time :-)
tjr 4y ago
I wonder, why single Lisp out for a winter? If solving computational problems moves away from writing programs and toward writing GPT prompts, then wouldn't that forecast a winter for every programming language?
- comfypotato 4y ago
  I think Lisp is singled out here because it was the original AI language. The original Lisp machine was very expensive, and it was predominantly purchased by institutions for the purpose of AI research. My source for this is just tidbits I’ve picked up over the years, so I’m not certain it’s factual.
  - anthk 4y ago
    Now you have a libre licensed LISP Machine emulator with a libre firmware: https://tumbleweed.nu https://tumbleweed.nu
- gjadi 4y ago
  IMHO it's because lisp shines to manipulate symbols whereas the current AI trend is crunching matrices. When AI was about building grammars, trees, developing expert systems builds rules etc. symbol manipulation was king. Look at PAIP for some examples: https://github.com/norvig/paip-lisp https://github.com/norvig/paip-lisp This paradigm has changed.
  - kazinator 4y ago
    That stuff works, and requires very low resources by today's standards. You can do symbolic processing in a tiny, low-powered embedded system with no cloud access. Even today, the accomplishment of SHRDLU remains impressive. And you can actually go into the code to understand why and how it comes up with answers.
    - anthk 4y ago
      Eliza on Emacs (ok, Doctor) it's a good example on this. Pick any Pentium, 486, or RpiB, it will run at crazy speeds. And you can edit it in ELISP to enhance the "personality". Even if Elisp is not the fastest LISP here at all, it runs fine. Now if you use SBCL, or cross the parallel Scheme road with JIT'ted Guile3 or use natively compiled Chicken, the performance skyrockets.

mepian 4y ago

In my view, GPGPU is much closer to contemporary Connection Machines, which used Lisp machines for their front-ends. Now the front-end is x86-64 machines running Python instead of Lisp. Guy Steele wrote the first edition of Common Lisp the Language, the original de facto standard, while working for the makers of Connection Machines.

larve 4y ago
Similar. So much of llm is about language design and leveraging being able to quickly manipulate semi formal dsls that I think lisp is a brilliant “driver”, so much more so than then current “string templates and regex parsing” approaches.

zackmorris 4y ago

I've been surviving a winter of my own since 1997 when video cards took over and Moore's Law for CPUs started slowing down, finally ending around 2007 when smartphones arrived and research dollars got redirected to cost and power reduction. We also got stuck with stopgap SIMD measures like SSE/AltiVec instead of a truly scalable multicore/multiprocessor MIMD architecture. Which made it much more difficult to experiment with stuff like genetic algorithms or any other machine learning algorithm that don't map well to vectors. Because a 10 billion transistor GPU that reports thousands of shading units may only have 64 physical cores. Whereas something like a 1 GHz DEC Alpha with 15 million transistors would give us more on the order of 100-1000 cores on the same die, especially if the cache was removed and it used a distributed content-addressable memory for data locality and a near-linear bandwidth increase for workloads like map-reduce. Anyway, maybe something like MIMD could be built on GPGPU? I just want to get info on my system and see 1024+ threads and 10 TB/sec memory bandwidth on a chip that costs no more than $1000. The fact that this doesn't exist today after all these years, and that people are perfectly happy to keep repurchasing the same desktop computer the same speed as ones from a decade ago, and on top of that go to the ends of the earth to get the simplest shader code working (rather than just coding directly in stuff like Rust/D/C#/Java/C++ and Docker and/or Lisp) just absolutely blows my mind. https://en.wikipedia.org/wiki/Multiple_instruction,_multiple_data https://en.wikipedia.org/wiki/Multiple_instruction,_multiple... https://en.wikipedia.org/wiki/Transputer https://en.wikipedia.org/wiki/Transputer http://aggregate.org/MOG/ http://aggregate.org/MOG/ https://www.microsoft.com/en-us/research/video/mimd-on-gpu/ https://www.microsoft.com/en-us/research/video/mimd-on-gpu/

mepian 4y ago
>We also got stuck with stopgap SIMD measures like SSE/AltiVec instead of a truly scalable multicore/multiprocessor MIMD architecture. We did have the Xeon Phi until recently: https://en.wikipedia.org/wiki/Xeon_Phi https://en.wikipedia.org/wiki/Xeon_Phi It was a lot of simple (derived from the original Pentium) x86 cores on the same die, as you want.
LargoLasskhyfv 4y ago
Have you heard of https://en.wikipedia.org/wiki/Libre-SOC https://en.wikipedia.org/wiki/Libre-SOC ? Would be nice if scaled up, I think. Give it a few years time :-)

tjr 4y ago

I wonder, why single Lisp out for a winter? If solving computational problems moves away from writing programs and toward writing GPT prompts, then wouldn't that forecast a winter for every programming language?

comfypotato 4y ago
I think Lisp is singled out here because it was the original AI language. The original Lisp machine was very expensive, and it was predominantly purchased by institutions for the purpose of AI research. My source for this is just tidbits I’ve picked up over the years, so I’m not certain it’s factual.
- anthk 4y ago
  Now you have a libre licensed LISP Machine emulator with a libre firmware: https://tumbleweed.nu https://tumbleweed.nu
gjadi 4y ago
IMHO it's because lisp shines to manipulate symbols whereas the current AI trend is crunching matrices. When AI was about building grammars, trees, developing expert systems builds rules etc. symbol manipulation was king. Look at PAIP for some examples: https://github.com/norvig/paip-lisp https://github.com/norvig/paip-lisp This paradigm has changed.
- kazinator 4y ago
  That stuff works, and requires very low resources by today's standards. You can do symbolic processing in a tiny, low-powered embedded system with no cloud access. Even today, the accomplishment of SHRDLU remains impressive. And you can actually go into the code to understand why and how it comes up with answers.
  - anthk 4y ago
    Eliza on Emacs (ok, Doctor) it's a good example on this. Pick any Pentium, 486, or RpiB, it will run at crazy speeds. And you can edit it in ELISP to enhance the "personality". Even if Elisp is not the fastest LISP here at all, it runs fine. Now if you use SBCL, or cross the parallel Scheme road with JIT'ted Guile3 or use natively compiled Chicken, the performance skyrockets.