8 ms·

Did we really “lose” the Lisp Machine, or did we just realize that compiling Lisp to a general-purpose instruction set like x86 and then having someone else in

by runeks 6y ago

Did we really “lose” the Lisp Machine, or did we just realize that compiling Lisp to a general-purpose instruction set like x86 and then having someone else in charge of creating hardware to efficiently execute that is much more clever?

ori_b 6y ago
We lost the environment where the whole OS was a debugger, and everything could be inspected and modified as the system was running.
- zabzonk 6y ago
  We still have Smalltalk.
  - abzug 6y ago
    I know what you are meaning but in reality we don't have.
    - jcelerier 6y ago
      yet there are likely an order of magnitude more smalltalk programmers today than there were at smalltalk's "peak"
      - spillguard 6y ago
        I'd be curious to know how you quantified this. I was curious myself, so I dug into where Smalltalk landed in Stack Overflow's developer survey (which, of course, is not a complete overview of the entire programming industry!) - it seems that Smalltalk was the second most Loved language in 2017 [0], but it completely disappears off the chart in 2018 [1]. I would imagine that we've lost quite a few Smalltalk programmers... [0] https://insights.stackoverflow.com/survey/2017 https://insights.stackoverflow.com/survey/2017 [1] https://insights.stackoverflow.com/survey/2018 https://insights.stackoverflow.com/survey/2018
        xkriva11 6y ago
        More likely, they changed the conditions.
- cb321 6y ago
  The linked video seems mostly about UI usage, and one rarely sees nice live action UI demos for machines that (mostly) no longer exist. They should do a sequel that shows these other very powerful system features you mention. One practical downside is that the live editing / code & continue / source code-free operation (mentioned in the Interlisp-D segment of the video) could lead to needing "just the right image" with all the installed definitions you wanted. Sometimes "source code" might not even exist which could lead to coordination issues.
  - thequux 6y ago
    Sadly the deep dive got cut due to time limitations (both in terms of time during the talk and finding out that all of my alphas were in various states of non-working as I started preparing that segment) Had I had time, I would have patched the NFS code to run in "permissionless" mode, where it asks what creds are necessary to access a file, then just sends those creds. Hopefully somebody will be inspired to explore and blog about it though :-D
- burlesona 6y ago
  Except, that’s what the browser is, and increasingly that’s where software lives.
  - layoutIfNeeded 6y ago
    The browser? Oh, you mean the WebAssembly Virtual Machine?
    - oconnor663 6y ago
      Can we point to the S-expression syntax for Wasm and say we've come full circle? :)
  - yawn 6y ago
    The browser is only half (the client-facing side) of the equation.
    - iso8859-1 6y ago
      GraphQL endpoints are introspectable. A model with unrestricted introspectability wouldn't have worked as a web-replacement anyway, it is not secure enough! There are even servers like Servant for Haskell that make REST a first-class citizen of the server programming language. So we have the best of both worlds now: security and convenience.
- pjc50 6y ago
  How does that work when you drop to the debugger in the middle of an interrupt handler? A page fault? A task switch? How does the privilege separation work for that environment? (mostly rhetorical questions)
  - thequux 6y ago
    You'd think I was an expert, given that I gave that segment of the talk, but I haven't yet dived that deep into Genera's internals. As I understand it though, you don't have the concept of page faults at the image level; IIRC, Genera doesn't have any form of virtual memory, or if it does, it's all in microcode. Task switches are, iirc, handled in the image, but in the same way that it's done in Smalltalk: there's a function in the "kernel" to decide what task to switch to next, but the actual context switch is handled in microcode. Also, there's no such thing as privilege separation; everything's running in a single address space and namespacing is handled using Common Lisp's package system.
    - hestefisk 6y ago
      It was intentionally designed as a single user mode-machine wasn’t it?
      - db48x 6y ago
        Absolutely. There was no isolation between programs. In fact, you could write new programs by reusing parts of existing programs.
    - rjsw 6y ago
      The closest modern equivalent is probably a paravirtualized unikernel.
  - rjsw 6y ago
    None of those actions were visible to the virtual machine that contained the debugger. The CPU didn't even have the concept of an interrupt. The MIT Lisp Machine ran a simple kernel written in assembler that also included an interpreter for the user-visible instruction set, it would check periodically whether devices had raised an interrupt then return back into Lisp to handle the event.
- thu2111 6y ago
  Well, did we? You can still do that! Those environments were extremely primitive and it doesn't take much work to match or vastly exceed their capabilities. This is so much the case we often don't even recognize such environments staring us in the face. Consider another story on the HN front page right now: IntelliJ 2021 is out. Load IntelliJ and you have something that which can be arbitrarily scripted, altered and extended as it runs, and which appears to match/exceed Genera and the other demo'd systems. Here are some features of IntelliJ that are reminiscent of a single-address-space fully reflective operating system: • A multi-language scripting console that lets you access arbitrary parts of the IDE and give it interactive commands, either via its real API or (using reflection) anything else: https://www.jetbrains.com/help/idea/ide-scripting-console.html https://www.jetbrains.com/help/idea/ide-scripting-console.ht... • An interactive debugger that can debug any app, including the IDE itself. Of course it can be risky to use breakpoints as you might suspend the debugger you're using and get stuck, but that's a conceptual issue, not a technical one. • An interactive auto-completing command line like thing for commands, files, symbols, menu items etc. Double-shift opens it for me. • Real-time CPU, memory and task monitors. Try opening the command palette, type CPU and then pick "CPU and memory live charts", or search for "activity monitor" and open it. • A fully virtualized filesystem layer with sophisticated file explorer. You can explore 'into' files at the structural/symbolic level. • Obviously it's the JVM so everything is fully garbage collected. • A whole UI framework that supports both tiling and floating windows. • The ability to load, unload and upgrade apps (plugins) on the fly without restarts. They can alter the behaviour of the environment and other plugins, either via real APIs or by the big hammer of loading a JVM agent on the fly. JVM agents can rewrite code in any arbitrary way and swap it into the app as it's running. And you can program it with Lisp if you really want to, via Clojure. I watched the talk, it was a nice talk, but I couldn't help feeling that the answer to "What have we lost?" is actually - cynically - not much. Those environments had some interesting ideas and the good ones seem to have made it into production, just not via custom hardware and operating systems. Edit: interesting how quick this hit -1. I think a lot of people like the idea of lost treasures in computing. I used to like it too. But I can't identify something these machines or operating systems do that is strictly superior to modern technology or large-scale 'operating environments' like IntelliJ.
  - 6y ago
- salawat 6y ago
  This was an intentional move even if no one else wants to admit it. You can't hide implementation from the User in such an environment; at least not to the degree you can with precompiled programs with debug symbols stripped. One requires you to infer function from raw assembly, the other, lets you step through areas of code that vendors want to keep out of the reach of user understanding. 90% of established security practices around code delivery are around keeping your product from being too easily reversed or "owned" by those who use it. Thus the popularity of -as-a-Service business models. You can't invoice for something a user just uses.
  - db48x 6y ago
    True enough. Thankfully we still have Emacs.
  - AtlasBarfed 6y ago
    Operating Systems that are actually in use show that the real difficulty of Computer Science isn't deeply complex algorithmic complexity (although OSs have algorithms and data structures in spades) The real problem is just mundane management / organization / interface / documentation that plain old enterprisey software fails on. A lot of what is lost was handling a much smaller breadth of hardware, applications, and as you pointed out, a much simpler security exposure profile.
- deleted 6y ago
  [deleted]
HugoDaniel 6y ago
Fitting Lisp in a stack based machine (or vice-versa) is mostly a compromise, not exactly a "realisation" of a "clever" method, because Lisp makes extensive use of list appends/concats, environments and garbage collection, which are not an easy fit for heap/stack C abstractions.
- runeks 6y ago
  I challenge you to create a piece of hardware that can execute Lisp faster than a modern x86 CPU. This compromise was chosen because you can’t without spending billion of dollars.
  - HugoDaniel 6y ago
    which does not imply that we haven't lost the architecture and hardware innovation associated with it
EdwardCoffin 6y ago
I've found that the best explanation of what we really lost is in some of the comments by Kent Pitman in some comp.lang.lisp discussions, like this one: [1] and this one [2] [1] https://groups.google.com/g/comp.lang.lisp/c/QzKZCbf-S6g/m/KnsOdcyWzc8J https://groups.google.com/g/comp.lang.lisp/c/QzKZCbf-S6g/m/K... [2] https://groups.google.com/g/comp.lang.lisp/c/XpvUwF2xKbk/m/o6CKxgzNLFUJ https://groups.google.com/g/comp.lang.lisp/c/XpvUwF2xKbk/m/o...
dkersten 6y ago
My takeaway from the first segment wasn't the Lisp part, but the OS and user interface, especially that any "object" can provide handlers that then provide deep and highly dynamic integration with the rest of the system, in a react-like immediate mode style to boot. And a very slick merging of CLI and GUI. All back in the 80's!

by runeks 6y ago

ori_b 6y ago

We lost the environment where the whole OS was a debugger, and everything could be inspected and modified as the system was running.

zabzonk 6y ago
We still have Smalltalk.
- abzug 6y ago
  I know what you are meaning but in reality we don't have.
  - jcelerier 6y ago
    yet there are likely an order of magnitude more smalltalk programmers today than there were at smalltalk's "peak"
    - spillguard 6y ago
      I'd be curious to know how you quantified this. I was curious myself, so I dug into where Smalltalk landed in Stack Overflow's developer survey (which, of course, is not a complete overview of the entire programming industry!) - it seems that Smalltalk was the second most Loved language in 2017 [0], but it completely disappears off the chart in 2018 [1]. I would imagine that we've lost quite a few Smalltalk programmers... [0] https://insights.stackoverflow.com/survey/2017 https://insights.stackoverflow.com/survey/2017 [1] https://insights.stackoverflow.com/survey/2018 https://insights.stackoverflow.com/survey/2018
      - xkriva11 6y ago
        More likely, they changed the conditions.
cb321 6y ago
The linked video seems mostly about UI usage, and one rarely sees nice live action UI demos for machines that (mostly) no longer exist. They should do a sequel that shows these other very powerful system features you mention. One practical downside is that the live editing / code & continue / source code-free operation (mentioned in the Interlisp-D segment of the video) could lead to needing "just the right image" with all the installed definitions you wanted. Sometimes "source code" might not even exist which could lead to coordination issues.
- thequux 6y ago
  Sadly the deep dive got cut due to time limitations (both in terms of time during the talk and finding out that all of my alphas were in various states of non-working as I started preparing that segment) Had I had time, I would have patched the NFS code to run in "permissionless" mode, where it asks what creds are necessary to access a file, then just sends those creds. Hopefully somebody will be inspired to explore and blog about it though :-D
burlesona 6y ago
Except, that’s what the browser is, and increasingly that’s where software lives.
- layoutIfNeeded 6y ago
  The browser? Oh, you mean the WebAssembly Virtual Machine?
  - oconnor663 6y ago
    Can we point to the S-expression syntax for Wasm and say we've come full circle? :)
- yawn 6y ago
  The browser is only half (the client-facing side) of the equation.
  - iso8859-1 6y ago
    GraphQL endpoints are introspectable. A model with unrestricted introspectability wouldn't have worked as a web-replacement anyway, it is not secure enough! There are even servers like Servant for Haskell that make REST a first-class citizen of the server programming language. So we have the best of both worlds now: security and convenience.
pjc50 6y ago
How does that work when you drop to the debugger in the middle of an interrupt handler? A page fault? A task switch? How does the privilege separation work for that environment? (mostly rhetorical questions)
- thequux 6y ago
  You'd think I was an expert, given that I gave that segment of the talk, but I haven't yet dived that deep into Genera's internals. As I understand it though, you don't have the concept of page faults at the image level; IIRC, Genera doesn't have any form of virtual memory, or if it does, it's all in microcode. Task switches are, iirc, handled in the image, but in the same way that it's done in Smalltalk: there's a function in the "kernel" to decide what task to switch to next, but the actual context switch is handled in microcode. Also, there's no such thing as privilege separation; everything's running in a single address space and namespacing is handled using Common Lisp's package system.
  - hestefisk 6y ago
    It was intentionally designed as a single user mode-machine wasn’t it?
    - db48x 6y ago
      Absolutely. There was no isolation between programs. In fact, you could write new programs by reusing parts of existing programs.
  - rjsw 6y ago
    The closest modern equivalent is probably a paravirtualized unikernel.
- rjsw 6y ago
  None of those actions were visible to the virtual machine that contained the debugger. The CPU didn't even have the concept of an interrupt. The MIT Lisp Machine ran a simple kernel written in assembler that also included an interpreter for the user-visible instruction set, it would check periodically whether devices had raised an interrupt then return back into Lisp to handle the event.
thu2111 6y ago
Well, did we? You can still do that! Those environments were extremely primitive and it doesn't take much work to match or vastly exceed their capabilities. This is so much the case we often don't even recognize such environments staring us in the face. Consider another story on the HN front page right now: IntelliJ 2021 is out. Load IntelliJ and you have something that which can be arbitrarily scripted, altered and extended as it runs, and which appears to match/exceed Genera and the other demo'd systems. Here are some features of IntelliJ that are reminiscent of a single-address-space fully reflective operating system: • A multi-language scripting console that lets you access arbitrary parts of the IDE and give it interactive commands, either via its real API or (using reflection) anything else: https://www.jetbrains.com/help/idea/ide-scripting-console.html https://www.jetbrains.com/help/idea/ide-scripting-console.ht... • An interactive debugger that can debug any app, including the IDE itself. Of course it can be risky to use breakpoints as you might suspend the debugger you're using and get stuck, but that's a conceptual issue, not a technical one. • An interactive auto-completing command line like thing for commands, files, symbols, menu items etc. Double-shift opens it for me. • Real-time CPU, memory and task monitors. Try opening the command palette, type CPU and then pick "CPU and memory live charts", or search for "activity monitor" and open it. • A fully virtualized filesystem layer with sophisticated file explorer. You can explore 'into' files at the structural/symbolic level. • Obviously it's the JVM so everything is fully garbage collected. • A whole UI framework that supports both tiling and floating windows. • The ability to load, unload and upgrade apps (plugins) on the fly without restarts. They can alter the behaviour of the environment and other plugins, either via real APIs or by the big hammer of loading a JVM agent on the fly. JVM agents can rewrite code in any arbitrary way and swap it into the app as it's running. And you can program it with Lisp if you really want to, via Clojure. I watched the talk, it was a nice talk, but I couldn't help feeling that the answer to "What have we lost?" is actually - cynically - not much. Those environments had some interesting ideas and the good ones seem to have made it into production, just not via custom hardware and operating systems. Edit: interesting how quick this hit -1. I think a lot of people like the idea of lost treasures in computing. I used to like it too. But I can't identify something these machines or operating systems do that is strictly superior to modern technology or large-scale 'operating environments' like IntelliJ.
- 6y ago
salawat 6y ago
This was an intentional move even if no one else wants to admit it. You can't hide implementation from the User in such an environment; at least not to the degree you can with precompiled programs with debug symbols stripped. One requires you to infer function from raw assembly, the other, lets you step through areas of code that vendors want to keep out of the reach of user understanding. 90% of established security practices around code delivery are around keeping your product from being too easily reversed or "owned" by those who use it. Thus the popularity of -as-a-Service business models. You can't invoice for something a user just uses.
- db48x 6y ago
  True enough. Thankfully we still have Emacs.
- AtlasBarfed 6y ago
  Operating Systems that are actually in use show that the real difficulty of Computer Science isn't deeply complex algorithmic complexity (although OSs have algorithms and data structures in spades) The real problem is just mundane management / organization / interface / documentation that plain old enterprisey software fails on. A lot of what is lost was handling a much smaller breadth of hardware, applications, and as you pointed out, a much simpler security exposure profile.
deleted 6y ago
[deleted]

HugoDaniel 6y ago

Fitting Lisp in a stack based machine (or vice-versa) is mostly a compromise, not exactly a "realisation" of a "clever" method, because Lisp makes extensive use of list appends/concats, environments and garbage collection, which are not an easy fit for heap/stack C abstractions.

runeks 6y ago
I challenge you to create a piece of hardware that can execute Lisp faster than a modern x86 CPU. This compromise was chosen because you can’t without spending billion of dollars.
- HugoDaniel 6y ago
  which does not imply that we haven't lost the architecture and hardware innovation associated with it

EdwardCoffin 6y ago

I've found that the best explanation of what we really lost is in some of the comments by Kent Pitman in some comp.lang.lisp discussions, like this one: [1] and this one [2] [1] https://groups.google.com/g/comp.lang.lisp/c/QzKZCbf-S6g/m/KnsOdcyWzc8J https://groups.google.com/g/comp.lang.lisp/c/QzKZCbf-S6g/m/K... [2] https://groups.google.com/g/comp.lang.lisp/c/XpvUwF2xKbk/m/o6CKxgzNLFUJ https://groups.google.com/g/comp.lang.lisp/c/XpvUwF2xKbk/m/o...

dkersten 6y ago

My takeaway from the first segment wasn't the Lisp part, but the OS and user interface, especially that any "object" can provide handlers that then provide deep and highly dynamic integration with the rest of the system, in a react-like immediate mode style to boot. And a very slick merging of CLI and GUI. All back in the 80's!