6 ms·

Symbolics Genera - The Best Software Environment Available

51 points by udzinari 16y ago

julian37 16y ago
Wikipedia has more information for people (like me) who haven't heard of Genera before: http://en.wikipedia.org/wiki/Genera_(operating_system) http://en.wikipedia.org/wiki/Genera_(operating_system)
ScottBurson 16y ago
I have a Symbolics XL1200 that still works (at least, it worked the last time I turned it on, which was a few years ago). I'd be happy to demo it at YC sometime. It was a fun environment to work in. Although the hardware was very slow by modern standards, the environment was designed to maximize hacker productivity. For example, you didn't have to run a program under the debugger to debug it -- the debugger was always there. Any program that hit an error would drop you into the debugger. One keystroke would then take you to the source for the current function. You could change that function, recompile it, and restart from the point where the function was called. The environment was completely open; there was no access control of any kind. There was also only a single address space. Did that make it crash-prone? Remarkably, no. The hardware tagging and bounds checking kept programs from stepping on objects they didn't own. Of course such a design would never survive outside of a research lab, but it made it amazingly easy to hack almost anything in the system.
- daeken 16y ago
  I don't know that I'd agree that it wouldn't survive outside of research. In recent years, there's been a huge push for pure-managed operating systems, where everything runs in one address space and depends on the compiler to ensure security and reliability. While no such systems are in practice (to my knowledge), the potential is incredible. We'll see how it pans out in the next few years. (Full disclosure: I started such an OS, Renraku, so I have a bit of a vested interest in the success of such ideas)
  - joe_bleau 16y ago
    What about IBM's OS/400? http://users.snip.net/~gbooker/AS400/ibm.htm http://users.snip.net/~gbooker/AS400/ibm.htm
    - hga 16y ago
      Indeed, OS/400 is just wonderfully advanced under the hood. Get one of the cheap used copies of this book for a lot more: http://www.amazon.com/Inside-AS-400-Frank-Soltis/dp/1882419669/ http://www.amazon.com/Inside-AS-400-Frank-Soltis/dp/18824196...
- sedachv 16y ago
  Hi Scott, didn't see this comment before I posted mine (take a look at it if you have the time). Thank you for writing Zeta-C and proving the concept of reasonable C->Lisp compilation, and for making the code public domain.
  - ScottBurson 16y ago
    You're welcome. I hope it makes for an interesting curiosity -- I'm not sure it can be much more at this point :-)
sedachv 16y ago
Lisp Machines are something that you think is really cool when you first learn about them, then you come to the realization that pining for them is a waste of time. I've had a flash of inspiration recently and have been thinking about Lisp Machines a lot in the past three weeks. But first, a digression. There's an important lesson to be learned about why Symbolics failed. I think Richard Gabriel came to the completely wrong conclusion with "Worse is Better" (http://www.dreamsongs.com/WorseIsBetter.html http://www.dreamsongs.com/WorseIsBetter.html). There are two reasons why: 1. Out of all the LispM-era Lisp hackers, only RMS understood the value of what's now known as Free Software. (If you haven't read it yet, read Steven Levy's Hackers - it describes the MIT/LMI/Symbolics split and how RMS came to start FSF and GNU). 2. Portability is really important. The key lesson to draw from Unix isn't that "Worse is Better," it's that survivable software is Free and portable. Free because getting software to someone's harddrive is 80% of success, and portable because you don't know where people will want to use your software (there are some really weird places). Symbolics was neither. If Genera had been Free Software, it would by definition still be around today. If Genera had been portable, it's likely Symbolics would never have gone out of business (the Alpha virtual machine would have been done sooner, with less resources, and for more systems). Being released as Free Software today wouldn't help. Genera's predecessor, MIT CADR, was made available under an MIT-style license in 2004 (http://www.heeltoe.com/retro/mit/mit_cadr_lmss.html http://www.heeltoe.com/retro/mit/mit_cadr_lmss.html). There's a VM emulator which runs the code. The whole system is pretty useless. Now on to the inspiration part: It's possible to make a very high-performance, portable Lisp operating system on modern hardware. This has been a possibility ever since the Pentium came out. The main bottleneck to conventional Lisp runtime performance is the way operating systems manage memory allocation and virtual memory. A type-safe runtime that has control over memory layout, virtual memory, and is aware of DMA can provide extremely high throughput for allocation and GC (this has been shown by Azure's Linux patches for their JVM), true zero-copy I/O, almost optimal levels of fragmentation, and excellent locality properties. If you go single address space (and there's no reason not to) and move paging into software (object faulting and specialized array access), you've also eliminated TLB misses. Throw in the fact that it now becomes trivial to do exokernel-type stuff like for example caching pre-formatted IP packets, and it should be possible to build network servers that have throughput many times that of anything that kernel/user-space split OSes like Linux or FreeBSD are capable of for dynamic content (ie - not just issuing DMA requests from one device to another). The only problem is device drivers. Lisp doesn't make writing device drivers any more fun, or reduce the number of devices you have to support. What to do? The reason I've been thinking about this is that I came across this: http://www.cliki.net/Zeta-C http://www.cliki.net/Zeta-C I've heard of Zeta-C multiple times before, but for some reason this time I made the connection - "why not use Zeta-C to compile an OS kernel?" I explored the idea further, and it seems to me that it wouldn't be an unreasonable amount of work to take the NetBSD device subsystem and have it running on top of a Lisp runtime with the necessary emulation of those parts of the NetBSD kernel that the drivers depend on. If you don't know, NetBSD's device drivers are modular - they're written on top of bus abstraction layers, which are written on top of other abstraction layers (for example, memory-mapped vs port I/O is abstracted). So the actual system twiddling bits can be neatly encapsulated (which isn't necessarily true for Linux drivers, for example). I'm aware of Movitz (http://common-lisp.net/project/movitz/ http://common-lisp.net/project/movitz/) and LoperOS (http://www.loper-os.org/ http://www.loper-os.org/). Movitz makes the mistake of trying not to be portable, but there's useful things there. I haven't spoken to Slava about this yet so I don't know what's going on with LoperOS. I am also aware of TUNES, and think it was an interesting waste of time. The main thing is to get Zeta-C to work on Common Lisp. Then it's to build a new portable, boot-strappable runtime (I think the Portable Standard Lisp approach of having a SYSLISP layered on top of VOPs is the right way to go for this), and either build a compiler targeting that runtime, or adapt the IR-generating parts of one of SBCL, CMUCL or Clozure. Further bootstrapping can be done with SWANK and X11 once a basic networking stack is in place. I think such a system would be quite fun to hack on. If you've gotten this far, let me know what you think about this idea. I also have some preliminary thoughts about how this can be worked into the base of a new high-performance/scalability transactional database startup, if you want to hear about that email me: vsedach@gmail.com
cageface 16y ago
There is no such thing as the "best" software environment, anymore than there is the best car. It all depends on what you're trying to do and what your priorities and constraints are.
- regularfry 16y ago
  That's the dogma, but is there a reason for it to actually be true? All we know for sure is that there isn't a clear winner yet.
  - gukjoon 16y ago
    Software environments, languages, and other such abstractions aren't there to be perfect. They are there to help human beings manage absurd numbers of machine instructions. Maybe when we fully understand the human mind, we'll figure out the perfect way to map machine instructions to concepts that humans can grok. Until then, we'll have to make somewhat subjective design decisions in the face of uncertain constraints and priorities, which is certainly not going to lead to a "best" solution.
  - cageface 16y ago
    Not dogma, but the conclusion of reason and experience. You don't build a bridge and a house with the same tools. Why should software be different?
- artsrc 16y ago
  If you know what you know your priorities, constraints and "what are trying to do" (goals), then, is there such a thing as a best software environment?
rwmj 16y ago
That page is giving a 403 error now. What am I missing?

51 points by udzinari 16y ago

Wikipedia has more information for people (like me) who haven't heard of Genera before: http://en.wikipedia.org/wiki/Genera_(operating_system) http://en.wikipedia.org/wiki/Genera_(operating_system)

ScottBurson 16y ago

I have a Symbolics XL1200 that still works (at least, it worked the last time I turned it on, which was a few years ago). I'd be happy to demo it at YC sometime. It was a fun environment to work in. Although the hardware was very slow by modern standards, the environment was designed to maximize hacker productivity. For example, you didn't have to run a program under the debugger to debug it -- the debugger was always there. Any program that hit an error would drop you into the debugger. One keystroke would then take you to the source for the current function. You could change that function, recompile it, and restart from the point where the function was called. The environment was completely open; there was no access control of any kind. There was also only a single address space. Did that make it crash-prone? Remarkably, no. The hardware tagging and bounds checking kept programs from stepping on objects they didn't own. Of course such a design would never survive outside of a research lab, but it made it amazingly easy to hack almost anything in the system.

daeken 16y ago
I don't know that I'd agree that it wouldn't survive outside of research. In recent years, there's been a huge push for pure-managed operating systems, where everything runs in one address space and depends on the compiler to ensure security and reliability. While no such systems are in practice (to my knowledge), the potential is incredible. We'll see how it pans out in the next few years. (Full disclosure: I started such an OS, Renraku, so I have a bit of a vested interest in the success of such ideas)
- joe_bleau 16y ago
  What about IBM's OS/400? http://users.snip.net/~gbooker/AS400/ibm.htm http://users.snip.net/~gbooker/AS400/ibm.htm
  - hga 16y ago
    Indeed, OS/400 is just wonderfully advanced under the hood. Get one of the cheap used copies of this book for a lot more: http://www.amazon.com/Inside-AS-400-Frank-Soltis/dp/1882419669/ http://www.amazon.com/Inside-AS-400-Frank-Soltis/dp/18824196...
sedachv 16y ago
Hi Scott, didn't see this comment before I posted mine (take a look at it if you have the time). Thank you for writing Zeta-C and proving the concept of reasonable C->Lisp compilation, and for making the code public domain.
- ScottBurson 16y ago
  You're welcome. I hope it makes for an interesting curiosity -- I'm not sure it can be much more at this point :-)

sedachv 16y ago

Lisp Machines are something that you think is really cool when you first learn about them, then you come to the realization that pining for them is a waste of time. I've had a flash of inspiration recently and have been thinking about Lisp Machines a lot in the past three weeks. But first, a digression. There's an important lesson to be learned about why Symbolics failed. I think Richard Gabriel came to the completely wrong conclusion with "Worse is Better" (http://www.dreamsongs.com/WorseIsBetter.html http://www.dreamsongs.com/WorseIsBetter.html). There are two reasons why: 1. Out of all the LispM-era Lisp hackers, only RMS understood the value of what's now known as Free Software. (If you haven't read it yet, read Steven Levy's Hackers - it describes the MIT/LMI/Symbolics split and how RMS came to start FSF and GNU). 2. Portability is really important. The key lesson to draw from Unix isn't that "Worse is Better," it's that survivable software is Free and portable. Free because getting software to someone's harddrive is 80% of success, and portable because you don't know where people will want to use your software (there are some really weird places). Symbolics was neither. If Genera had been Free Software, it would by definition still be around today. If Genera had been portable, it's likely Symbolics would never have gone out of business (the Alpha virtual machine would have been done sooner, with less resources, and for more systems). Being released as Free Software today wouldn't help. Genera's predecessor, MIT CADR, was made available under an MIT-style license in 2004 (http://www.heeltoe.com/retro/mit/mit_cadr_lmss.html http://www.heeltoe.com/retro/mit/mit_cadr_lmss.html). There's a VM emulator which runs the code. The whole system is pretty useless. Now on to the inspiration part: It's possible to make a very high-performance, portable Lisp operating system on modern hardware. This has been a possibility ever since the Pentium came out. The main bottleneck to conventional Lisp runtime performance is the way operating systems manage memory allocation and virtual memory. A type-safe runtime that has control over memory layout, virtual memory, and is aware of DMA can provide extremely high throughput for allocation and GC (this has been shown by Azure's Linux patches for their JVM), true zero-copy I/O, almost optimal levels of fragmentation, and excellent locality properties. If you go single address space (and there's no reason not to) and move paging into software (object faulting and specialized array access), you've also eliminated TLB misses. Throw in the fact that it now becomes trivial to do exokernel-type stuff like for example caching pre-formatted IP packets, and it should be possible to build network servers that have throughput many times that of anything that kernel/user-space split OSes like Linux or FreeBSD are capable of for dynamic content (ie - not just issuing DMA requests from one device to another). The only problem is device drivers. Lisp doesn't make writing device drivers any more fun, or reduce the number of devices you have to support. What to do? The reason I've been thinking about this is that I came across this: http://www.cliki.net/Zeta-C http://www.cliki.net/Zeta-C I've heard of Zeta-C multiple times before, but for some reason this time I made the connection - "why not use Zeta-C to compile an OS kernel?" I explored the idea further, and it seems to me that it wouldn't be an unreasonable amount of work to take the NetBSD device subsystem and have it running on top of a Lisp runtime with the necessary emulation of those parts of the NetBSD kernel that the drivers depend on. If you don't know, NetBSD's device drivers are modular - they're written on top of bus abstraction layers, which are written on top of other abstraction layers (for example, memory-mapped vs port I/O is abstracted). So the actual system twiddling bits can be neatly encapsulated (which isn't necessarily true for Linux drivers, for example). I'm aware of Movitz (http://common-lisp.net/project/movitz/ http://common-lisp.net/project/movitz/) and LoperOS (http://www.loper-os.org/ http://www.loper-os.org/). Movitz makes the mistake of trying not to be portable, but there's useful things there. I haven't spoken to Slava about this yet so I don't know what's going on with LoperOS. I am also aware of TUNES, and think it was an interesting waste of time. The main thing is to get Zeta-C to work on Common Lisp. Then it's to build a new portable, boot-strappable runtime (I think the Portable Standard Lisp approach of having a SYSLISP layered on top of VOPs is the right way to go for this), and either build a compiler targeting that runtime, or adapt the IR-generating parts of one of SBCL, CMUCL or Clozure. Further bootstrapping can be done with SWANK and X11 once a basic networking stack is in place. I think such a system would be quite fun to hack on. If you've gotten this far, let me know what you think about this idea. I also have some preliminary thoughts about how this can be worked into the base of a new high-performance/scalability transactional database startup, if you want to hear about that email me: vsedach@gmail.com

cageface 16y ago

There is no such thing as the "best" software environment, anymore than there is the best car. It all depends on what you're trying to do and what your priorities and constraints are.

regularfry 16y ago
That's the dogma, but is there a reason for it to actually be true? All we know for sure is that there isn't a clear winner yet.
- gukjoon 16y ago
  Software environments, languages, and other such abstractions aren't there to be perfect. They are there to help human beings manage absurd numbers of machine instructions. Maybe when we fully understand the human mind, we'll figure out the perfect way to map machine instructions to concepts that humans can grok. Until then, we'll have to make somewhat subjective design decisions in the face of uncertain constraints and priorities, which is certainly not going to lead to a "best" solution.
- cageface 16y ago
  Not dogma, but the conclusion of reason and experience. You don't build a bridge and a house with the same tools. Why should software be different?
artsrc 16y ago
If you know what you know your priorities, constraints and "what are trying to do" (goals), then, is there such a thing as a best software environment?

rwmj 16y ago

That page is giving a 403 error now. What am I missing?