4 ms·

I've always been fascinated by Wirthian languages. The radical simplicity, the focus on fast compilation and programmer productivity. One question I haven't re

I've always been fascinated by Wirthian languages. The radical simplicity, the focus on fast compilation and programmer productivity. One question I haven't researched enough is, how did the people at ETH Zürich used Oberon for OS programming. AFAIK, all Wirthian languages have garbage collection, and the common narrative is that you cannot use a GC'ed language in an OS kernel. How did Wirth et al. work around this in the Oberon OS family?

lazyjones 8y ago
> the common narrative is that you cannot use a GC'ed language in an OS kernel. How did Wirth et al. work around this in the Oberon OS family? OS do not consist only of a kernel (nitpicking, sorry). Oberon hides the low-level functionality in special modules where implementations are provided by the compiler (in assembly language if needed), i.e. „builtins“. In other words, what is part of the kernel in e.g. Linux can be implemented as a language feature (system module) in the compiler and the resulting code can run on „bare metal“.
badsectoracula 8y ago
You can check it here, the first ~100 lines are about memory management: https://www.inf.ethz.ch/personal/wirth/ProjectOberon/Sources/Kernel.Mod.txt https://www.inf.ethz.ch/personal/wirth/ProjectOberon/Sources... The exposed procedures (exposed symbols have a star after their name in their declaration) are New (it is treated in a special way by the compiler to remain type safe even if the language itself wouldn't allow it normally), Mark and Scan. It seems that it uses a block allocator that the garbage collector uses itself privately. These procedures use special "GET"and "PUT" procedures which are translated to raw memory access by the compiler (these are also used to access the hardware which is memory mapped). A test for garbage collection is called periodically as seen below (check PROCEDURE GC): https://www.inf.ethz.ch/personal/wirth/ProjectOberon/Sources/Oberon.Mod.txt https://www.inf.ethz.ch/personal/wirth/ProjectOberon/Sources... The collection occurs either when the system is in low memory or some countdown is reached (set by a call to Collect, although looking at the code it seems other bits can change it too). The rest of the OS just uses the above facilities.
pankajdoharey 8y ago
Actually before Apple implemented ARC in its OS Around Mac OS X Lion and iOS 5, the Objective-C runtime in Mac OS X was a garbage collection runtime.
- cturner 8y ago
  Wikipedia describes GC as being automatic memory management. I don't think it is valid to describe the traditional (pre-2007) osx/objective-c model of manual reference counting as a form of garbage collection. Darwin itself is mostly unix+C. Hence, this does not respond to the parent, asking about GC in the OS itself.
  - Volt 8y ago
    >Darwin itself is mostly unix+C. Hence, this does not respond to the parent, asking about GC in the OS itself. You're right, but just to clarify WRT your first paragraph, Obj-C did briefly have a GC'd runtime between the eras of manual and automatic ref counting.
    - cturner 8y ago
      As you and neighbour inferred, I had not know of this. Thanks.
  - pjmlp 8y ago
    Objective-C 2.0 got a tracing GC, which was later replaced by ARC, because it was a failure. Although it was possible to link frameworks compiled in mixed modes, it lead most of the time to crashs and the C semantics did not help. Hence the pragmatic solution of ARC, which basically has the compiler inserting the retain/release calls that used to be done manually. While not the same, it was a good enough solution. https://developer.apple.com/documentation/foundation/object_runtime/objective_c_garbage_collection?language=objc https://developer.apple.com/documentation/foundation/object_...
- cellularmitosis 8y ago
  That was true for Mac OS but not for iOS. You had to manage memory manually with alloc and release.
stewbrew 8y ago
Oberon (originally) wasn't just a language but also a single-threaded environment with cooperative multitasking (somewhat similar to Smalltalk but with a fast & efficient compiler).
pjmlp 8y ago
That common narrative is urban myths from modern OS Luddites. There are many examples of system programming languages, used to implement OSes with GC at kernel level. - Interlisp-D/Lisp Machines - Smalltalk - Mesa/Cedar - Modula-2+ - Modula-3 - Oberon, Oberon-2, Active Oberon, Component Pascal, Oberon-07 - Sing# and System C# - D The only problem is having a company willing to put enough money so that the naysayers don't have other option other than adopting the platform. However given the short sight of business regarding roadmaps, there isn't a company willing to go full speed on it. So we get the half-way solution, with Swift, Android, UWP, ChromeOS, .NET. The kernels might not have a GC, but almost everything on userspace is done in GC enabled languages.
- coldtea 8y ago
  >There are many examples of system programming languages, used to implement OSes Yeah, but are there any examples of "system programming languages, used to implement OSes with GC at kernel level" where said OSes are fast, practical, and used by way more people and businesses than 100 researchers and hard core fans?
  - pjmlp 8y ago
    Yes, Oberon was one of them, used by the complete IT department at ETHZ, including normal employees. Were they more than 100, I don't know, never counted them. Or IoT deployments from Oberon Systems and Astrobe, with Astrobe still focused on selling Oberon compilers for embedded development. As mentioned, it is always a matter of money and business willingness. Joe Duffy mentions on his Rustconf talk how his team had to deal with lots of Luddites, even though they had Midori running in front of them. If the likes of Apple, Google and Microsoft were willing to spend the same money as they did/are making JavaScript JIT compiliers fast, we would already get something in mainstream. Instead they focus on the easy path, closing down the kernel access, and making the userspace available to GC only languages.
    - srean 8y ago
      Given your knowledge on the subject could comment on how Modula-2 compares with Go. Modula-2 had coroutines, did it have a notion of a channel ? I have feeling that Go is a Modula-2/Pascal with a garbage collector and may be with a better concurrency story. Would that be a fair assessment ?
todd8 8y ago
Wirth’s early languages, Euler, Algol-W, Pascal, and Modula did not have garbage collection—memory management was up to the programmer. Later languages Modula-3, Oberon, and Component Pascal do have garbage collection.
- srean 8y ago
  Wirth had nothing to do with Modula-3 though. [Prior to edit:] Modula-3 had nothing to do with Modula-3 though.
  - newnewpdro 8y ago
    > Modula-3 had nothing to do with Modula-3 though. What?
    - srean 8y ago
      I snorted my drink out after reading it myself. Thank you for catching that.
  - pjmlp 8y ago
    Yes he had, although it is not a direct creation from him. Modula-3 is the evolution of Mesa/Cedar, after the researchers went to DEC and created Modula-2+, an improved version of Mesa/Cedar. They collaborated with Wirth on their work. "The design of the programming language Modula-3 was a joint effort by the Digital Systems Research Center and the Olivetti Research Center, undertaken with the guidance and inspiration of Niklaus Wirth." http://lucacardelli.name/papers/modula3typesystem.a4.pdf http://lucacardelli.name/papers/modula3typesystem.a4.pdf
nickpsecurity 8y ago
What they do is make things safe by default with a keyword to turn it off. Ada and PL/S (IBM's) do this, too. They also have other keywords to tell the compiler how to handle some statements that might be undefined in C. For unsafe, the module will either use the same language without safety or GC, or assembly language, or some combo of that. Developers using such features often have type checks and input validation on their arguments since their usage is dangerous. So, the majority of the code is using safe primitives with GC while specific parts that can't are behind protected interfaces. The next thing you can do is use lightweight formal methods, static analysis, etc on the unsafe code. So, those primitives are verified in unsafe HLL or in assembly. You know they'll probably act right given preconditions are met. Then, the rest of code is protected by type- and memory-safety. Combining best tool for each job makes your whole system safe despite unsafe code. One person I know had a successful test of using SPARK Ada for fast, otherwise-unsafe components of Go applications. My Brute-Force Assurance idea would likely, given most use C language underneath, do these in Frama-C compiled with CompCert or C0 after I threw every static/symbolic analyzer, concurrency checker, and fuzzer at it. Very unlikely to have trouble out of anything that passes all that if the spec or mental model itself is correct. Rest is safe, memory-managed code. It also helps to keep same data types, calling conventions, and so on to prevent abstraction-gap attacks. There's ongoing work on abstraction-safe, verified integration of components in different languages, though. Also, note that you can use different memory managers in different apps or even parts of an app. The runtime has to be designed to do that. JX OS in Java is one that is. There are GC's optimized for throughput, average latency, hard real-time, size, and so on. A standard allocator gives the code a block to use, they assign to or divide it up for the other GC's, and they work from there. There might also be some work to do on scheduling how they run, too. I can't remember how easy or difficult that is on a common platform. It was mostly done on custom OS's.
vidarh 8y ago
The main issue with GC for system level use is avoiding GC pauses in latency critical parts of the code. You can achieve that a large number of ways, but one of the simplest is to simply disable the GC in latency critical sections (such as interrupts) and ensure you set aside enough memory to prevent needing a collection to let the critical sections complete. If you need dynamic allocation in such sections, you can still do so as long as you set safety limits to ensure sufficient memory will be available, or e.g. let the latency critical code allocate from separate parts of the heap or using a different allocation scheme (e.g. pre-allocated arenas). There are lots of details to get right, but the "common narrative" basically assumes you're not able to disable the collector when you want to and no way of safely pre-empting the collector. But making it possible to selectively disable a GC and ensure your latency critical code can safely do so is fairly straightforward.
- eterps 8y ago
  Do you know examples cases of such critical parts in system level code where a GC pause would be problematic?
  - vidarh 8y ago
    Pretty much any interrupt code, and lots of hardware drivers.

lazyjones 8y ago

> the common narrative is that you cannot use a GC'ed language in an OS kernel. How did Wirth et al. work around this in the Oberon OS family? OS do not consist only of a kernel (nitpicking, sorry). Oberon hides the low-level functionality in special modules where implementations are provided by the compiler (in assembly language if needed), i.e. „builtins“. In other words, what is part of the kernel in e.g. Linux can be implemented as a language feature (system module) in the compiler and the resulting code can run on „bare metal“.

badsectoracula 8y ago

You can check it here, the first ~100 lines are about memory management: https://www.inf.ethz.ch/personal/wirth/ProjectOberon/Sources/Kernel.Mod.txt https://www.inf.ethz.ch/personal/wirth/ProjectOberon/Sources... The exposed procedures (exposed symbols have a star after their name in their declaration) are New (it is treated in a special way by the compiler to remain type safe even if the language itself wouldn't allow it normally), Mark and Scan. It seems that it uses a block allocator that the garbage collector uses itself privately. These procedures use special "GET"and "PUT" procedures which are translated to raw memory access by the compiler (these are also used to access the hardware which is memory mapped). A test for garbage collection is called periodically as seen below (check PROCEDURE GC): https://www.inf.ethz.ch/personal/wirth/ProjectOberon/Sources/Oberon.Mod.txt https://www.inf.ethz.ch/personal/wirth/ProjectOberon/Sources... The collection occurs either when the system is in low memory or some countdown is reached (set by a call to Collect, although looking at the code it seems other bits can change it too). The rest of the OS just uses the above facilities.

pankajdoharey 8y ago

Actually before Apple implemented ARC in its OS Around Mac OS X Lion and iOS 5, the Objective-C runtime in Mac OS X was a garbage collection runtime.

cturner 8y ago
Wikipedia describes GC as being automatic memory management. I don't think it is valid to describe the traditional (pre-2007) osx/objective-c model of manual reference counting as a form of garbage collection. Darwin itself is mostly unix+C. Hence, this does not respond to the parent, asking about GC in the OS itself.
- Volt 8y ago
  >Darwin itself is mostly unix+C. Hence, this does not respond to the parent, asking about GC in the OS itself. You're right, but just to clarify WRT your first paragraph, Obj-C did briefly have a GC'd runtime between the eras of manual and automatic ref counting.
  - cturner 8y ago
    As you and neighbour inferred, I had not know of this. Thanks.
- pjmlp 8y ago
  Objective-C 2.0 got a tracing GC, which was later replaced by ARC, because it was a failure. Although it was possible to link frameworks compiled in mixed modes, it lead most of the time to crashs and the C semantics did not help. Hence the pragmatic solution of ARC, which basically has the compiler inserting the retain/release calls that used to be done manually. While not the same, it was a good enough solution. https://developer.apple.com/documentation/foundation/object_runtime/objective_c_garbage_collection?language=objc https://developer.apple.com/documentation/foundation/object_...
cellularmitosis 8y ago
That was true for Mac OS but not for iOS. You had to manage memory manually with alloc and release.

stewbrew 8y ago

Oberon (originally) wasn't just a language but also a single-threaded environment with cooperative multitasking (somewhat similar to Smalltalk but with a fast & efficient compiler).

pjmlp 8y ago

That common narrative is urban myths from modern OS Luddites. There are many examples of system programming languages, used to implement OSes with GC at kernel level. - Interlisp-D/Lisp Machines - Smalltalk - Mesa/Cedar - Modula-2+ - Modula-3 - Oberon, Oberon-2, Active Oberon, Component Pascal, Oberon-07 - Sing# and System C# - D The only problem is having a company willing to put enough money so that the naysayers don't have other option other than adopting the platform. However given the short sight of business regarding roadmaps, there isn't a company willing to go full speed on it. So we get the half-way solution, with Swift, Android, UWP, ChromeOS, .NET. The kernels might not have a GC, but almost everything on userspace is done in GC enabled languages.

coldtea 8y ago
>There are many examples of system programming languages, used to implement OSes Yeah, but are there any examples of "system programming languages, used to implement OSes with GC at kernel level" where said OSes are fast, practical, and used by way more people and businesses than 100 researchers and hard core fans?
- pjmlp 8y ago
  Yes, Oberon was one of them, used by the complete IT department at ETHZ, including normal employees. Were they more than 100, I don't know, never counted them. Or IoT deployments from Oberon Systems and Astrobe, with Astrobe still focused on selling Oberon compilers for embedded development. As mentioned, it is always a matter of money and business willingness. Joe Duffy mentions on his Rustconf talk how his team had to deal with lots of Luddites, even though they had Midori running in front of them. If the likes of Apple, Google and Microsoft were willing to spend the same money as they did/are making JavaScript JIT compiliers fast, we would already get something in mainstream. Instead they focus on the easy path, closing down the kernel access, and making the userspace available to GC only languages.
  - srean 8y ago
    Given your knowledge on the subject could comment on how Modula-2 compares with Go. Modula-2 had coroutines, did it have a notion of a channel ? I have feeling that Go is a Modula-2/Pascal with a garbage collector and may be with a better concurrency story. Would that be a fair assessment ?

todd8 8y ago

Wirth’s early languages, Euler, Algol-W, Pascal, and Modula did not have garbage collection—memory management was up to the programmer. Later languages Modula-3, Oberon, and Component Pascal do have garbage collection.

srean 8y ago
Wirth had nothing to do with Modula-3 though. [Prior to edit:] Modula-3 had nothing to do with Modula-3 though.
- newnewpdro 8y ago
  > Modula-3 had nothing to do with Modula-3 though. What?
  - srean 8y ago
    I snorted my drink out after reading it myself. Thank you for catching that.
- pjmlp 8y ago
  Yes he had, although it is not a direct creation from him. Modula-3 is the evolution of Mesa/Cedar, after the researchers went to DEC and created Modula-2+, an improved version of Mesa/Cedar. They collaborated with Wirth on their work. "The design of the programming language Modula-3 was a joint effort by the Digital Systems Research Center and the Olivetti Research Center, undertaken with the guidance and inspiration of Niklaus Wirth." http://lucacardelli.name/papers/modula3typesystem.a4.pdf http://lucacardelli.name/papers/modula3typesystem.a4.pdf

nickpsecurity 8y ago

What they do is make things safe by default with a keyword to turn it off. Ada and PL/S (IBM's) do this, too. They also have other keywords to tell the compiler how to handle some statements that might be undefined in C. For unsafe, the module will either use the same language without safety or GC, or assembly language, or some combo of that. Developers using such features often have type checks and input validation on their arguments since their usage is dangerous. So, the majority of the code is using safe primitives with GC while specific parts that can't are behind protected interfaces. The next thing you can do is use lightweight formal methods, static analysis, etc on the unsafe code. So, those primitives are verified in unsafe HLL or in assembly. You know they'll probably act right given preconditions are met. Then, the rest of code is protected by type- and memory-safety. Combining best tool for each job makes your whole system safe despite unsafe code. One person I know had a successful test of using SPARK Ada for fast, otherwise-unsafe components of Go applications. My Brute-Force Assurance idea would likely, given most use C language underneath, do these in Frama-C compiled with CompCert or C0 after I threw every static/symbolic analyzer, concurrency checker, and fuzzer at it. Very unlikely to have trouble out of anything that passes all that if the spec or mental model itself is correct. Rest is safe, memory-managed code. It also helps to keep same data types, calling conventions, and so on to prevent abstraction-gap attacks. There's ongoing work on abstraction-safe, verified integration of components in different languages, though. Also, note that you can use different memory managers in different apps or even parts of an app. The runtime has to be designed to do that. JX OS in Java is one that is. There are GC's optimized for throughput, average latency, hard real-time, size, and so on. A standard allocator gives the code a block to use, they assign to or divide it up for the other GC's, and they work from there. There might also be some work to do on scheduling how they run, too. I can't remember how easy or difficult that is on a common platform. It was mostly done on custom OS's.

vidarh 8y ago

The main issue with GC for system level use is avoiding GC pauses in latency critical parts of the code. You can achieve that a large number of ways, but one of the simplest is to simply disable the GC in latency critical sections (such as interrupts) and ensure you set aside enough memory to prevent needing a collection to let the critical sections complete. If you need dynamic allocation in such sections, you can still do so as long as you set safety limits to ensure sufficient memory will be available, or e.g. let the latency critical code allocate from separate parts of the heap or using a different allocation scheme (e.g. pre-allocated arenas). There are lots of details to get right, but the "common narrative" basically assumes you're not able to disable the collector when you want to and no way of safely pre-empting the collector. But making it possible to selectively disable a GC and ensure your latency critical code can safely do so is fairly straightforward.

eterps 8y ago
Do you know examples cases of such critical parts in system level code where a GC pause would be problematic?
- vidarh 8y ago
  Pretty much any interrupt code, and lots of hardware drivers.