10 ms·

What Is Systems Programming, Really?

248 points by wcrichton 8y ago

mrec 8y ago
> Andrei Alexandrescu (creator of D) Andrei didn't create D; he's been influential in it, especially w.r.t. D's metaprogramming story, but the language was created and is primarily maintained by Walter Bright.
- wcrichton 8y ago
  Ah, my mistake. Clarified the wording, thanks.
  - bausshf 8y ago
    It was no mistake, but it is now.
- mhh__ 8y ago
  IIRC He is a - listed - co-creator of D2 (There are two versions of D, the first one being entirely Walter?)
- bausshf 8y ago
  D1 was created by Walter Bright, but D2 was primarily created by both Walter Bright and Andrei Alexandrescu.
  - mrec 8y ago
    Is D2 so different as to be considered a separate language? (I haven't really followed it for years now.)
    - bausshf 8y ago
      Yes it's very different. D2 introduced a lot of new keywords, new semantics, not two "standard" libraries like with D1 where there were Tango. Tango is pretty much completely obsolete in D2. Most D1 code might work to an extend with the D2 compiler, BUT almost no D2 code will be able to run using a D1 compiler and it'll even be difficult to translate it directly. D2 is basically when Andrei joined the development of D back in 2007.
  - bausshf 8y ago
    Why am I getting downvoted for this? Lmao
    - bausshf 8y ago
      Is HN turning into Reddit?
thegeomaster 8y ago
Great article. This is an important distinction to make. I've noticed that the more experienced I got, I felt less and less need to micro-optimize every aspect of a program at the expense of code readability. That's where I think the design and implementation of a system intersect; making it easy to understand how it works and giving future maintainers the confidence to make changes is best accomplished with clear and readable code with the right abstractions (even if it hurts performance a little). I think that documentation will always be second to that.
- codr4 8y ago
  Depends on the kind of software, though. If you're writing an interpreter/compiler, web server, database or operating system you don't have much choice.
chubot 8y ago
Hm I really like the observation that "systems programming" is overloaded. I gave my take here: https://lobste.rs/s/jrzwgy/what_is_systems_programming_really#c_ubvdwf https://lobste.rs/s/jrzwgy/what_is_systems_programming_reall... In short, I think shell is actually a more appropriate language for describing systems, rather than OCaml or Haskell! If that sounds weird, then I'll point you to the evidence of shell already being used for this purpose (e.g. the 40K lines of shell in the Kubernetes repo I mentioned). Although, on reading the blog post a little more closely, we may not be agreeing precisely on the problem. I should probably write my own blog post about this ... there are a couple I linked that give something a flavor.
- v_lisivka 8y ago
  It's simple: Bugs in application code causes errors in that application. Bugs in system code causes errors in an user application. If bugs in your code causes problems only to your code - it's single application. If bugs in your code causes problems in other programs - it's system of programs.
dleavitt 8y ago
The author's point that functional programming principles are important to systems programming and should be taught alongside it is well taken. I think he confuses the issue somewhat with all the discussion of what is a "systems programming language" as these days it's just used to refer to languages substitutable for C.
antirez 8y ago
My idea of system programming is that, other than the "near to the bare metal" element, which may not be always true, has this quality of creating infrastructures for other layers to use. A game 3D engine and a DNS server may be both written in C++ and may use the same low level programming techniques to achieve speed, but the fundamental difference is that one is just part of an application program of some type, and the other a system that provides a general service to other programs. So clearly it's technically possible to do system programming in higher level languages, even if often this will require anyway dealing with many system calls regardless of the language, both explicitly or implicitly via abstractions.
- zaarn 8y ago
  I think systems programming should be split in two, honestly. You have the kernel-level systems programming where you need to be near the bare metal, where you write kernels or program microcontrollers with less RAM than a x86 CPU has L1 cache, where taking a microsecond longer can mean the overall system crashing (or even costing a human life). And you have system-level systems programming where you write services and supportive infrastructure to enable user programs, where garbage collection and non-realtime behaviour is acceptable, where you can skip a microsecond and it'll be alright.
  - _dps 8y ago
    I think the two "poles" you've identified are good ones (~microseconds matter and ~100ms matters) but there's a wide range in between. If you do anything with video at modern resolutions, then you're in the single-millisecond-matters regime (this includes games as mentioned upthread, and just about anything in VR as well).
    - zaarn 8y ago
      Video streaming and gaming have some soft realtime guarantees, yeah, though generally these aren't systems programming and application programming instead. The plumbing below, ie graphics rendering pipeline, tend to fall into the category of kernel systems programming, controlling the GPU is pretty close to doing what a kernel does IMO.
toolslive 8y ago
> You should be able to forge a number into a pointer, since that’s how hardware works. It's a C-centric view of the world and I wonder what old school FORTRAN77 people or lispers would think of this. Anyway, I think the right to do this should be a privilege of the compiler. As soon as you claim this right, you abandon all possible support she can give you in battling all kinds of silly mistakes.
- alxlaz 8y ago
  I'm not an "old-school lisper", but I have done some Lisp development related to embedded systems. The phrasing may be somewhat unfortunate but it ultimately does describe how hardware works: you need to write something at a numerical address that you find in the reference guide or the datasheet. At the end of the line, you do have to forge a number into a pointer. In James Micken's words, "You can’t just place a LISP book on top of an x86 chip and hope that the hardware learns about lambda calculus by osmosis." "Higher-level" languages (like Lisp) can help you in other regards. They can include proper primitives (or give you the proper tools to write the proper primitives yourself) to ensure that direct-to-hardware access is safe, can give you better tools to manipulate DMA buffers (e.g. proper support for coroutines when manipulating double-buffered data is cool), can help you write more generic state-applying and state-reading code (i.e. code for "turn these bytes from this circular buffer into this structure" or "take this structure that represents a peripheral's current config and use it to write the proper magic numbers in the right registers"). They can offer you better semantics for translating human-readable(-ish) configuration into the right stream of bytes, or manipulate peripherals in real-time. They can help you write a better VM (or their runtime can outright be the better VM you need). But at the end of the day, writing to the config space of a memory-mapped device is still going to consist of taking a number (or building it from several numbers) and writing bytes at that address, no matter what language you're using. Better runtime support for safe operation under this scenario, better semantics -- they're all important, but what the author expresses is not C-centric in any way.
- Athas 8y ago
  Pointers are not even necessary for writing operating systems or memory allocators. The Oberon system writes those low-level components using intrinsic peek/poke functions (like some ancient BASIC!) that are recognised specially by the compiler and turned into direct memory modifications. This is clearly just as unsafe as pointers (probably more so), but it means you don't generally pollute the language to support some very specialised code. The rest of Oberon is memory-safe and garbage-collected (with the garbage collector also written in Oberon).
jillesvangurp 8y ago
System programming is definitely a bit overloaded these days though I believe it is now widely understood to be languages like C/C++ that can be used for OS, kernel, and embedded development. I would agree though that the language ecosystem is shifting a lot in the last few years. IMHO there are now a few languages that are becoming proper full stack languages in the sense that they scale from embedded all the way to browser development. Rust is a great example. I'm really impressed with what the Rust community has done in just a few short years. All my life, C/C++ was the only game in town for the stuff that is now done with Rust with arguably the same or in some cases slightly better performance. Stuff like WASM allows the use of system programming languages in places where they would not be used in the past. Using Rust transpiled to WASM makes it a proper fullstack language. And people are using it for that. And people do server development as well in it as well as OS development.
- guelo 8y ago
  Rust is nice but I wouldn't call it full stack because I don't think you should choose it in situations where you can afford a garbage collector. In those cases there are much more ergonomic options available.
  - jillesvangurp 8y ago
    Well, I'm seeing people doing web apps in Rust and doing operating systems as well. Certainly the Rust community has been all over wasm and rust tooling for that is relatively mature compared to other languages. Precisely because it doesn't need a garbage collector and they don't have to wait for standard solutions for that to ship in the next year or so. But I agree, languages like Swift and Kotlin are a bit more logical choice as fullstack languages. Both of those are also extending their reach. Particulary, Kotlin code is a very nice upgrade from javascript in terms of expressiveness, tooling, and safety. I'm using it on top of the JVM currently. People are doing Android with it and the native compiler currently under development is explicitly targeted at doing native IOS and Android apps (i.e. not the current java based DEX vm common on Android).
- saagarjha 8y ago
RickJWagner 8y ago
Nice article. My view of Systems Programming will forever be colored by my start with IBM mainframes. The Systems Programmer maintained all the utilities that the mainframe provided, and also the frameworks that supported online programming. I suppose today's equivalent is a combination of a sysadmin and an applications developer, providing the applications are things like container components, build pipelines, etc.
- vram11 8y ago
  They call them devops these days.
tempodox 8y ago
I think the author makes fair points. My conclusion: Do the bit twiddling and strongly constrained parts in a low level language and the rest in a language that supports strong static typing and good interfaces.
boulos 8y ago
You’re HN famous, Will! But I disagree with this: > Companies like Dropbox were able to build surprisingly large and scalable systems on just Python. Many of Dropbox’s services are written in Go, and Magic Pocket is written (partly, at least) in Rust. Earlier in their development, Dropbox relied on S3, which is obviously not in Python. Fundamentally, I think the important part of the “production” aspect of “systems programming” is that it’ll be used a lot. That’s what drives the requirement for efficiency: if you’re 10-100x less efficient, someone else will be more cost effective / solve bigger problems. As an additional example, GitHub was written in Ruby, but that’s fine because the underlying Git and filesystem manipulation is all in C. The same thing is sort of true of Facebook’s world of PHP: PHP is mostly a wrapper around C libraries. Until it got complex enough that they rewrote the language. tl;dr: I don’t think systems programming needs to be “low level”, but production systems do need to be efficient. More powerful computing just moves that further to the right!
burfog 8y ago
A systems programming language can run on bare hardware by itself, or nearly so. It is acceptable to require a very small amount of assembly code, for example to implement something like memcpy or bcopy, or to provide atomic operations. A language is disqualified if it requires an OS or if it requires code written in a different non-assembly language. Cheating, by adding that as a huge (impractical) amount of assembly, also disqualifies a language. Funny story about gcc: The compiler demands a memcpy, which it sometimes uses for struct assignment. If you implement memcpy in the normal way, gcc will helpfully replace your implementation with a call to memcpy! Despite this, C is still a systems programming language.
- tralarpa 8y ago
  > It is acceptable to require a very small amount of assembly code, for example to implement something like memcpy or bcopy, or to provide atomic operations. There is some stuff missing in your list: preparing the stack pointer, address layout, etc. You also need tools to produce text and data sections that can be loaded at a specific address. Even if C is a low level language, there is still quite some stuff between it and the "bare hardware". In that sense, the only language that gives you enough control to produce the binary exactly in the form needed by the hardware (without relying on external tools or libraries) is assembly.
  - sigjuice 8y ago
    stack pointer is an implementation detail. The standard does not mention the word stack even once.
  - saagarjha 8y ago
    > You also need tools to produce text and data sections that can be loaded at a specific address. Though, of course, most compilers provide extensions that make this task generally possible within C (using the loose definition of "I don't need any separate files or inline assembly, just attributes and flags).
    - raggi 8y ago
gjvc 8y ago
... this deserves a supplementary question: what will get us beyond UNIX to the next platform, not just the next paystub?
- pjmlp 8y ago
  Maybe something like Android, OS X/iOS, ChromeOS, UWP, Fuchsia, Unikernels, Language runtimes on Hypervisors. And since I sense the question coming, Android, OS X/iOS, ChromeOS might use an UNIXy kernel, but its exposure surface to applications is so small, that it can be easily exchange for something else, it is just a matter of cost. Which brings us to the next point, given the commodity of free UNIX-like OSes, maybe only an hardware reboot like Quantum computers will actually do it.
  - saagarjha 8y ago
    > And since I sense the question coming, Android, OS X/iOS, ChromeOS might use an UNIXy kernel, but its exposure surface to applications is so small, that it can be easily exchange for something else, it is just a matter of cost. Android, macOS, and iOS expose almost the full set of POSIX to the programmer. Now, you may argue that it goes mostly unused, but it is there, and there are people using it.
    - pjmlp 8y ago
      Android does not, use at your own peril as it is not part of the official APIs and will get your app terminated if you use unauthorized APIs. Using libc is not the same as POSIX. https://developer.android.com/ndk/guides/stable_apis https://developer.android.com/ndk/guides/stable_apis https://developer.android.com/about/versions/nougat/android-7.0-changes#ndk https://developer.android.com/about/versions/nougat/android-... https://android-developers.googleblog.com/2016/06/android-changes-for-ndk-developers.html https://android-developers.googleblog.com/2016/06/android-ch... As for macOS, and iOS, if Apple removed POSIX how many apps written in Objective-C and Swift (not UNIX cli tools ported from Linux) would actually be affected? Very few, because those written in C and C++ ported from other platforms are most likely making use of ANSI C and C++ standards.
tanilama 8y ago
Nowadays it broadly means using C++ programming, involving dealing with different levels of system calls.
n_t 8y ago
Irrespective of language and framework, if the code needs to be aware of underlying hardware then it is systems programming. ex. kernel code, user-space drivers, databases, portions of cloud or server code which need to be hardware aware, etc. What golang targets, I think, should be more of middleware or service-level programming, whether its containers or servers.
- steveklabnik 8y ago
  Alan Perlis has a thoughtful way of expressing this: “a programming language is low level when its programs require attention to the irrelevant.” At first I thought it was poking fun; now I see it as a comment about what is relevant at a given level of abstraction.
  - parley 8y ago
    I like that line of thinking. When discussing the suitability of different programming languages I always point to the problem at hand and identifying the abstraction level the problem is at. What your problem requires you to care about and pay attention to gives this. Ideally, solving your problem would be a one-liner in an already existing DSL created for your specific problem (domain). In reality, this rarely happens and you must have your pick beteeen everything from niche DSL:s to assembly, but the key really is identifying that problem abstraction level. I work in a mainly C/JS shop whereas I privately prefer Rust/ReasonML. Rust would be a great fit at work, but Go would also work nicely for tons of things we do. Alas, the inertia of organizations.
    - ModernMech 8y ago
      > Ideally, solving your problem would be a one-liner in an already existing DSL created for your specific problem I like thinking about the practice of programming as creating a DSL in a language to solve your specific problem.
  - deleted 8y ago
    [deleted]
jkingsbery 8y ago
I'm curious where Erlang fits into this story. It was built to be a "systems" language but algorithms are tersely implemented (I forget exactly, but I remember reading about a study that an Erlang program is much shorter than the comparable C program), and has gradual typing of a sort. It also is garbage collected (I don't think you could "write your own memory allocator in it", as Andrei Alexandrescu described).
danmg 8y ago
640x480, 16 colours, Ring 0.
Analoganalog 8y ago
Hacker News is a site where someone corrects a slight error in a parent comment which then leads us down a bottomless rabbit hole that gets further and further away from the general point of the original post. Hacker News is a honeypot for people on the autism/aspergers spectrum. What is the color of chartreuse?
MichaelMoser123 8y ago
I don't think a single definition makes any sense now: decades ago you may have had a definition of a 'computer system' that describes every big program. Nowadays there are many sets of requirements depending on context, each set of requirements implies its own constraints on memory management and scheduling so that there is no one description of how a complex program would look like.
matachuan 8y ago
IDK about systems programming, but one thing I do know: Systems researchers = ^[AI researchers]
dfox 8y ago
I think that it is not useful to build on Ousterhout's definition as his definition of systems programming language is mostly meant as contrast to Tcl with it's everything is a string memory model (in that view one could very well conclude that Python is a systems programming language).
kmooney 8y ago
If you are making system calls–without too much abstraction–you are doing systems programming.
qubax 8y ago
In one word - syscalls. What do I win?
theoldgit 8y ago
when I was programming in the 70s and 80s (in the UK), the term systems programmer applied to those people involved in the configuration, tuning, management of software such as CICS, MVS, and their equivalents on other hardware bases. I worked at Sperry Univac for several years and the "systems programmers" were the engineers who tweaked the OS, the TP schedulers, built the product installation scripts, designed the configs that controlled the software. It was a specific tradecraft suited to those who were more technically focused, remember that COBOL was the prevalent language for nearly all programming roles in the 70s and that there was no "web" per se. The term started to fade away roughly when PCs started entering into mainstream computing
sytelus 8y ago
In old days, computers that came from manufacturers like IBM, DCE was refrred to as system. The operating software for the system was referred to as “system software”. People who created that software were called “systems programmer”. Much of it invariably involved writing direct to metal/assembly code which is why systems programming has been associated with low level programming. People using this term to refer to task of doing some complex application architecture are doing it wrong.

248 points by wcrichton 8y ago

mrec 8y ago

> Andrei Alexandrescu (creator of D) Andrei didn't create D; he's been influential in it, especially w.r.t. D's metaprogramming story, but the language was created and is primarily maintained by Walter Bright.

wcrichton 8y ago
Ah, my mistake. Clarified the wording, thanks.
- bausshf 8y ago
  It was no mistake, but it is now.
mhh__ 8y ago
IIRC He is a - listed - co-creator of D2 (There are two versions of D, the first one being entirely Walter?)
bausshf 8y ago
D1 was created by Walter Bright, but D2 was primarily created by both Walter Bright and Andrei Alexandrescu.
- mrec 8y ago
  Is D2 so different as to be considered a separate language? (I haven't really followed it for years now.)
  - bausshf 8y ago
    Yes it's very different. D2 introduced a lot of new keywords, new semantics, not two "standard" libraries like with D1 where there were Tango. Tango is pretty much completely obsolete in D2. Most D1 code might work to an extend with the D2 compiler, BUT almost no D2 code will be able to run using a D1 compiler and it'll even be difficult to translate it directly. D2 is basically when Andrei joined the development of D back in 2007.
- bausshf 8y ago
  Why am I getting downvoted for this? Lmao
  - bausshf 8y ago
    Is HN turning into Reddit?

thegeomaster 8y ago

Great article. This is an important distinction to make. I've noticed that the more experienced I got, I felt less and less need to micro-optimize every aspect of a program at the expense of code readability. That's where I think the design and implementation of a system intersect; making it easy to understand how it works and giving future maintainers the confidence to make changes is best accomplished with clear and readable code with the right abstractions (even if it hurts performance a little). I think that documentation will always be second to that.

codr4 8y ago
Depends on the kind of software, though. If you're writing an interpreter/compiler, web server, database or operating system you don't have much choice.

chubot 8y ago

Hm I really like the observation that "systems programming" is overloaded. I gave my take here: https://lobste.rs/s/jrzwgy/what_is_systems_programming_really#c_ubvdwf https://lobste.rs/s/jrzwgy/what_is_systems_programming_reall... In short, I think shell is actually a more appropriate language for describing systems, rather than OCaml or Haskell! If that sounds weird, then I'll point you to the evidence of shell already being used for this purpose (e.g. the 40K lines of shell in the Kubernetes repo I mentioned). Although, on reading the blog post a little more closely, we may not be agreeing precisely on the problem. I should probably write my own blog post about this ... there are a couple I linked that give something a flavor.

v_lisivka 8y ago
It's simple: Bugs in application code causes errors in that application. Bugs in system code causes errors in an user application. If bugs in your code causes problems only to your code - it's single application. If bugs in your code causes problems in other programs - it's system of programs.

dleavitt 8y ago

The author's point that functional programming principles are important to systems programming and should be taught alongside it is well taken. I think he confuses the issue somewhat with all the discussion of what is a "systems programming language" as these days it's just used to refer to languages substitutable for C.

antirez 8y ago

My idea of system programming is that, other than the "near to the bare metal" element, which may not be always true, has this quality of creating infrastructures for other layers to use. A game 3D engine and a DNS server may be both written in C++ and may use the same low level programming techniques to achieve speed, but the fundamental difference is that one is just part of an application program of some type, and the other a system that provides a general service to other programs. So clearly it's technically possible to do system programming in higher level languages, even if often this will require anyway dealing with many system calls regardless of the language, both explicitly or implicitly via abstractions.

zaarn 8y ago
I think systems programming should be split in two, honestly. You have the kernel-level systems programming where you need to be near the bare metal, where you write kernels or program microcontrollers with less RAM than a x86 CPU has L1 cache, where taking a microsecond longer can mean the overall system crashing (or even costing a human life). And you have system-level systems programming where you write services and supportive infrastructure to enable user programs, where garbage collection and non-realtime behaviour is acceptable, where you can skip a microsecond and it'll be alright.
- _dps 8y ago
  I think the two "poles" you've identified are good ones (~microseconds matter and ~100ms matters) but there's a wide range in between. If you do anything with video at modern resolutions, then you're in the single-millisecond-matters regime (this includes games as mentioned upthread, and just about anything in VR as well).
  - zaarn 8y ago
    Video streaming and gaming have some soft realtime guarantees, yeah, though generally these aren't systems programming and application programming instead. The plumbing below, ie graphics rendering pipeline, tend to fall into the category of kernel systems programming, controlling the GPU is pretty close to doing what a kernel does IMO.

toolslive 8y ago

> You should be able to forge a number into a pointer, since that’s how hardware works. It's a C-centric view of the world and I wonder what old school FORTRAN77 people or lispers would think of this. Anyway, I think the right to do this should be a privilege of the compiler. As soon as you claim this right, you abandon all possible support she can give you in battling all kinds of silly mistakes.

alxlaz 8y ago
I'm not an "old-school lisper", but I have done some Lisp development related to embedded systems. The phrasing may be somewhat unfortunate but it ultimately does describe how hardware works: you need to write something at a numerical address that you find in the reference guide or the datasheet. At the end of the line, you do have to forge a number into a pointer. In James Micken's words, "You can’t just place a LISP book on top of an x86 chip and hope that the hardware learns about lambda calculus by osmosis." "Higher-level" languages (like Lisp) can help you in other regards. They can include proper primitives (or give you the proper tools to write the proper primitives yourself) to ensure that direct-to-hardware access is safe, can give you better tools to manipulate DMA buffers (e.g. proper support for coroutines when manipulating double-buffered data is cool), can help you write more generic state-applying and state-reading code (i.e. code for "turn these bytes from this circular buffer into this structure" or "take this structure that represents a peripheral's current config and use it to write the proper magic numbers in the right registers"). They can offer you better semantics for translating human-readable(-ish) configuration into the right stream of bytes, or manipulate peripherals in real-time. They can help you write a better VM (or their runtime can outright be the better VM you need). But at the end of the day, writing to the config space of a memory-mapped device is still going to consist of taking a number (or building it from several numbers) and writing bytes at that address, no matter what language you're using. Better runtime support for safe operation under this scenario, better semantics -- they're all important, but what the author expresses is not C-centric in any way.
Athas 8y ago
Pointers are not even necessary for writing operating systems or memory allocators. The Oberon system writes those low-level components using intrinsic peek/poke functions (like some ancient BASIC!) that are recognised specially by the compiler and turned into direct memory modifications. This is clearly just as unsafe as pointers (probably more so), but it means you don't generally pollute the language to support some very specialised code. The rest of Oberon is memory-safe and garbage-collected (with the garbage collector also written in Oberon).

jillesvangurp 8y ago

System programming is definitely a bit overloaded these days though I believe it is now widely understood to be languages like C/C++ that can be used for OS, kernel, and embedded development. I would agree though that the language ecosystem is shifting a lot in the last few years. IMHO there are now a few languages that are becoming proper full stack languages in the sense that they scale from embedded all the way to browser development. Rust is a great example. I'm really impressed with what the Rust community has done in just a few short years. All my life, C/C++ was the only game in town for the stuff that is now done with Rust with arguably the same or in some cases slightly better performance. Stuff like WASM allows the use of system programming languages in places where they would not be used in the past. Using Rust transpiled to WASM makes it a proper fullstack language. And people are using it for that. And people do server development as well in it as well as OS development.

guelo 8y ago
Rust is nice but I wouldn't call it full stack because I don't think you should choose it in situations where you can afford a garbage collector. In those cases there are much more ergonomic options available.
- jillesvangurp 8y ago
  Well, I'm seeing people doing web apps in Rust and doing operating systems as well. Certainly the Rust community has been all over wasm and rust tooling for that is relatively mature compared to other languages. Precisely because it doesn't need a garbage collector and they don't have to wait for standard solutions for that to ship in the next year or so. But I agree, languages like Swift and Kotlin are a bit more logical choice as fullstack languages. Both of those are also extending their reach. Particulary, Kotlin code is a very nice upgrade from javascript in terms of expressiveness, tooling, and safety. I'm using it on top of the JVM currently. People are doing Android with it and the native compiler currently under development is explicitly targeted at doing native IOS and Android apps (i.e. not the current java based DEX vm common on Android).
saagarjha 8y ago

RickJWagner 8y ago

Nice article. My view of Systems Programming will forever be colored by my start with IBM mainframes. The Systems Programmer maintained all the utilities that the mainframe provided, and also the frameworks that supported online programming. I suppose today's equivalent is a combination of a sysadmin and an applications developer, providing the applications are things like container components, build pipelines, etc.

vram11 8y ago
They call them devops these days.

tempodox 8y ago

I think the author makes fair points. My conclusion: Do the bit twiddling and strongly constrained parts in a low level language and the rest in a language that supports strong static typing and good interfaces.

boulos 8y ago

You’re HN famous, Will! But I disagree with this: > Companies like Dropbox were able to build surprisingly large and scalable systems on just Python. Many of Dropbox’s services are written in Go, and Magic Pocket is written (partly, at least) in Rust. Earlier in their development, Dropbox relied on S3, which is obviously not in Python. Fundamentally, I think the important part of the “production” aspect of “systems programming” is that it’ll be used a lot. That’s what drives the requirement for efficiency: if you’re 10-100x less efficient, someone else will be more cost effective / solve bigger problems. As an additional example, GitHub was written in Ruby, but that’s fine because the underlying Git and filesystem manipulation is all in C. The same thing is sort of true of Facebook’s world of PHP: PHP is mostly a wrapper around C libraries. Until it got complex enough that they rewrote the language. tl;dr: I don’t think systems programming needs to be “low level”, but production systems do need to be efficient. More powerful computing just moves that further to the right!

burfog 8y ago

A systems programming language can run on bare hardware by itself, or nearly so. It is acceptable to require a very small amount of assembly code, for example to implement something like memcpy or bcopy, or to provide atomic operations. A language is disqualified if it requires an OS or if it requires code written in a different non-assembly language. Cheating, by adding that as a huge (impractical) amount of assembly, also disqualifies a language. Funny story about gcc: The compiler demands a memcpy, which it sometimes uses for struct assignment. If you implement memcpy in the normal way, gcc will helpfully replace your implementation with a call to memcpy! Despite this, C is still a systems programming language.

tralarpa 8y ago
> It is acceptable to require a very small amount of assembly code, for example to implement something like memcpy or bcopy, or to provide atomic operations. There is some stuff missing in your list: preparing the stack pointer, address layout, etc. You also need tools to produce text and data sections that can be loaded at a specific address. Even if C is a low level language, there is still quite some stuff between it and the "bare hardware". In that sense, the only language that gives you enough control to produce the binary exactly in the form needed by the hardware (without relying on external tools or libraries) is assembly.
- sigjuice 8y ago
  stack pointer is an implementation detail. The standard does not mention the word stack even once.
- saagarjha 8y ago
  > You also need tools to produce text and data sections that can be loaded at a specific address. Though, of course, most compilers provide extensions that make this task generally possible within C (using the loose definition of "I don't need any separate files or inline assembly, just attributes and flags).
  - raggi 8y ago

gjvc 8y ago

... this deserves a supplementary question: what will get us beyond UNIX to the next platform, not just the next paystub?

pjmlp 8y ago
Maybe something like Android, OS X/iOS, ChromeOS, UWP, Fuchsia, Unikernels, Language runtimes on Hypervisors. And since I sense the question coming, Android, OS X/iOS, ChromeOS might use an UNIXy kernel, but its exposure surface to applications is so small, that it can be easily exchange for something else, it is just a matter of cost. Which brings us to the next point, given the commodity of free UNIX-like OSes, maybe only an hardware reboot like Quantum computers will actually do it.
- saagarjha 8y ago
  > And since I sense the question coming, Android, OS X/iOS, ChromeOS might use an UNIXy kernel, but its exposure surface to applications is so small, that it can be easily exchange for something else, it is just a matter of cost. Android, macOS, and iOS expose almost the full set of POSIX to the programmer. Now, you may argue that it goes mostly unused, but it is there, and there are people using it.
  - pjmlp 8y ago
    Android does not, use at your own peril as it is not part of the official APIs and will get your app terminated if you use unauthorized APIs. Using libc is not the same as POSIX. https://developer.android.com/ndk/guides/stable_apis https://developer.android.com/ndk/guides/stable_apis https://developer.android.com/about/versions/nougat/android-7.0-changes#ndk https://developer.android.com/about/versions/nougat/android-... https://android-developers.googleblog.com/2016/06/android-changes-for-ndk-developers.html https://android-developers.googleblog.com/2016/06/android-ch... As for macOS, and iOS, if Apple removed POSIX how many apps written in Objective-C and Swift (not UNIX cli tools ported from Linux) would actually be affected? Very few, because those written in C and C++ ported from other platforms are most likely making use of ANSI C and C++ standards.

tanilama 8y ago

Nowadays it broadly means using C++ programming, involving dealing with different levels of system calls.

n_t 8y ago

Irrespective of language and framework, if the code needs to be aware of underlying hardware then it is systems programming. ex. kernel code, user-space drivers, databases, portions of cloud or server code which need to be hardware aware, etc. What golang targets, I think, should be more of middleware or service-level programming, whether its containers or servers.

steveklabnik 8y ago
Alan Perlis has a thoughtful way of expressing this: “a programming language is low level when its programs require attention to the irrelevant.” At first I thought it was poking fun; now I see it as a comment about what is relevant at a given level of abstraction.
- parley 8y ago
  I like that line of thinking. When discussing the suitability of different programming languages I always point to the problem at hand and identifying the abstraction level the problem is at. What your problem requires you to care about and pay attention to gives this. Ideally, solving your problem would be a one-liner in an already existing DSL created for your specific problem (domain). In reality, this rarely happens and you must have your pick beteeen everything from niche DSL:s to assembly, but the key really is identifying that problem abstraction level. I work in a mainly C/JS shop whereas I privately prefer Rust/ReasonML. Rust would be a great fit at work, but Go would also work nicely for tons of things we do. Alas, the inertia of organizations.
  - ModernMech 8y ago
    > Ideally, solving your problem would be a one-liner in an already existing DSL created for your specific problem I like thinking about the practice of programming as creating a DSL in a language to solve your specific problem.
- deleted 8y ago
  [deleted]

jkingsbery 8y ago

I'm curious where Erlang fits into this story. It was built to be a "systems" language but algorithms are tersely implemented (I forget exactly, but I remember reading about a study that an Erlang program is much shorter than the comparable C program), and has gradual typing of a sort. It also is garbage collected (I don't think you could "write your own memory allocator in it", as Andrei Alexandrescu described).

danmg 8y ago

640x480, 16 colours, Ring 0.

Analoganalog 8y ago

Hacker News is a site where someone corrects a slight error in a parent comment which then leads us down a bottomless rabbit hole that gets further and further away from the general point of the original post. Hacker News is a honeypot for people on the autism/aspergers spectrum. What is the color of chartreuse?

MichaelMoser123 8y ago

I don't think a single definition makes any sense now: decades ago you may have had a definition of a 'computer system' that describes every big program. Nowadays there are many sets of requirements depending on context, each set of requirements implies its own constraints on memory management and scheduling so that there is no one description of how a complex program would look like.

matachuan 8y ago

IDK about systems programming, but one thing I do know: Systems researchers = ^[AI researchers]

dfox 8y ago

I think that it is not useful to build on Ousterhout's definition as his definition of systems programming language is mostly meant as contrast to Tcl with it's everything is a string memory model (in that view one could very well conclude that Python is a systems programming language).

kmooney 8y ago

If you are making system calls–without too much abstraction–you are doing systems programming.

qubax 8y ago

In one word - syscalls. What do I win?

theoldgit 8y ago

when I was programming in the 70s and 80s (in the UK), the term systems programmer applied to those people involved in the configuration, tuning, management of software such as CICS, MVS, and their equivalents on other hardware bases. I worked at Sperry Univac for several years and the "systems programmers" were the engineers who tweaked the OS, the TP schedulers, built the product installation scripts, designed the configs that controlled the software. It was a specific tradecraft suited to those who were more technically focused, remember that COBOL was the prevalent language for nearly all programming roles in the 70s and that there was no "web" per se. The term started to fade away roughly when PCs started entering into mainstream computing

sytelus 8y ago

In old days, computers that came from manufacturers like IBM, DCE was refrred to as system. The operating software for the system was referred to as “system software”. People who created that software were called “systems programmer”. Much of it invariably involved writing direct to metal/assembly code which is why systems programming has been associated with low level programming. People using this term to refer to task of doing some complex application architecture are doing it wrong.