5 ms·

Years ago I modified the Postgresql source code (8.xx). It's something I was terrified of doing. But once I got in there and started poking around, I realized

Years ago I modified the Postgresql source code (8.xx). It's something I was terrified of doing. But once I got in there and started poking around, I realized it was just ordinary plain-vanilla C code. Not C++. Just C code. With my local copy, I started to hack pg_dump to do something special that we wanted at the time. Even after 30 years of coding, I'm not that especially good of a programmer. But I ended up getting our own special version of pg_dump that did what we wanted at the time and it went into production dumping hundreds of gigs of data every day! But what I'm not, is afraid. I'm not afraid to try anything. And that's what it takes to do deep, systems level programming. Don't be afraid. Those bits are just bits. And it's just code... and most of it was not written by wizards. Just ordinary people like you and me. Don't be afraid man. Clone the repo and setup a workable build environment and start tinkering and compiling and running to see what happens. You would be totally shocked to find out what you can actually achieve. Those books will only go so far.

wheels 5y ago
I basically came here to suggest the same thing. The biggest part is a mental shift: realize that if you find a bug (or feature you'd like to have) in an open source project that you have the ability to fix it. The rabbit hole of reading code you go down of reading code when you do that is, I believe, the best real introduction to systems programming. Once you've been doing that for a while, you internalize the idioms. I'd recommend large projects because you're less likely to just end up copying some person or couple of people's quirks, and learning to read large codebases is a major skill in itself. The process of submitting patches often turns into ad hoc mentoring, where you'll be told what you didn't get right this go around. After you've done that for a while, you realize the same things apply to other large systems projects, and you end up exposed to a number of different styles of systems stuff.
littlestymaar 5y ago
> Those bits are just bits. And it's just code... and most of it was not written by wizards. Just ordinary people like you and me. Don't be afraid man. Code is easy and doesn't scare me. I'm scared about the platform though: code runs on platforms (POSIX, mostly), which are incredibly dated, ill-designed and full of terrible corner cases[1]. There's so many ways to shoot yourself in the foot I'm afraid to do anything sensitive on my own. «It is not UNIX’s job to stop you from shooting your foot. If you so choose to do so, then it is UNIX’s job to deliver Mr. Bullet to Mr Foot in the most efficient way it knows.» [1]: Dan Lu's writing on files are a good illustration of that: - http://danluu.com/file-consistency/ http://danluu.com/file-consistency/ - https://danluu.com/deconstruct-files/ https://danluu.com/deconstruct-files/ - https://danluu.com/filesystem-errors/ https://danluu.com/filesystem-errors/
- wheels 5y ago
  Most POSIX APIs are crap, but most significant systems projects don't code to them directly. Almost all large projects have their own internal (or third party) libraries that either wrap or reimplement the underlying POSIX APIs. That said, at some point knowing the POSIX APIs does become necessary, but that's usually relevant a bit later when you start becoming interested in modifying the toolkit of large projects. If you're wanting to start a project on your own rather than jump into an existing project, in a nutshell, if you don't have hard real-time requirements, if you're using C, use glib, if you're using C++, use Qt. There are other libs that are useful for real-time usage (basically things that don't ever have hidden memory allocation), but I'm not familiar enough with that space to recommend one.
  - littlestymaar 5y ago
    How is that not just good old “application programming”? My own definition of “system programming” is exactly “developing on top of the system” vs “developing in the comfort of a helper library” (and its limits). I consider myself a decent application programmer, but I'm not a system programmer (at least not yet ;). The OP was talking about Postgres, if you don't know how the pitfalls of write(2), or don't know how to use mmap(2), you're going to have trouble making a database on your own.
    - wheels 5y ago
      Most people don't get into systems programming by writing a database completely on their own, and it's fair to say that the line between application and systems programming is fuzzy. Here are a bunch of things I would broadly consider systems programming: - Kernel and driver development - Low level library development (pretty much anything involving bit wrangling) - Platform abstraction libraries - Database development (not usage) - Message queuing systems - Daemon / server development Perhaps the recurring pattern there, and differentiated from application programming is that most of those are tools for other applications, rather than applications themselves. I don't think that all library / daemon development is systems programming, but a whole lot of it is. I'm coming from a background of having done 5 of the 6 of those groups (though there could obviously be more listed there). For most significant projects, as noted, there will be internal APIs that wrap system APIs (or in the case of the kernel, where the system APIs are totally irrelevant, except for the parts that implement the system calls). Generally someone first jumping into those projects isn't going to immediately start hacking on the internal libraries. As it were, I have actually written a database from scratch [1]. That database uses Qt and Qt-like APIs internally where possible. There are places in there where you need to know the intricacies of mmap, fsync and similar, but they're compartmentalized to a couple of classes. I'd still call code that isn't in those couple of classes "systems programming". [1] https://blog.directededge.com/2009/02/27/on-building-a-stupidly-fast-graph-database/ https://blog.directededge.com/2009/02/27/on-building-a-stupi...
Mc91 5y ago
My thoughts as well. Send patches in to FLOSS projects. I sent in a patch to an experienced C programmer a few years ago and in his code review he said about my #ifdef flags - "Typically HAVE_X are about the environment and USE_X are optional features within that environment." Listening to Linux kernel developers online, they say a lot of people start kernel development with one small patch, or attempting to port some new device or chip to work with Linux. Once their patch gets merged, they sometimes continue to send in patches. The developers say that good contributors tend to not last long as independent contributors, and tend to get pulled into companies who need such programmers. And there are open jobs for people with such skills ( https://us-redhat.icims.com/jobs/84882/senior-software-engineer/job?hub=7 https://us-redhat.icims.com/jobs/84882/senior-software-engin... )
skytreader 5y ago
I want to add to this. People might think they lack opportunities for this exercise because they don't have special requirements out of the open source packages they use. But realize that breaking a piece of code in a controlled manner also needs roughly the same level of understanding as adding a feature. It could be as simple as a `sleep` in the right circumstances. Maybe your resulting patch is a boring handful like: bool cond1 = ...; bool cond2 = ...; if (cond1 || cond2) { sleep(1000); } but you need to read and develop understanding of the code to define the conditions in terms of the program's state. And you also need the same understanding to know where exactly in the code to introduce this snippet. Not to mention compiling it; even with Makefiles, lower-level languages could still be tricky to build. How this can be useful: Long time ago, we were observing odd behavior from Redis under extremely high load. Of course it did not replicate consistently and when it did, it lasted too short for us to properly observe and make hypotheses. So I had the brilliant idea of installing a custom Redis binary in our test environment that tweaked the odds so that the behavior happened almost consistently, no need to thrash the server. I had to read through Redis source code to make it happen (and added a hell lot of logs too). Plus in the end, it's not quite compiling Linux but, boy, I did compile Redis from scratch!
handrous 5y ago
My perspective: Anything that's been done by a lot of people, has been done well-enough by some stupid asshole. I am a stupid asshole. Therefore, I can probably figure out how to do [thing] well-enough, if it's been done by a lot of people. It's worked out OK so far.
- bayindirh 5y ago
  I have printed out a foreword of an old calculus book, and taped to my wardrobe door. It's a long-ish rant, but it ends with this: What a fool can do, another can.
  - handrous 5y ago
    Pretty sure that's Calculus Made Easy, a book which I, as a fool, certainly ought to recognize, and yeah, that's the gist of my approach to "scary" topics. "Have lots and lots of people done it? Then I'll likely be fine, because some of those people were assuredly at least as dumb as I am."
  - jvanderbot 5y ago
    Considering how many fools can calculate, it is surprising that it should be thought either a difficult or a tedious task for any other fool to learn how to master the same tricks. Some calculus-tricks are quite easy. Some are enormously difficult. The fools who write the textbooks of advanced mathematics – and they are mostly clever fools – seldom take the trouble to show you how easy the easy calculations are. On the contrary, they seem to desire to impress you with their tremendous cleverness by going about it in the most difficult way. Being myself a remarkably stupid fellow, I have had to unteach myself the difficult, and now beg to present to my fellow fools the parts that are not hard. Master these thoroughly, and the rest will follow. What one fool can do, another can. http://djm.cc/library/Calculus_Made_Easy_Thompson.pdf http://djm.cc/library/Calculus_Made_Easy_Thompson.pdf
gentleman11 5y ago
A small counterpoint: I once worked in a code base where small changes would regularly ripple out and cause bugs far far away. You had to traverse a 30-50 source file chain of logic to find what was happening. This was “normal.” I’m not saying you have to be afraid, just that you sometimes have to tread a bit slower and do a lot of testing as you go until you learn the internals of whatever system you are working on (Aside: this is what coupling does — you should fear coupling!)
Omie6541 5y ago
This. I can relate to this as well. I will share couple of things just to add to this. I have ~8YoE now. When I was at my first job, I did not have any formal CS degree, I had completed bachelors in commerce and was struggling with masters in computer applications (had year drops). My first job was in a company started by ~7-8 ex-veritas folks, all of them being hardcore system developers. I had dreams of being the same like them some day (yet to happen). ~1 year in this job and I shared my aspirations to become a system developer - I was given a task, implement persistent ram mechanism, something that will persist data in the RAM even after soft reboot, without dumping data on hard drive, using Linux kernel. "what to do" (trick/technique) was told by them, how to do it was left for me. It took me 4 days (2 weekends) to complete this. Over first weekend I learnt how to get Linux source code, add custom syscall, compile kernel etc. On second weekend I actually got to go through the code, find places to add patches, test etc. I was also afraid before starting and my then boss had said few things like .. "because you think folks sitting in the west are something special, they are not ..", "whole thing is man made. If one man can do it, so can you". It was a matter of going through the code and understand. Do things repeatedly without giving up. Spend long hours, take notes. Once you have the context and that code is running in your head - you get what to do! I also did the something similar few years after this. I was learning Go and wanted to do something better. I got into delve codebase (debugger for Go) and I patched it to work for cgo binaries. Its a small patch but for that I had to learn delve's architecture + what is dwarf standard and stuff. Context was pretty huge compared to what I needed to patch it. Same story though, I was little afraid thinking - omg! it's a freaking debugger, how am I going to understand all this to make changes. But in fact they are just the same constructs. Go ahead and jump into some project you use, it helps understanding codebase faster. Read Read Read. Give yourself time to learn the codebase. You will definitely be able to contribute "properly" for that project. Don't be afraid :)

wheels 5y ago

I basically came here to suggest the same thing. The biggest part is a mental shift: realize that if you find a bug (or feature you'd like to have) in an open source project that you have the ability to fix it. The rabbit hole of reading code you go down of reading code when you do that is, I believe, the best real introduction to systems programming. Once you've been doing that for a while, you internalize the idioms. I'd recommend large projects because you're less likely to just end up copying some person or couple of people's quirks, and learning to read large codebases is a major skill in itself. The process of submitting patches often turns into ad hoc mentoring, where you'll be told what you didn't get right this go around. After you've done that for a while, you realize the same things apply to other large systems projects, and you end up exposed to a number of different styles of systems stuff.

littlestymaar 5y ago

> Those bits are just bits. And it's just code... and most of it was not written by wizards. Just ordinary people like you and me. Don't be afraid man. Code is easy and doesn't scare me. I'm scared about the platform though: code runs on platforms (POSIX, mostly), which are incredibly dated, ill-designed and full of terrible corner cases[1]. There's so many ways to shoot yourself in the foot I'm afraid to do anything sensitive on my own. «It is not UNIX’s job to stop you from shooting your foot. If you so choose to do so, then it is UNIX’s job to deliver Mr. Bullet to Mr Foot in the most efficient way it knows.» [1]: Dan Lu's writing on files are a good illustration of that: - http://danluu.com/file-consistency/ http://danluu.com/file-consistency/ - https://danluu.com/deconstruct-files/ https://danluu.com/deconstruct-files/ - https://danluu.com/filesystem-errors/ https://danluu.com/filesystem-errors/

wheels 5y ago
Most POSIX APIs are crap, but most significant systems projects don't code to them directly. Almost all large projects have their own internal (or third party) libraries that either wrap or reimplement the underlying POSIX APIs. That said, at some point knowing the POSIX APIs does become necessary, but that's usually relevant a bit later when you start becoming interested in modifying the toolkit of large projects. If you're wanting to start a project on your own rather than jump into an existing project, in a nutshell, if you don't have hard real-time requirements, if you're using C, use glib, if you're using C++, use Qt. There are other libs that are useful for real-time usage (basically things that don't ever have hidden memory allocation), but I'm not familiar enough with that space to recommend one.
- littlestymaar 5y ago
  How is that not just good old “application programming”? My own definition of “system programming” is exactly “developing on top of the system” vs “developing in the comfort of a helper library” (and its limits). I consider myself a decent application programmer, but I'm not a system programmer (at least not yet ;). The OP was talking about Postgres, if you don't know how the pitfalls of write(2), or don't know how to use mmap(2), you're going to have trouble making a database on your own.
  - wheels 5y ago
    Most people don't get into systems programming by writing a database completely on their own, and it's fair to say that the line between application and systems programming is fuzzy. Here are a bunch of things I would broadly consider systems programming: - Kernel and driver development - Low level library development (pretty much anything involving bit wrangling) - Platform abstraction libraries - Database development (not usage) - Message queuing systems - Daemon / server development Perhaps the recurring pattern there, and differentiated from application programming is that most of those are tools for other applications, rather than applications themselves. I don't think that all library / daemon development is systems programming, but a whole lot of it is. I'm coming from a background of having done 5 of the 6 of those groups (though there could obviously be more listed there). For most significant projects, as noted, there will be internal APIs that wrap system APIs (or in the case of the kernel, where the system APIs are totally irrelevant, except for the parts that implement the system calls). Generally someone first jumping into those projects isn't going to immediately start hacking on the internal libraries. As it were, I have actually written a database from scratch [1]. That database uses Qt and Qt-like APIs internally where possible. There are places in there where you need to know the intricacies of mmap, fsync and similar, but they're compartmentalized to a couple of classes. I'd still call code that isn't in those couple of classes "systems programming". [1] https://blog.directededge.com/2009/02/27/on-building-a-stupidly-fast-graph-database/ https://blog.directededge.com/2009/02/27/on-building-a-stupi...

Mc91 5y ago

My thoughts as well. Send patches in to FLOSS projects. I sent in a patch to an experienced C programmer a few years ago and in his code review he said about my #ifdef flags - "Typically HAVE_X are about the environment and USE_X are optional features within that environment." Listening to Linux kernel developers online, they say a lot of people start kernel development with one small patch, or attempting to port some new device or chip to work with Linux. Once their patch gets merged, they sometimes continue to send in patches. The developers say that good contributors tend to not last long as independent contributors, and tend to get pulled into companies who need such programmers. And there are open jobs for people with such skills ( https://us-redhat.icims.com/jobs/84882/senior-software-engineer/job?hub=7 https://us-redhat.icims.com/jobs/84882/senior-software-engin... )

skytreader 5y ago

I want to add to this. People might think they lack opportunities for this exercise because they don't have special requirements out of the open source packages they use. But realize that breaking a piece of code in a controlled manner also needs roughly the same level of understanding as adding a feature. It could be as simple as a `sleep` in the right circumstances. Maybe your resulting patch is a boring handful like: bool cond1 = ...; bool cond2 = ...; if (cond1 || cond2) { sleep(1000); } but you need to read and develop understanding of the code to define the conditions in terms of the program's state. And you also need the same understanding to know where exactly in the code to introduce this snippet. Not to mention compiling it; even with Makefiles, lower-level languages could still be tricky to build. How this can be useful: Long time ago, we were observing odd behavior from Redis under extremely high load. Of course it did not replicate consistently and when it did, it lasted too short for us to properly observe and make hypotheses. So I had the brilliant idea of installing a custom Redis binary in our test environment that tweaked the odds so that the behavior happened almost consistently, no need to thrash the server. I had to read through Redis source code to make it happen (and added a hell lot of logs too). Plus in the end, it's not quite compiling Linux but, boy, I did compile Redis from scratch!

handrous 5y ago

My perspective: Anything that's been done by a lot of people, has been done well-enough by some stupid asshole. I am a stupid asshole. Therefore, I can probably figure out how to do [thing] well-enough, if it's been done by a lot of people. It's worked out OK so far.

bayindirh 5y ago
I have printed out a foreword of an old calculus book, and taped to my wardrobe door. It's a long-ish rant, but it ends with this: What a fool can do, another can.
- handrous 5y ago
  Pretty sure that's Calculus Made Easy, a book which I, as a fool, certainly ought to recognize, and yeah, that's the gist of my approach to "scary" topics. "Have lots and lots of people done it? Then I'll likely be fine, because some of those people were assuredly at least as dumb as I am."
- jvanderbot 5y ago
  Considering how many fools can calculate, it is surprising that it should be thought either a difficult or a tedious task for any other fool to learn how to master the same tricks. Some calculus-tricks are quite easy. Some are enormously difficult. The fools who write the textbooks of advanced mathematics – and they are mostly clever fools – seldom take the trouble to show you how easy the easy calculations are. On the contrary, they seem to desire to impress you with their tremendous cleverness by going about it in the most difficult way. Being myself a remarkably stupid fellow, I have had to unteach myself the difficult, and now beg to present to my fellow fools the parts that are not hard. Master these thoroughly, and the rest will follow. What one fool can do, another can. http://djm.cc/library/Calculus_Made_Easy_Thompson.pdf http://djm.cc/library/Calculus_Made_Easy_Thompson.pdf

gentleman11 5y ago

A small counterpoint: I once worked in a code base where small changes would regularly ripple out and cause bugs far far away. You had to traverse a 30-50 source file chain of logic to find what was happening. This was “normal.” I’m not saying you have to be afraid, just that you sometimes have to tread a bit slower and do a lot of testing as you go until you learn the internals of whatever system you are working on (Aside: this is what coupling does — you should fear coupling!)

Omie6541 5y ago

This. I can relate to this as well. I will share couple of things just to add to this. I have ~8YoE now. When I was at my first job, I did not have any formal CS degree, I had completed bachelors in commerce and was struggling with masters in computer applications (had year drops). My first job was in a company started by ~7-8 ex-veritas folks, all of them being hardcore system developers. I had dreams of being the same like them some day (yet to happen). ~1 year in this job and I shared my aspirations to become a system developer - I was given a task, implement persistent ram mechanism, something that will persist data in the RAM even after soft reboot, without dumping data on hard drive, using Linux kernel. "what to do" (trick/technique) was told by them, how to do it was left for me. It took me 4 days (2 weekends) to complete this. Over first weekend I learnt how to get Linux source code, add custom syscall, compile kernel etc. On second weekend I actually got to go through the code, find places to add patches, test etc. I was also afraid before starting and my then boss had said few things like .. "because you think folks sitting in the west are something special, they are not ..", "whole thing is man made. If one man can do it, so can you". It was a matter of going through the code and understand. Do things repeatedly without giving up. Spend long hours, take notes. Once you have the context and that code is running in your head - you get what to do! I also did the something similar few years after this. I was learning Go and wanted to do something better. I got into delve codebase (debugger for Go) and I patched it to work for cgo binaries. Its a small patch but for that I had to learn delve's architecture + what is dwarf standard and stuff. Context was pretty huge compared to what I needed to patch it. Same story though, I was little afraid thinking - omg! it's a freaking debugger, how am I going to understand all this to make changes. But in fact they are just the same constructs. Go ahead and jump into some project you use, it helps understanding codebase faster. Read Read Read. Give yourself time to learn the codebase. You will definitely be able to contribute "properly" for that project. Don't be afraid :)