4 ms·

Most POSIX APIs are crap, but most significant systems projects don't code to them directly. Almost all large projects have their own internal (or third party)

by wheels 5y ago

Most POSIX APIs are crap, but most significant systems projects don't code to them directly. Almost all large projects have their own internal (or third party) libraries that either wrap or reimplement the underlying POSIX APIs. That said, at some point knowing the POSIX APIs does become necessary, but that's usually relevant a bit later when you start becoming interested in modifying the toolkit of large projects. If you're wanting to start a project on your own rather than jump into an existing project, in a nutshell, if you don't have hard real-time requirements, if you're using C, use glib, if you're using C++, use Qt. There are other libs that are useful for real-time usage (basically things that don't ever have hidden memory allocation), but I'm not familiar enough with that space to recommend one.

littlestymaar 5y ago
How is that not just good old “application programming”? My own definition of “system programming” is exactly “developing on top of the system” vs “developing in the comfort of a helper library” (and its limits). I consider myself a decent application programmer, but I'm not a system programmer (at least not yet ;). The OP was talking about Postgres, if you don't know how the pitfalls of write(2), or don't know how to use mmap(2), you're going to have trouble making a database on your own.
- wheels 5y ago
  Most people don't get into systems programming by writing a database completely on their own, and it's fair to say that the line between application and systems programming is fuzzy. Here are a bunch of things I would broadly consider systems programming: - Kernel and driver development - Low level library development (pretty much anything involving bit wrangling) - Platform abstraction libraries - Database development (not usage) - Message queuing systems - Daemon / server development Perhaps the recurring pattern there, and differentiated from application programming is that most of those are tools for other applications, rather than applications themselves. I don't think that all library / daemon development is systems programming, but a whole lot of it is. I'm coming from a background of having done 5 of the 6 of those groups (though there could obviously be more listed there). For most significant projects, as noted, there will be internal APIs that wrap system APIs (or in the case of the kernel, where the system APIs are totally irrelevant, except for the parts that implement the system calls). Generally someone first jumping into those projects isn't going to immediately start hacking on the internal libraries. As it were, I have actually written a database from scratch [1]. That database uses Qt and Qt-like APIs internally where possible. There are places in there where you need to know the intricacies of mmap, fsync and similar, but they're compartmentalized to a couple of classes. I'd still call code that isn't in those couple of classes "systems programming". [1] https://blog.directededge.com/2009/02/27/on-building-a-stupidly-fast-graph-database/ https://blog.directededge.com/2009/02/27/on-building-a-stupi...
  - wheels 5y ago
    I realize it's unlikely anyone will see this, but to amplify this point a little, I thought I'd poke through the Posgres source (which I'm not otherwise familiar with). By my count, mmap is only called in 4 places, in three files. The implementation behind this file is one of them: https://doxygen.postgresql.org/fd_8h.html https://doxygen.postgresql.org/fd_8h.html That is the actual file API that you'd use inside of Postgres. While the API there isn't going to win any beauty pageants, it's significantly more sane than raw POSIX. For comparison, this is the interface to mmaped files within my company's internal database: https://gist.github.com/scotchi/83609f5eb9b98ac3b4b4476ed621a033 https://gist.github.com/scotchi/83609f5eb9b98ac3b4b4476ed621... There's a grand total of 1 call to mmap in all of our source. Most of the time for writing code using mapped files here, all you'd see is the interface above.

by wheels 5y ago

littlestymaar 5y ago

How is that not just good old “application programming”? My own definition of “system programming” is exactly “developing on top of the system” vs “developing in the comfort of a helper library” (and its limits). I consider myself a decent application programmer, but I'm not a system programmer (at least not yet ;). The OP was talking about Postgres, if you don't know how the pitfalls of write(2), or don't know how to use mmap(2), you're going to have trouble making a database on your own.

wheels 5y ago
Most people don't get into systems programming by writing a database completely on their own, and it's fair to say that the line between application and systems programming is fuzzy. Here are a bunch of things I would broadly consider systems programming: - Kernel and driver development - Low level library development (pretty much anything involving bit wrangling) - Platform abstraction libraries - Database development (not usage) - Message queuing systems - Daemon / server development Perhaps the recurring pattern there, and differentiated from application programming is that most of those are tools for other applications, rather than applications themselves. I don't think that all library / daemon development is systems programming, but a whole lot of it is. I'm coming from a background of having done 5 of the 6 of those groups (though there could obviously be more listed there). For most significant projects, as noted, there will be internal APIs that wrap system APIs (or in the case of the kernel, where the system APIs are totally irrelevant, except for the parts that implement the system calls). Generally someone first jumping into those projects isn't going to immediately start hacking on the internal libraries. As it were, I have actually written a database from scratch [1]. That database uses Qt and Qt-like APIs internally where possible. There are places in there where you need to know the intricacies of mmap, fsync and similar, but they're compartmentalized to a couple of classes. I'd still call code that isn't in those couple of classes "systems programming". [1] https://blog.directededge.com/2009/02/27/on-building-a-stupidly-fast-graph-database/ https://blog.directededge.com/2009/02/27/on-building-a-stupi...
- wheels 5y ago
  I realize it's unlikely anyone will see this, but to amplify this point a little, I thought I'd poke through the Posgres source (which I'm not otherwise familiar with). By my count, mmap is only called in 4 places, in three files. The implementation behind this file is one of them: https://doxygen.postgresql.org/fd_8h.html https://doxygen.postgresql.org/fd_8h.html That is the actual file API that you'd use inside of Postgres. While the API there isn't going to win any beauty pageants, it's significantly more sane than raw POSIX. For comparison, this is the interface to mmaped files within my company's internal database: https://gist.github.com/scotchi/83609f5eb9b98ac3b4b4476ed621a033 https://gist.github.com/scotchi/83609f5eb9b98ac3b4b4476ed621... There's a grand total of 1 call to mmap in all of our source. Most of the time for writing code using mapped files here, all you'd see is the interface above.