7 ms·

The state of low-level system programming

98 points by bluehex 14y ago

zem 14y ago
there's also ats [http://www.ats-lang.org/ http://www.ats-lang.org/]. somewhere on my long-term todo list is to write a library of efficient data strucures in ats, to take advantage of some of its safety guarantees.
- doublec 14y ago
  It's a pity ATS doesn't get more attention in the "Systems programming language" field. It can call into C, be called from C, provides a powerful type system, and can be used to write very low level code. I've written a few posts about how ATS can interact with C on my weblog here: http://www.bluishcoder.co.nz/tags/ats/ http://www.bluishcoder.co.nz/tags/ats/
  - erichocean 14y ago
    I like reading your blog posts on ATS, but I don't think I could use it effectively. I've had much better luck with exhaustive testing with Klee and related libraries (e.g. for testing multi-threaded programs). For whatever reason, they map better to how I think about problems, and I've found it easier to develop new code that is testable within those constraints.
    - doublec 14y ago
      If you start off treating ATS like an ML variant interfacing with C without using any of the type annotations for safety you can go far. Then you can tighten the types up for safety as you get more experienced or find you need it. You could write tests as needed and when a test fails write types that make that failure as close to compile time as possible. Another approach is to just write C code, either embedded in ATS files or in C files that are linked into ATS. Then slowly convert C to ATS as you feel comfortable.
bluehex 14y ago
I really hope to see Go and Rust move in the direction of pluggable runtimes that can be embedded into a C host program. Currently, both runtimes require being seated as the main entry-point of the program to function. Both projects also have issues in their bug trackers to rectify this: Rust issue: https://github.com/mozilla/rust/issues/3608 https://github.com/mozilla/rust/issues/3608 Go issue: http://code.google.com/p/go/issues/detail?id=2790 http://code.google.com/p/go/issues/detail?id=2790 I wonder who will get there first.
- PuerkitoBio 14y ago
  Also, a way to compile without kernel (system) calls, so that it can be use for kernel development. It used to be possible with go, but no more with go1. Then these languages would truly be able to replace C.
rpearl 14y ago
C code can call into rust: http://static.rust-lang.org/doc/rust.html#extern-functions http://static.rust-lang.org/doc/rust.html#extern-functions
- bluehex 14y ago
  True, and so can C call back into go code with cgo. The problem is that it still requires that the main function be in a Rust/Go binary. A program written in C can't currently call into a library written in one of these two languages since the runtimes don't work unless they control the main entry point. From the Rust doc linked: "The primary motivation of extern functions is to create callbacks for foreign functions that expect to receive function pointers."
  - jzwinck 14y ago
    Then is it even possible to make a program that can use libraries written in Rust and Go and C++? If everyone wants to control main(), is there an escape hatch provided by any of these languages, like a C function that can be explicitly invoked near program start?
    - smosher 14y ago
      You can do this with C++ already. Rust's plan seems to be to eliminate or reduce its runtime and give up the entry point. Go doesn't appear to have a plan.
- josephlord 14y ago
  If you read what you linked to it says the main purpose is to provide C functions with callback arguments when called from Rust. It isn't clear from what you have written that this mechanism is available to generally access from C programs and scripting languages.
zanny 14y ago
Compatability with the C ABI I feel isn't a necesssity. On platforms built on C it is, but if you wanted to invent a new machine running a new system language with the ABI messes of C exonerated, you could. You would need to devise a translation layer in the OS executor to be able to interpret classic C ABI .so or .dll files for legacy applications and so you could compile C on these platforms, but that isn't a prohibitively hard challenge.
- keypusher 14y ago
  Doesn't almost all systems programming take place on platforms built on C? If creating a new machine and new language, convincing the existing systems programmers to move is going to take something very compelling.
  - davidw 14y ago
    Historically, the "turtles all the way down" approach has not been very successful outside of niches. Lisp, Forth, Java have all been used for OS's that never seemed to go anywhere.
    - Symmetry 14y ago
      I've always wondered about the possibility of creating a stack system, application, and scripting language all designed to work together though.
      - zanny 14y ago
        I'd much rather see three distinct languages, with a shared base syntax and style (ie, common formatting of comments, common scope delimiters, common operand behavior - something like C / Java / Javascript). That way, you could have your low level language use pointers, nullables, inline assembly, goto and other "dangerous" things that are the most efficient ways to do stuff. You wouldn't want to have that language crippled like modern C++ though, you would still want modules, a well planned object oriented syntax, a better solution to name mangling, and better readability. You could have an interpreted Java/C# esque applications language that would mostly be similar to modern Java/C#, maybe with auto, a better namespaces implementation (in C#, a class defined in two places with the same namespace and signature have situational behavior that gets really messy, for example). You would use this language for the application layer. Scripting could just be an autoboxed dynamic language like Python. Maybe have a whitespace significant and insignificant dialect. I'd like to see dialects of such a language, where the context it runs from dictates available modules. Since it would be permissions based, it could function as a shell, web, and glue language depending on where you use it. I do agree on this sentiment though. The wide base of languages in modern computing makes the whole thing a lot messier than it needs to be, and the lack of any full stack that really ties together nicely and is well thought out without the smudges would be really nice.
gmt2027 14y ago
Vala does not seem to get much attention. It borrows C# syntax (and maybe Java), compiles down to C and offers nearly symmetrical interoperability as a result. I believe Ubuntu's Unity is written in Vala. Knowing Java, it was easy to acquire the syntax from the tutorial [https://live.gnome.org/Vala/Tutorial https://live.gnome.org/Vala/Tutorial] in very little time. For transparent access to compiled C libraries, external C functions need to be declared in a 'vapi' file and although these were already provided for a large part of the GNOME ecosystem, documentation on vapi syntax was nearly non-existent. Object-orientation and standard libraries are integrated with Gnome's GObject system and Glib libraries which I found unfamiliar and usually better documented in C at the time.
- qxcv 14y ago
  If it has GObject as a dependency, then it's not designed for "low-level systems programming". Hell, just having a GC more-or-less discounts it due to the associated problems with runtime size and performance[0]. As a general rule of thumb: if you can't write a kernel in it, it's not a systems language. [0] It is possible to write a systems language with these features, but low-level systems stuff is not what Vala was designed for.
  - gmt2027 14y ago
    GObject itself is not a dependency, Inheriting from GLib.GObject provides support for features like introspection and better integration with the collections framework and GLib libraries. However, lightweight 'non-object classes' are supported but treated differently by the compiler. There is a slight variation in constructor syntax between plain objects and GObjects. As far as I know there is no Vala runtime or garbage collector. Memory management is by automatic reference counting but there is support for manual C++-style pointer syntax with new and delete operators. I agree that Vala was not specifically designed for 'low-level systems stuff' but I see no reason why this is not technically possible. The Vala compiler is a source-to-source translator to C and in my previous experience worked fairly well interfacing with fairly low-level C libraries like MPI and OpenCL. I assume that a kernel writer does not require much more than a limited subset of the language that supports C pointers, structs and arrays.
justincormack 14y ago
The C ABI is not great for interacting with scripting languages though, even if it is the best we have right now. For a start there is no type information obtained with dlsym and you basically have to parse headers the interpretation of which is not clearly defined as the C language really defines an API and the ABI is defined by the compiler. Many libraries for this reason do not define stable ABIs and are a pain to use from a scripting language. Hence Python has two ffi modes an ABI interface and a compiler mode. Now if someone produced a better solution to that it would be useful...
- qznc 14y ago
  In practice it works quite well, if you restrict the ABI to function calls and word-sized values (long and void). It especially means not to use structs. Instead you need void and getter/setter functions.
  - justincormack 14y ago
    Thats very limiting. A lot of applications need arrays (and maybe complex numbers) eg for numerical applications. And it is hard to manage without some sort of int64 type nowadays. And also say you want performance, then using getters and setters sucks compared to direct struct access, as every access needs a non inlineable function call.
- angersock 14y ago
  You know, it's strange, because the Lua folks seem to be getting along just fine.
  - justincormack 14y ago
    Yeah but us Lua folks are picky about the sort of C we like to interface to.
tehwalrus 14y ago
I asked a very relevant question on stackoverflow about this recently: "Writing a Python extension in Go" http://stackoverflow.com/questions/12443203/writing-a-python-extension-in-go-golang http://stackoverflow.com/questions/12443203/writing-a-python... Now technically you can compile a special python which can handle Go extensions, which is the wrapper mentioned in the second answer, but you can't distribute python modules that ask people to install "this special python" to run (which will reliably work with all their other installed code..) I would like to see a "go init" function in a C API, that would let you interop the two whichever had the `main()`, since that would let you link with all the other c-implemented scripting languages (python, ruby, etc) which would mean that Go extension support could be added to the main cpython trunk as a compile-time option.
deleted 14y ago
[deleted]
damian2000 14y ago
If programmers wanted a statically-typed, compiled system language for their scripting efforts, they would be using the existing system languages for that already. I think Go has been fairly successful at Google in terms of improved code maintainability, and as an alternative to not just C but C++.
- pretoriusB 14y ago
  >I think Go has been fairly successful at Google in terms of improved code maintainability, and as an alternative to not just C but C++. Google at large barely uses the language. All the examples they've given are of peripheral stuff, like a load balancer for MySQl for YouTube and an even simpler replacement for an aged and cranky C++ system for Google Downloads. Nothing to write home about, and no replacing C++/Java for core systems at all.
- slurry 14y ago
  It seems to me that they are, in fact, using "statically-typed, compiled system language[s] for their scripting efforts" - that is in fact what people are doing with server-side Java, Scala and C#, not to mention Java/Dalvik and Objective-C on mobile platforms. Perhaps this article misses the point a bit. For several decades we have been using scripting languages to glue together sharp pointy C[++] tools (sh/Perl/Python/Ruby approach) or to puppetize monolithic C[++] hulks from within (VB/elisp approach). More recent practice, on both mobile and server platforms, has been to abstract away from C[++] with a middle layer of memory-managed, compiled C-ish goop - JVM, .Net, Objective-C, whatever. This is the level where people are doing everything from amazing high-availability big data things to amazing little bleep-bloop handheld things. I think the answer to "what's the state of low-level systems programming?" is "not much is going on, but damn, there's a lot of fresh stuff happening in mid-level systems programming." Go and Rust are late entrants to this party, and probably exist more as a hedge against Oracle litigating Java/JVM into the ground than anything else.
cturner 14y ago
I don't think this stands up: "It must be possible for such contender to be called from C code, because that is what it takes to be called from a modern scripting engine too." Later the author says, "If programmers wanted a statically-typed, compiled system language for their scripting efforts, they would be using the existing system languages for that already." I think the reason that scripting languages exist is because of limitations in the established languages. What we think of as "a scripting language" is defined by the gap. If someone else could be a system programming language without the gap - maybe it doesn't need to be accessed. The traditional unix platform is (1) C and a compiler for it, with access to the unix standard library and (2) some sh or csh derivative, and access to the likes of sed and awk. Shell gave people a glimpse of what was possible. But shell has poor access to the unix API, and hits the wall quickly once you do anything other than very basic tool development. So these things called scripting languages started to pop up. People wanted easy access to the unix API like C. They want easy string manipulation like shell. And they want access to functions, maps and lists without having to think about memory allocation. If you start from the shell and awk and work into that gap you get perl - the canonical scripting languge. TCL inhabits a similar space to perl. It's also "a scripting language". But it started at the opposite end. TCL starts as an extension language for C. So I can understand the author's emphsais on close interaction with C. But I don't think they're important if you can take away the shortfalls they're compensating for. I hope Go will mean I don't have to think about these things. That I'll be able to get a standard build and a go compiler/runtime and have immediate access to lots of power. No more headaches from C build tools or reaching around in the murky water underneath my scripting language.
- jedbrown 14y ago
  You are leaving out the importance of libraries and coupling of systems that were initially written independently. This is the same mistake that many Lisps have made, assuming that just because the technology is "superior", people will flock to it and eagerly rewrite all their code (starting with their dependencies).
pretoriusB 14y ago
>The Go language and the Rust language. What they have in common, though, is that both Go and Rust are entirely incapable of walking like a duck. Since you cannot call Rust -or Go code from a scripting engine, they are inadvertently positioning themselves as alternatives to the scripting engines themselves. Before writing all these BS, maybe he should do a little research first? Here's Rust (which is not even released yet) being capable of being called from C. It's a feature they are actively work on: https://github.com/mozilla/rust/issues/1732 https://github.com/mozilla/rust/issues/1732
- bluehex 14y ago
  > Before writing all these BS, maybe he should do a little research first? There's a subtle difference between being able to call back into Go/Rust code from C (where the main function is in Go/Rust) which both languages allow now, and allowing a call to originate from a C program, which neither currently do. The author is talking about the later. I would agree the article could go into more detail about what's currently possible, and why it's not enough. But calling BS, I think, is going a bit too far.
  - pretoriusB 14y ago
    >and allowing a call to originate from a C program, which neither currently do. That one is also under consideration by the Rust developers.
groovy2shoes 14y ago
Clay[1] doesn't get as much attention as it deserves. The authors seem well-versed in programming language theory, but they also understand the practical needs of a systems language. What you end up with is a language that's sufficiently low-level to be a C replacement, but it offers a plethora of mechanisms for abstraction. Of all the new languages billed as "systems languages", Clay is the only one that excites me. It can also "quack like a duck" (to put it in TFA terms). [1] http://claylabs.com/clay/ http://claylabs.com/clay/
- MatthewPhillips 14y ago
  I can't find any kind of getting started guide, and most of the libs have no documentation at all.
  - qznc 14y ago
    I found a "Clay for C++ programmers" tutorial https://github.com/jckarter/clay/wiki/Clay-for-C---programmers https://github.com/jckarter/clay/wiki/Clay-for-C---programme...
- ponce 14y ago
  May I mention Nimrod [1]? It has a lot going for it when it comes to meta-programming. [1] http://nimrod-code.org/ http://nimrod-code.org/
MatthewPhillips 14y ago
So what's the catch when it comes to D? If I can use all C libs easily, and get some more palpable syntax, what am I missing exactly and why haven't people transitioned to it?
- pretoriusB 14y ago
  The article is not about using C libs, is about C programs using you (your D code, that is). That said, the problem with D: not enough marketing plus the two compilers / 2 libraries issue. And it's close to C++ complexity enough to not let the scripting guys feel welcome (like they do in Go).
frujka 14y ago
it's probably an interesting read except that being hosted on google's blogspot means it only display blank pages due to content being embedded in some javascript shenanigans. Even a screenshot would be better than this. Please move on to some actual html website that just works or rehost the content elsewhere before posting it here. Thanks.
- eropple 14y ago
  I used to be one of These Guys who never used JavaScript, too, but honestly? It's 2012 and JavaScript is a thing pretty much everywhere worth targeting. Demanding that somebody cater to your microsegment is a little much.
  - vq 14y ago
    Although I'm the kind of guy who usually have JS disabled I could be made to see your point. But this page is actually a good example of why I run with JS disabled. It magically positioned links and boxes over the scrolling bar and caused my initial attempt at scrolling down to navigate away from the page. Disabling JS usually have the effect of doing away with such nonsense but here that is no option. This time it was worth the hassle since it was an interesting article, but I definitely prefer it when JS is optional.
- nnq 14y ago
  > rehost the content elsewhere before posting it here Google's blogspot allows you to do normal html blogs (maybe now they've made this their "default", but still, it's OP's choice to have it this way)
- sublimit 14y ago
  My attempts to read this on my phone were futile. Every time I tried to scroll, it just jumped back to the top immediately.
dschiptsov 14y ago
This is called FFI.) And yes, a language with which you cannot dload and call any shared library you wish is, how to say, handicapped language. NodeJS, Java to name a few. Look what they do due to inability to call, say, libpq.so.) But of course, re-monkey-patching everything from scratch by amateurs coders is much better idea. And, you know, you could create an entire "standard" with hundreds of pages of specification and then push your bloatware - JDBC.
- pjscott 14y ago
  I believe his thesis was actually that most languages are handicapped because they can not easily be called via the usual "load a .so file and call a function" ABI. Java, for instance, would be a more useful language to me if I could easily create Python bindings for useful Java libraries like Lucene, without needing to use something like Jython that runs on the JVM.
  - amirouche 14y ago
    tl;dr: use JNI through pyjnius of jpype. You can run Python 2.5 on lastest Jython, then you have access to JVM libs and the JVM libs to Python libs 2.5 compatible. Calling JVM from CPython is actually possible: - with pyjnius from kivy fame built with cython it is still in development, AFAIK it is missing the possibility to call Python code from Java code aka. submitting a Python "callback" to a Java method, I think it is called IoC pattern and Hollywood principle «Don't call me, I call you». jnius is used in Kivy to make it possible to call Android libs. As an example, I created Python bindings of Blueprints with it. Blueprints is an abstraction library for graphdb like Neo4j, OrientDB, Titan, available here https://github.com/Printemps/python-blueprints/ https://github.com/Printemps/python-blueprints/. Tutorial: «JavaClass = autoclass('path.to.JavaClass'); JavaClass.JavaMethod(python_object)». Those bindings are merely Pythonification but they are gotchas in types conversions. - There is also jpype which looks like an established solution used by Neo4J fame to make it possible to embed Neo4J embedded into CPython. I choosed jnius because it's faster; Those are funny benchmarks in log scale: https://docs.google.com/spreadsheet/ccc?key=0Ajc7MnpSjCokdGtsME01MnVmNnhid1ZhT210ZUt4WEE#gid=0 https://docs.google.com/spreadsheet/ccc?key=0Ajc7MnpSjCokdGt...
  - 14y ago
pcwalton 14y ago
Freestanding, ABI-level interoperability with C is planned for Rust, but is not yet implemented. There are two main reasons why being able to be called with dlsym() was less important for us than the current setup: 1. M:N scheduling. We wanted high scalability with hundreds of thousands or millions of tasks, which means we need the capability to have our own scheduler. OS threads aren't lightweight enough for this, especially on Windows, where over 80% of our Firefox users are. Having our own scheduler means we need our own ABI. We would like to make the scheduler optional, but getting it working was a higher priority than making it optional. 2. Safety. On many platforms (basically, anything but MSVC on Windows), the C stack is not bounds-checked; either there is no guard page or the guard page is unreliable because you can overshoot it. This is not acceptable for our Rust code, since stack overflows are a frequent source of exploitable security vulnerabilities for us. So we do our own stack management with a custom ABI that performs stack checks at every function entry, freeing us from this source of pain on all operating systems. Again, this is planned to be optional, but it was more important at this early stage for us to demonstrate advantages over C than for us to duplicate its ABI exactly. Many libraries, however, would prefer direct interoperability with C to M:N scheduling or stack overflow safety. So we plan to provide a mechanism for Rust code to be called from C without having to set up a scheduler or perform stack checks. It's not implemented yet, but it's very much part of the current design, and I would be thrilled to see it happen.
lispertoascheme 14y ago
The state of programmers trying to avoid coding in the language their system is coded in. C.
martincmartin 14y ago
For example, Java and C# cannot walk like a duck, not even if that is what it would take to save themselves from drowning. Therefore, they are absolutely unsuitable as system languages. Hadoop held the world record for sorting a Terrabyte for a while. If sorting large amounts of data as fast as possible isn't a system's programming task, then I'm a three-headed monkey.
- eswangren 14y ago
  Yes, because "must sort quickly" is the only important requirement in a systems level language.

98 points by bluehex 14y ago

zem 14y ago

there's also ats [http://www.ats-lang.org/ http://www.ats-lang.org/]. somewhere on my long-term todo list is to write a library of efficient data strucures in ats, to take advantage of some of its safety guarantees.

doublec 14y ago
It's a pity ATS doesn't get more attention in the "Systems programming language" field. It can call into C, be called from C, provides a powerful type system, and can be used to write very low level code. I've written a few posts about how ATS can interact with C on my weblog here: http://www.bluishcoder.co.nz/tags/ats/ http://www.bluishcoder.co.nz/tags/ats/
- erichocean 14y ago
  I like reading your blog posts on ATS, but I don't think I could use it effectively. I've had much better luck with exhaustive testing with Klee and related libraries (e.g. for testing multi-threaded programs). For whatever reason, they map better to how I think about problems, and I've found it easier to develop new code that is testable within those constraints.
  - doublec 14y ago
    If you start off treating ATS like an ML variant interfacing with C without using any of the type annotations for safety you can go far. Then you can tighten the types up for safety as you get more experienced or find you need it. You could write tests as needed and when a test fails write types that make that failure as close to compile time as possible. Another approach is to just write C code, either embedded in ATS files or in C files that are linked into ATS. Then slowly convert C to ATS as you feel comfortable.

bluehex 14y ago

I really hope to see Go and Rust move in the direction of pluggable runtimes that can be embedded into a C host program. Currently, both runtimes require being seated as the main entry-point of the program to function. Both projects also have issues in their bug trackers to rectify this: Rust issue: https://github.com/mozilla/rust/issues/3608 https://github.com/mozilla/rust/issues/3608 Go issue: http://code.google.com/p/go/issues/detail?id=2790 http://code.google.com/p/go/issues/detail?id=2790 I wonder who will get there first.

PuerkitoBio 14y ago
Also, a way to compile without kernel (system) calls, so that it can be use for kernel development. It used to be possible with go, but no more with go1. Then these languages would truly be able to replace C.

rpearl 14y ago

C code can call into rust: http://static.rust-lang.org/doc/rust.html#extern-functions http://static.rust-lang.org/doc/rust.html#extern-functions

bluehex 14y ago
True, and so can C call back into go code with cgo. The problem is that it still requires that the main function be in a Rust/Go binary. A program written in C can't currently call into a library written in one of these two languages since the runtimes don't work unless they control the main entry point. From the Rust doc linked: "The primary motivation of extern functions is to create callbacks for foreign functions that expect to receive function pointers."
- jzwinck 14y ago
  Then is it even possible to make a program that can use libraries written in Rust and Go and C++? If everyone wants to control main(), is there an escape hatch provided by any of these languages, like a C function that can be explicitly invoked near program start?
  - smosher 14y ago
    You can do this with C++ already. Rust's plan seems to be to eliminate or reduce its runtime and give up the entry point. Go doesn't appear to have a plan.
josephlord 14y ago
If you read what you linked to it says the main purpose is to provide C functions with callback arguments when called from Rust. It isn't clear from what you have written that this mechanism is available to generally access from C programs and scripting languages.

zanny 14y ago

Compatability with the C ABI I feel isn't a necesssity. On platforms built on C it is, but if you wanted to invent a new machine running a new system language with the ABI messes of C exonerated, you could. You would need to devise a translation layer in the OS executor to be able to interpret classic C ABI .so or .dll files for legacy applications and so you could compile C on these platforms, but that isn't a prohibitively hard challenge.

keypusher 14y ago
Doesn't almost all systems programming take place on platforms built on C? If creating a new machine and new language, convincing the existing systems programmers to move is going to take something very compelling.
- davidw 14y ago
  Historically, the "turtles all the way down" approach has not been very successful outside of niches. Lisp, Forth, Java have all been used for OS's that never seemed to go anywhere.
  - Symmetry 14y ago
    I've always wondered about the possibility of creating a stack system, application, and scripting language all designed to work together though.
    - zanny 14y ago
      I'd much rather see three distinct languages, with a shared base syntax and style (ie, common formatting of comments, common scope delimiters, common operand behavior - something like C / Java / Javascript). That way, you could have your low level language use pointers, nullables, inline assembly, goto and other "dangerous" things that are the most efficient ways to do stuff. You wouldn't want to have that language crippled like modern C++ though, you would still want modules, a well planned object oriented syntax, a better solution to name mangling, and better readability. You could have an interpreted Java/C# esque applications language that would mostly be similar to modern Java/C#, maybe with auto, a better namespaces implementation (in C#, a class defined in two places with the same namespace and signature have situational behavior that gets really messy, for example). You would use this language for the application layer. Scripting could just be an autoboxed dynamic language like Python. Maybe have a whitespace significant and insignificant dialect. I'd like to see dialects of such a language, where the context it runs from dictates available modules. Since it would be permissions based, it could function as a shell, web, and glue language depending on where you use it. I do agree on this sentiment though. The wide base of languages in modern computing makes the whole thing a lot messier than it needs to be, and the lack of any full stack that really ties together nicely and is well thought out without the smudges would be really nice.

gmt2027 14y ago

Vala does not seem to get much attention. It borrows C# syntax (and maybe Java), compiles down to C and offers nearly symmetrical interoperability as a result. I believe Ubuntu's Unity is written in Vala. Knowing Java, it was easy to acquire the syntax from the tutorial [https://live.gnome.org/Vala/Tutorial https://live.gnome.org/Vala/Tutorial] in very little time. For transparent access to compiled C libraries, external C functions need to be declared in a 'vapi' file and although these were already provided for a large part of the GNOME ecosystem, documentation on vapi syntax was nearly non-existent. Object-orientation and standard libraries are integrated with Gnome's GObject system and Glib libraries which I found unfamiliar and usually better documented in C at the time.

qxcv 14y ago
If it has GObject as a dependency, then it's not designed for "low-level systems programming". Hell, just having a GC more-or-less discounts it due to the associated problems with runtime size and performance[0]. As a general rule of thumb: if you can't write a kernel in it, it's not a systems language. [0] It is possible to write a systems language with these features, but low-level systems stuff is not what Vala was designed for.
- gmt2027 14y ago
  GObject itself is not a dependency, Inheriting from GLib.GObject provides support for features like introspection and better integration with the collections framework and GLib libraries. However, lightweight 'non-object classes' are supported but treated differently by the compiler. There is a slight variation in constructor syntax between plain objects and GObjects. As far as I know there is no Vala runtime or garbage collector. Memory management is by automatic reference counting but there is support for manual C++-style pointer syntax with new and delete operators. I agree that Vala was not specifically designed for 'low-level systems stuff' but I see no reason why this is not technically possible. The Vala compiler is a source-to-source translator to C and in my previous experience worked fairly well interfacing with fairly low-level C libraries like MPI and OpenCL. I assume that a kernel writer does not require much more than a limited subset of the language that supports C pointers, structs and arrays.

justincormack 14y ago

The C ABI is not great for interacting with scripting languages though, even if it is the best we have right now. For a start there is no type information obtained with dlsym and you basically have to parse headers the interpretation of which is not clearly defined as the C language really defines an API and the ABI is defined by the compiler. Many libraries for this reason do not define stable ABIs and are a pain to use from a scripting language. Hence Python has two ffi modes an ABI interface and a compiler mode. Now if someone produced a better solution to that it would be useful...

qznc 14y ago
In practice it works quite well, if you restrict the ABI to function calls and word-sized values (long and void). It especially means not to use structs. Instead you need void and getter/setter functions.
- justincormack 14y ago
  Thats very limiting. A lot of applications need arrays (and maybe complex numbers) eg for numerical applications. And it is hard to manage without some sort of int64 type nowadays. And also say you want performance, then using getters and setters sucks compared to direct struct access, as every access needs a non inlineable function call.
angersock 14y ago
You know, it's strange, because the Lua folks seem to be getting along just fine.
- justincormack 14y ago
  Yeah but us Lua folks are picky about the sort of C we like to interface to.

tehwalrus 14y ago

I asked a very relevant question on stackoverflow about this recently: "Writing a Python extension in Go" http://stackoverflow.com/questions/12443203/writing-a-python-extension-in-go-golang http://stackoverflow.com/questions/12443203/writing-a-python... Now technically you can compile a special python which can handle Go extensions, which is the wrapper mentioned in the second answer, but you can't distribute python modules that ask people to install "this special python" to run (which will reliably work with all their other installed code..) I would like to see a "go init" function in a C API, that would let you interop the two whichever had the `main()`, since that would let you link with all the other c-implemented scripting languages (python, ruby, etc) which would mean that Go extension support could be added to the main cpython trunk as a compile-time option.

deleted 14y ago

[deleted]

damian2000 14y ago

If programmers wanted a statically-typed, compiled system language for their scripting efforts, they would be using the existing system languages for that already. I think Go has been fairly successful at Google in terms of improved code maintainability, and as an alternative to not just C but C++.

pretoriusB 14y ago
>I think Go has been fairly successful at Google in terms of improved code maintainability, and as an alternative to not just C but C++. Google at large barely uses the language. All the examples they've given are of peripheral stuff, like a load balancer for MySQl for YouTube and an even simpler replacement for an aged and cranky C++ system for Google Downloads. Nothing to write home about, and no replacing C++/Java for core systems at all.
slurry 14y ago
It seems to me that they are, in fact, using "statically-typed, compiled system language[s] for their scripting efforts" - that is in fact what people are doing with server-side Java, Scala and C#, not to mention Java/Dalvik and Objective-C on mobile platforms. Perhaps this article misses the point a bit. For several decades we have been using scripting languages to glue together sharp pointy C[++] tools (sh/Perl/Python/Ruby approach) or to puppetize monolithic C[++] hulks from within (VB/elisp approach). More recent practice, on both mobile and server platforms, has been to abstract away from C[++] with a middle layer of memory-managed, compiled C-ish goop - JVM, .Net, Objective-C, whatever. This is the level where people are doing everything from amazing high-availability big data things to amazing little bleep-bloop handheld things. I think the answer to "what's the state of low-level systems programming?" is "not much is going on, but damn, there's a lot of fresh stuff happening in mid-level systems programming." Go and Rust are late entrants to this party, and probably exist more as a hedge against Oracle litigating Java/JVM into the ground than anything else.

cturner 14y ago

I don't think this stands up: "It must be possible for such contender to be called from C code, because that is what it takes to be called from a modern scripting engine too." Later the author says, "If programmers wanted a statically-typed, compiled system language for their scripting efforts, they would be using the existing system languages for that already." I think the reason that scripting languages exist is because of limitations in the established languages. What we think of as "a scripting language" is defined by the gap. If someone else could be a system programming language without the gap - maybe it doesn't need to be accessed. The traditional unix platform is (1) C and a compiler for it, with access to the unix standard library and (2) some sh or csh derivative, and access to the likes of sed and awk. Shell gave people a glimpse of what was possible. But shell has poor access to the unix API, and hits the wall quickly once you do anything other than very basic tool development. So these things called scripting languages started to pop up. People wanted easy access to the unix API like C. They want easy string manipulation like shell. And they want access to functions, maps and lists without having to think about memory allocation. If you start from the shell and awk and work into that gap you get perl - the canonical scripting languge. TCL inhabits a similar space to perl. It's also "a scripting language". But it started at the opposite end. TCL starts as an extension language for C. So I can understand the author's emphsais on close interaction with C. But I don't think they're important if you can take away the shortfalls they're compensating for. I hope Go will mean I don't have to think about these things. That I'll be able to get a standard build and a go compiler/runtime and have immediate access to lots of power. No more headaches from C build tools or reaching around in the murky water underneath my scripting language.

jedbrown 14y ago
You are leaving out the importance of libraries and coupling of systems that were initially written independently. This is the same mistake that many Lisps have made, assuming that just because the technology is "superior", people will flock to it and eagerly rewrite all their code (starting with their dependencies).

pretoriusB 14y ago

>The Go language and the Rust language. What they have in common, though, is that both Go and Rust are entirely incapable of walking like a duck. Since you cannot call Rust -or Go code from a scripting engine, they are inadvertently positioning themselves as alternatives to the scripting engines themselves. Before writing all these BS, maybe he should do a little research first? Here's Rust (which is not even released yet) being capable of being called from C. It's a feature they are actively work on: https://github.com/mozilla/rust/issues/1732 https://github.com/mozilla/rust/issues/1732

bluehex 14y ago
> Before writing all these BS, maybe he should do a little research first? There's a subtle difference between being able to call back into Go/Rust code from C (where the main function is in Go/Rust) which both languages allow now, and allowing a call to originate from a C program, which neither currently do. The author is talking about the later. I would agree the article could go into more detail about what's currently possible, and why it's not enough. But calling BS, I think, is going a bit too far.
- pretoriusB 14y ago
  >and allowing a call to originate from a C program, which neither currently do. That one is also under consideration by the Rust developers.

groovy2shoes 14y ago

Clay[1] doesn't get as much attention as it deserves. The authors seem well-versed in programming language theory, but they also understand the practical needs of a systems language. What you end up with is a language that's sufficiently low-level to be a C replacement, but it offers a plethora of mechanisms for abstraction. Of all the new languages billed as "systems languages", Clay is the only one that excites me. It can also "quack like a duck" (to put it in TFA terms). [1] http://claylabs.com/clay/ http://claylabs.com/clay/

MatthewPhillips 14y ago
I can't find any kind of getting started guide, and most of the libs have no documentation at all.
- qznc 14y ago
  I found a "Clay for C++ programmers" tutorial https://github.com/jckarter/clay/wiki/Clay-for-C---programmers https://github.com/jckarter/clay/wiki/Clay-for-C---programme...
ponce 14y ago
May I mention Nimrod [1]? It has a lot going for it when it comes to meta-programming. [1] http://nimrod-code.org/ http://nimrod-code.org/

MatthewPhillips 14y ago

So what's the catch when it comes to D? If I can use all C libs easily, and get some more palpable syntax, what am I missing exactly and why haven't people transitioned to it?

pretoriusB 14y ago
The article is not about using C libs, is about C programs using you (your D code, that is). That said, the problem with D: not enough marketing plus the two compilers / 2 libraries issue. And it's close to C++ complexity enough to not let the scripting guys feel welcome (like they do in Go).

frujka 14y ago

it's probably an interesting read except that being hosted on google's blogspot means it only display blank pages due to content being embedded in some javascript shenanigans. Even a screenshot would be better than this. Please move on to some actual html website that just works or rehost the content elsewhere before posting it here. Thanks.

eropple 14y ago
I used to be one of These Guys who never used JavaScript, too, but honestly? It's 2012 and JavaScript is a thing pretty much everywhere worth targeting. Demanding that somebody cater to your microsegment is a little much.
- vq 14y ago
  Although I'm the kind of guy who usually have JS disabled I could be made to see your point. But this page is actually a good example of why I run with JS disabled. It magically positioned links and boxes over the scrolling bar and caused my initial attempt at scrolling down to navigate away from the page. Disabling JS usually have the effect of doing away with such nonsense but here that is no option. This time it was worth the hassle since it was an interesting article, but I definitely prefer it when JS is optional.
nnq 14y ago
> rehost the content elsewhere before posting it here Google's blogspot allows you to do normal html blogs (maybe now they've made this their "default", but still, it's OP's choice to have it this way)
sublimit 14y ago
My attempts to read this on my phone were futile. Every time I tried to scroll, it just jumped back to the top immediately.

dschiptsov 14y ago

This is called FFI.) And yes, a language with which you cannot dload and call any shared library you wish is, how to say, handicapped language. NodeJS, Java to name a few. Look what they do due to inability to call, say, libpq.so.) But of course, re-monkey-patching everything from scratch by amateurs coders is much better idea. And, you know, you could create an entire "standard" with hundreds of pages of specification and then push your bloatware - JDBC.

pjscott 14y ago
I believe his thesis was actually that most languages are handicapped because they can not easily be called via the usual "load a .so file and call a function" ABI. Java, for instance, would be a more useful language to me if I could easily create Python bindings for useful Java libraries like Lucene, without needing to use something like Jython that runs on the JVM.
- amirouche 14y ago
  tl;dr: use JNI through pyjnius of jpype. You can run Python 2.5 on lastest Jython, then you have access to JVM libs and the JVM libs to Python libs 2.5 compatible. Calling JVM from CPython is actually possible: - with pyjnius from kivy fame built with cython it is still in development, AFAIK it is missing the possibility to call Python code from Java code aka. submitting a Python "callback" to a Java method, I think it is called IoC pattern and Hollywood principle «Don't call me, I call you». jnius is used in Kivy to make it possible to call Android libs. As an example, I created Python bindings of Blueprints with it. Blueprints is an abstraction library for graphdb like Neo4j, OrientDB, Titan, available here https://github.com/Printemps/python-blueprints/ https://github.com/Printemps/python-blueprints/. Tutorial: «JavaClass = autoclass('path.to.JavaClass'); JavaClass.JavaMethod(python_object)». Those bindings are merely Pythonification but they are gotchas in types conversions. - There is also jpype which looks like an established solution used by Neo4J fame to make it possible to embed Neo4J embedded into CPython. I choosed jnius because it's faster; Those are funny benchmarks in log scale: https://docs.google.com/spreadsheet/ccc?key=0Ajc7MnpSjCokdGtsME01MnVmNnhid1ZhT210ZUt4WEE#gid=0 https://docs.google.com/spreadsheet/ccc?key=0Ajc7MnpSjCokdGt...
- 14y ago

pcwalton 14y ago

Freestanding, ABI-level interoperability with C is planned for Rust, but is not yet implemented. There are two main reasons why being able to be called with dlsym() was less important for us than the current setup: 1. M:N scheduling. We wanted high scalability with hundreds of thousands or millions of tasks, which means we need the capability to have our own scheduler. OS threads aren't lightweight enough for this, especially on Windows, where over 80% of our Firefox users are. Having our own scheduler means we need our own ABI. We would like to make the scheduler optional, but getting it working was a higher priority than making it optional. 2. Safety. On many platforms (basically, anything but MSVC on Windows), the C stack is not bounds-checked; either there is no guard page or the guard page is unreliable because you can overshoot it. This is not acceptable for our Rust code, since stack overflows are a frequent source of exploitable security vulnerabilities for us. So we do our own stack management with a custom ABI that performs stack checks at every function entry, freeing us from this source of pain on all operating systems. Again, this is planned to be optional, but it was more important at this early stage for us to demonstrate advantages over C than for us to duplicate its ABI exactly. Many libraries, however, would prefer direct interoperability with C to M:N scheduling or stack overflow safety. So we plan to provide a mechanism for Rust code to be called from C without having to set up a scheduler or perform stack checks. It's not implemented yet, but it's very much part of the current design, and I would be thrilled to see it happen.

lispertoascheme 14y ago

The state of programmers trying to avoid coding in the language their system is coded in. C.

martincmartin 14y ago

For example, Java and C# cannot walk like a duck, not even if that is what it would take to save themselves from drowning. Therefore, they are absolutely unsuitable as system languages. Hadoop held the world record for sorting a Terrabyte for a while. If sorting large amounts of data as fast as possible isn't a system's programming task, then I'm a three-headed monkey.

eswangren 14y ago
Yes, because "must sort quickly" is the only important requirement in a systems level language.