15 ms·

The search for easier safe systems programming

• pjmlp 2y ago
  This is a common trend among D, Chapel, Vale, Hylo, ParaSail, Haskell, OCaml, Swift, Ada, and now June. While Rust made Cyclone's type system more manageable for mainstream computing, everyone else is trying to combine the benefits of linear/affine type systems, with the productivity of automated resource management. Naturally it would be interesting to see if some of those attempts can equally ping back into Rust's ongoing designs.
  • valenterry 2y ago
    I think the resource management issue is mostly solved in practice when using pure functional programming. However that comes with performance drawbacks (or at least unpredictable/unreliable performance) which creates the need for languages like Rust. It's great to see the progress in those languages as well.
    • eru 2y ago
      > I think the resource management issue is mostly solved in practice when using pure functional programming. Well, in the sense that garbage collectors solve memory management. But pure functional programming in the Haskell sense doesn't really manage file handles for you, or database connections. You could have more predictable performance in Haskell, if you didn't have to deal with mandatory laziness for some functions. (Basically, in this hypothetical variant of Haskell, by default functions would be safe to be evaluated in any order, strict or lazy or whatever. If you want functions that need to be evaluated lazily, then you'd need to declare that; just like today you already need to declare that your function might have side effects.) The compiler would then be free to re-order evaluation a lot more, and pick predictable, fast performance.
      • valenterry 2y ago
        > Well, in the sense that garbage collectors solve memory management. But pure functional programming in the Haskell sense doesn't really manage file handles for you, or database connections. No, I was actually talking about the latter and not about memory management at all. In Scala we even achieve this on the library level. (e.g. https://zio.dev/reference/resource/ https://zio.dev/reference/resource/ which also compares the existing problematic try/catch with its own approach) And despite Scala being strict, the GC still makes it unsuitable for many cases where you just need something like Haskell.
      • mrkeen 2y ago
        > If you want functions that need to be evaluated lazily, then you'd need to declare that Not a sensible default. You cannot ask a strict function to behave lazily. But you can ask a lazy function to behave strictly.
        • throwaway17_17 2y ago
          I am unsure where the proof, or even supporting argument, would come from for such a statement. The literature is lengthy and deep, but I would sum up the current understanding and most popular argument against your stated position by citing Robert Harper (who I admit is notoriously anti-Laziness by default) in his commentary to the 2nd Edition to Practical Foundations of Programming Languages: "The problem is the very idea of a lazy language, which imposes a ruinous semantics on types. An eager semantics supports the expected semantics of types (for example, the natural numbers are the natural numbers) and, moreover, admits definition of the lazy forms of these types using suspension types (Chapter 36). A value of a suspension type is either a value of the underlying type, or a computation of such a value that may diverge. There being no representation of the eager types in a lazy language, it follows that, eager languages are strictly more expressive than lazy languages." [1] This paragraph is then followed by a link to Harper's "PCF by Value" supplement/errata to the above referenced PFPL, where such a statement is reinforced. [2] [1] - https://www.cs.cmu.edu/~rwh/pfpl/supplements/commentary.pdf https://www.cs.cmu.edu/~rwh/pfpl/supplements/commentary.pdf [2] - https://www.cs.cmu.edu/~rwh/pfpl/supplements/pcfv.pdf https://www.cs.cmu.edu/~rwh/pfpl/supplements/pcfv.pdf
  • eru 2y ago
    Agreed. (Though I wouldn't exactly call Haskell a 'systems programming language'. At least not in the sense the article uses it.) Yes, linear / affine types and uniqueness types can give you a lot of control over resources, while also allowing you to preserve functional semantics. I would like to see a version of Haskell that doesn't just track IO, but also track totality. Ie unless your function is annotated with a special tag, it has to be guaranteed to return a value in finite time, ie it has to be total. If you tag it with eg 'Partial', you can rely on laziness. That's very similar to how functions in Haskell can't do any IO, unless they are tagged with 'IO'. (I know that Haskell doesn't see 'IO' as a tag or annotation. But it behaves like one in the sense I am using here.)
    • Y_Y 2y ago
      Although I complain about them in a sibling comment, Agda and Idris do meet the criterion of "Haskell-like with totality checking".
      • eru 2y ago
        Yes. But I would want something a bit lighter-weight than them for this job. I'd want just barely more machinery and conceptual overhead than current Haskell has.
        • naasking 2y ago
          LiquidHaskell?
    • skybrian 2y ago
      “Finite time” isn’t much of a guarantee. A million years is finite. Totality checking is useful in language for doing proofs, where you need a guarantee that a function always returns a value even though you never run it. In languages for doing practical calculations, a progress dialog and a cancel button are more useful than a totality guarantee. It should be easier to make complex, long-running calculations safely cancellable. (Still true with laziness, though it changes a bit. At some point you will ask for a value to be calculated.)
  • Y_Y 2y ago
    Haskell is my favourite language ever, and the (vanilla) type system is a joy to use. Inevitably though I find myself wish for some dependent typing. Agda and Idris aren't quite there yet imho, but there's definitely a need for something that's at least as easy to work with as Haskell but a little more powerful (and ideally not some arcane GHC megahack).
    • valenterry 2y ago
      Same here, using Scala in a Haskell-like way and while Scala has more dependent typing than Haskell (ignoring liquid extension), it's still not sufficient and practical. I think we are quite far from having fully featured dependent type systems in a mainstream language. Maybe typescript comes closest so far.
  • arka2147483647 2y ago
    The problem for systems/low-level programming is that you want high-performance, and manual-control of resource management. As such, automated resource management can often look like a problem, instead of a feature. I think there is a deeper disconnect here between the language designers and the programmers in the trenches.
    • winternewt 2y ago
      Yes if I read the article right, every object is being allocated on the heap. That is a no-go for systems programming as far as I'm concerned.
      • christophilus 2y ago
        I read it the same way you did. But, I’d be really surprised if there was no stack allocation in the language, given the author’s experience.
      • DanielHB 2y ago
        My gut feeling agrees with you but I would really like more detailed reasons why this is the case. Is memory fragmentation that big of an issue? Are heap allocations more expensive somehow (even if memory is not fragmented yet)? Is there something else? Does re-arranging memory in the heap makes performance unpredictable like GC languages?
        • alchemio 2y ago
          Memory allocation is slow and undeterministic in perf. Some allocations also require a global lock on the system level. It’s also a point of failure if the allocation doesn’t succeed, so there’s an extra check somewhere. Furthermore if every object is a pointer you get indirection overhead (even though small but existent). Deallocation as well incurs an overhead. Without a compacting gc you run into memory fragmentation which further aggravate the issue. All of this overhead can be felt in tight loops.
          • 2y ago
  • brabel 2y ago
    A few other languages I think deserve a mention: * Carp: https://github.com/carp-lang/Carp https://github.com/carp-lang/Carp * Nim: https://nim-lang.org/ https://nim-lang.org/ * Zig: https://ziglang.org/ https://ziglang.org/ * Austral: https://borretti.me/article/introducing-austral https://borretti.me/article/introducing-austral
• cornholio 2y ago
  Seems like the wrong problem to solve. "Systems programming" is hard, and should be hard, for reasons unrelated to the programming language used. Something like Rust which forces you to constantly reevaluate your design before you even press the compile button, is ideal. What's really lacking are safe, easy, strong typed general purpose languages that can leverage AOT compilation for high performance and static analysis etc. A language with the learning curve of Python, near C performance and strong safety guarantees of Rust. There is nothing that suggests to me such a language would not be possible. Swift and C# come closest but they are warped by their respective corporate overlords.
  • farresito 2y ago
    I think that's what they are trying to achieve with Mojo.
    • sa-code 2y ago
      Agreed, although the language still feels like a tech demo. They don't even have a package manager yet. I'm curious and hopeful to see what it looks like in a year
      • mirekrusin 2y ago
        They don't yet have `class` support.
        • sa-code 2y ago
          They have their own version called struct which works in a very similar way. I feel like this should be mentioned when saying that `class` support didn't yet exist
          • mirekrusin 2y ago
            Yes, you're right and struct is _the_ interesting part about mojo and is enough to write interesting things in it. Floodgates will open though once class is also supported.
• taosx 2y ago
  I don't know why but Rust's syntax just nails it for me. The more I use it the more I appreciate it. I see many projects that diverge from Rust's syntax while being inspired by it. Why ?
• kkukshtel 2y ago
  Seeing Mads Torgersen on the list of collaborators for this made me take this seemingly random blog post 100x more seriously.
• nercury 2y ago
  struct Node<'a, 'b, 'c> { data1: &'a Data data2: &'b Data data3: &'c Data } Wow. It's like teaching C++ and starting from SFINAE. Or C# and starting from type parameter constraints. Please think of a real-world examples when teaching stuff. I am very eager to see the program a beginner would need to write that requires: 1) references in a struct; 2) 3 separate lifetime parameters for the same struct.
• deleted 2y ago
  [deleted]
• pron 2y ago
  > Rust's focus on embedded and system's development is a core strength. June, on the other hand, has a lean towards application development with a system's approach. This lets both co-exist and offer safe systems programming to a larger audience. I think this is a mistake, both on June's part and on Rust's. All low-level languages (by which I mean languages that offer control over all/most memory allocation) inherently suffer from low abstraction, i.e. there are fewer implementations of a particular interface or, conversely, more changes to the implementation require changes to the interface itself or to its client. This is why even though writing a program in many low-level languages can be not much more expensive than writing the program in a high-level language (one where memory management is entirely or largely automatic), costs accrue in maintenance. This feature of low-level programming isn't inherently good or bad -- it just is, and it's a tradeoff that is implicitly taken when choosing such a language. It seems that both June and Rust try to hide it, each in their own way, Rust by adopting C++'s "zero-cost abstraction approach", which is low abstraction masquerading as high abstraction when it appears as code on the screen, and June by yielding some amount of control. But because the tradeoff of low-level programming is real and inescapable, ultimately (after some years of seeing the maintenance costs) users learn to pick the right tradeoff for their domain. As such, languages should focus on domains that are most appropriate for the tradeoffs they force, while trying to aim for others usually backfires (as we've seen happen with C++). Given that ultimately virtually all users of a low level language will be those using it in a domain where the low-level tradeoff is appropriate -- i.e. programs in resource-constrained environments or programs requiring full and flexible control over every resource like OS kernels -- trying to hide the tradeoff in the (IMO) unattainable hope of growing the market beyond the appropriate domain, will result in disappointment due to a bad product-market fit. Sure, it's possible that C++'s vision of broadening the scope of low-level programming was right and it's only the execution that was wrong, but I wouldn't bet on it on both theoretical (low abstraction and its impact on maintenance) and empirical (for decades, no low-level languages have shown signs of taking a significant market share from high-level languages in the applications space) grounds. Trying to erase tradeoffs that appear fundamental -- to have your cake and eat it -- has consistently proven elusive.
  • gpderetta 2y ago
    C++ managed to get market share from C by being both as low level and higher level. It is true that it hasn't happened again since, as other higher level languages took market share from C++ for applications, but didn't replace it for lower level stuff. Still is conceivable that another language might do to C++ what it did to C.
• pjc50 2y ago
  This seems to be an "arena" or "pool" allocation approach. Conceptually quite a mature technique, but this adds the benefit of statically checking against pool lifetime? Probably works quite well for systems programming, where things are either "live forever", "reallocate within some pool" (thread handles, file descriptors, etc), or "transient" (for the lifetime of a system call or similar).
• samuell 2y ago
  The discussion of grouped lifetimes reminds me of the principles of Flow-based programming (without the visual part), where one main idea is that only one process owns a data packet (IP) at a time. My own experience coding in this style [1] has been extremely reassuring. You can generally really safely consider only the context of each process at a time, since there aren't even any function calls between processes, only data sharing. This meant for example that I could port a PHP application that I had been coding on for years, fighting bugs all over, into a flow-based Go application in two weeks, with a perfectly development time perfectly linear to the number of processes. I just coded each processes in the pipeline one by one, tested them and continued with the next. There were never any surprises as the application grew, as the interactions between the processes are just simple data sharing which can't really cause that much trouble. This is of course a radically different way of thinking and designing programs, but it really holds some enormous benefits. https://github.com/rdfio/rdf2smw/blob/master/main.go#L58-L150 https://github.com/rdfio/rdf2smw/blob/master/main.go#L58-L15...
  • mst 2y ago
    I shall have a read through https://github.com/flowbase/flowbase https://github.com/flowbase/flowbase later, thank you.
• anonymoushn 2y ago
  This lifetimes thing is maybe not even a top 3 mistake Rust makes. I hope successor languages can have a metaprogramming system that is less dreadful than proc macros, the ability for users to write libraries that are generic over user-provided writers, readers, and allocators, and the ability to bubble up errors from functions that call fallible functions from 2 different libraries without writing your own huge struct definition every time. It may also be nice if constructs that don't cause memory accesses and only ever do the correct thing or crash on the target cpu (such as integer division or pshufb without an address operand on any intel chip ever) were not unsafe. Placing "Well, LLVM says this arithmetic operation is UB and we won't bother to fix it" and "What if one day there's an x64 chip that does something other than crash if it encounters instructions from ISA extensions it does not have?" into the same bucket as "playing with raw pointers" is a bit weird.
  • anonymousdang 2y ago
    [flagged]
  • conaclos 2y ago
    proc_macros are so terrible and propagate attributes everywhere. It would be so much easier to write derive macros with reflection support. This would also avoid fragile code generation in some cases.
    • estebank 2y ago
      Proc macros in Rust are objectively bad in a number of ways (limited to only dealing with tokens/no type system access, hard for newcomers to grasp and write, force a lot of attributes to be written to annotate specific items, etc.) but they exemplify "worse is better" perfectly. They are incredibly powerful and allow people to build amazing abstractions that are in some cases best in class. We want to have something better, but there isn't a pressing need to rush a replacement.
  • estebank 2y ago
    Calling them mistakes is actively ignoring that there are design constraints and decisions that shape what the language looks and acts like. And that's without getting into "there are only so many hours in a day, a limited amount of people, so not everything is getting done at the same speed, if at all". It's the first time I hear your second point about intrinsics, for example, and it seems like something that could be done relatively straightforwardly, but introducing "inconsistency" for the benefit of skipping a single unsafe block sounds like the kind of RFC thread I wouldn't want to be a part of.
• osigurdson 2y ago
  So many comparisons to Go and C# on this thread. While at some level of abstraction, all languages are the same, comparing GCed languages to non-GC languages doesn't make sense in my opinion. Rust would have never been made if the creators were fine with Java.
  • pjmlp 2y ago
    The people that don't suffer from GC phobia are aware how to write Go and C# code that makes use of C like features, while taking advantage of their productivity in non critical code paths. Rust's ideal case are domains where allocation management is strictly controlled.
    • osigurdson 2y ago
      >> Rust's ideal case are domains where allocation management is strictly controlled What domain's are those? Do you mean just embedded or perhaps Firefox? Please elucidate.
      • pjmlp 2y ago
        Stuff that falls under high integrity computing certification where dynamic memory allocations is usually forbidden, GPGPU where memory primitives make quite hard to have automatic memory management as they are host controlled, specific kinds of drivers, or special VM flavours like eBPF. Everything else is a matter how much one wants to involve themselves into politics of the right way to do memory management, versus get to a compromise for more safer infrastructure.
• bjourne 2y ago
  How would the language guarantee no use-after-free scenarios? let x = new Foo() if (y == 1) { recycle x } println(x)
  • flohofwoe 2y ago
    I guess that's what the 'copy count' is for. After 'recycle x', the copy-count for the reference 'x' would be 0, so it's invalid to use. That would be trivial to do as a runtime check, don't know if the compiler is smart enough to catch such things at compile time. Also not sure if the idea of a 'copy count' is better than a 'generation counter' which also provides runtime protection against dangling access, e.g. see https://floooh.github.io/2018/06/17/handles-vs-pointers.html https://floooh.github.io/2018/06/17/handles-vs-pointers.html
• mightyham 2y ago
  I'm not very well versed in Rust, but isn't it possible to implement this sort of checked arena allocation in rust using lifetimes? Something like slotmap (https://docs.rs/slotmap/latest/slotmap/ https://docs.rs/slotmap/latest/slotmap/) except all of the pointers/keys have their lifetime tied to the arena/pool/map.
• netbioserror 2y ago
  For use cases which aren't bare-metal embedded: 1) Prefer the stack wherever possible. 2) RC. 3) Unique pointers by default unless explicitly noted otherwise. 4) Immutable arguments and returns by default unless explicitly noted otherwise. There's a language that does this all already, and it's called Nim.
• estebarb 2y ago
  I don't get why people are scared of GC. After working in embedded software, using mostly C, it was evident from profiling that C programs spend a lot of time allocating and releasing memory. And those programs usually release memory in place, unlike GC languages where the release is deferred, done in parallel, and with time-boxed pauses. Most "systems" programs use worse memory management strategies than what a modern GC actually offers. Sure, some devices require using static memory allocations or are quite restricted. But a lot of other "system programming" targets far more capable machines.
  • flohofwoe 2y ago
    > that C programs spend a lot of time allocating and releasing memory ...then this code didn't get the point of manual memory management, which is to minimize allocations and move them out of the hot path. Unfortunately a lot of C and C++ code is still stuck in a 1990's OOP model where each individual object is explicitly constructed and destroyed, and each construction and destruction is linked to an alloc/free call. If you do this (or reach for reference counting, which isn't much better), a GC is indeed the better choice.
    • estebarb 2y ago
      Moving out the allocations/releases out the hot path seems too much like a GC. Also, not all programs can be easily optimized moving the allocations: databases are an example, usually you don't know how much rows are you going to touch until you process the query.
      • flohofwoe 2y ago
        Tigerbeetle is famously a database that doesn't do dynamic memory allocation at all (after startup): https://tigerbeetle.com/blog/a-database-without-dynamic-memory https://tigerbeetle.com/blog/a-database-without-dynamic-memo...
      • Nevermark 2y ago
        Technically, GC can be part of manual memory management. The "manual" doesn't mean any algorithm is ruled out - it just means the developer mindfully chooses the right algorithm for the right situation. As apposed to the language, compiler or system always making those decisions.
• aetherspawn 2y ago
  Better systems programming is model based design and automatic code generation, period. It is the be-all and end-all that will make those scenes in Star Trek where they alter some core starship system programming in 2 minutes flat without mistakes actually plausible.
• zozbot234 2y ago
  > Effectively, this would mean that a data structure, like a linked list, would have a pointer pointing to the head which has a lifetime, and then every node in the list you can reach from that head has the same lifetime. Right, isn't that what GhostCell and its variants (QCell) are all about? This would be great if it led to a more elegant and principled implementation of that pattern, that could also end up being fully supported in Rust itself.
• keyle 2y ago
  Related, I really like the look of hare[1], sadly they don't seem to be interested in a cross-platform compiler. As I understand it, some of the design decisions have basically led it to be mostly a linux/bsd language. I personally love C. I think designing a language top-down is a poor approach overall, I prefer the bottom-up approach of the C-inspired for system languages, that aim to fix C rather than this is how the world should beeee! [1] https://harelang.org/ https://harelang.org/
• noelwelsh 2y ago
  Effect systems strike again! They've come up a few times recently on HN, and region-based memory management is another problem they can solve. This paper describes a type system that region-based memory management falls out of as a special case: https://dl.acm.org/doi/10.1145/3618003 https://dl.acm.org/doi/10.1145/3618003
  • mst 2y ago
    I was quite fascinated by koka's use of refcounting during compilation to be able to do June's 'recycle' trick automatically (i.e. if you consume or discard the last reference to something during an operation that returns a new 'something' it re-uses the memory of the now-defunct one).
• unaut 2y ago
  Amm, is this some kind of reinvention of C and C++ but from high-end backwards?