13 ms·

Giving up on wlroots-rs

283 points by Bl4ckb0ne 7y ago

ChrisRR 7y ago
Interesting to see someone's reasons for moving back from rust to C. Normally it's the other way round. Speaking as a C developer who has dabbled in Rust, It's always good to see the thoughts of someone who has spent a fair amount of time with the language who can give it a fair assessment.
steveklabnik 7y ago
A good post, thanks for sharing. > I want to make one (mildly controversial) thing clear: rewriting a library for the sake of only using Rust is not good engineering. Strong agree. > A literal rewrite of a project to Rust is not interesting, it’s not useful, it just causes churn and splits ecosystems. Time would be better spent either working with existing solutions that already have the effort put in to make them correct or to come up with new green-field projects. This... I'm not so sure about. It really depends on what your objective is. For example, if your goal is to learn, you're not going to cause churn, and you're not going to split ecosystems. Working on project you already know well is a good way to learn, because you can focus on the language, not the project. This also isn't exactly a re-write, in my mind. I mean it is, and it isn't. This is because... > The biggest problem when wrapping wlroots was defining the ownership model of the objects that wlroots exposes. The pain here isn't a re-write, it's an integration with an existing system. That's a good reason to not use something different! I also think this is interesting because it demonstrates something that's often said in discussions about Rust, but mostly in the abstract, and that's that Rust's rules influence the design of your system. It will guide you away from designs where the ownership of components is unclear. To many people, this is a benefit, but it can often cause struggles when learning. And, it can often cause struggles in situations like this: where you can't really re-write some external component to fit in the rules. That being said, there should be a way to do the ownership part that makes sense, but I don't know wlroots well enough to comment. That being said > Currently there is 11 THOUSAND lines of Rust in wlroots-rs. All of this code is just wrapper code, it doesn’t do anything but memory management. is also super legit. Managing this kind of thing is a pain. I can certainly understand not wanting to do it.
- ChuckMcM 7y ago
  I agree and was going to say the same. That there was a huge challenge in writing code to manage memory ownership, is not surprising because memory ownership is so fluid and adhoc in C and its derivatives. And when you are asked to make that ownership explicit, if the original designers hadn't been thinking about it, you get a lot of cases which they would not have considered. I would like to see a compositor written in Rust, I think it would make for a robust window system. I also know that starting from the position of "this will be written in Rust" would force the issue of memory ownership to the fore and result in a different architecture from the start.
  - ajross 7y ago
    Not a wayland, wlroots or rust expert, but dollars to donuts says that it's not "memory" management[1] at issue, but resource allocation on the other side of the graphics driver. Vertex arrays, textures, framebuffers, shaders et. al. all need some kind of allocation strategy, can in the modern world often be shared between process contexts, and they really don't fit well into the metaphors of either C or Rust. But where in C you can just do it anyway, Rust is likely to flip out over your wrapper object abstractions. [1] I mean sure, technically it is memory management, but you know what I mean.
    - klodolph 7y ago
      This is also something that causes some pain when using C++ with RAII. Logically it makes some amount of sense that you have e.g. a C++ wrapper for a texture in OpenGL, and it knows to call glDeleteTextures in its destructor. HOWEVER, the destructor can now only be called if the OpenGL context is active, and the texture may be deleted if the context is lost (which can happen). At some point anybody who's tried to wrap an OpenGL texture with a C++ class either decides to accept that RAII doesn't completely work here, or refactors it so that you manage something like handles to textures, which is a bit silly because you are really at that point managing handles to handles to textures.
      - ska 7y ago
lucideer 7y ago
Question (for any more knowledgable readers here) from someone with a somewhat shallow understanding of the topics discussed: Does this end up being primarily a negative reflection on the general structure of: (a) Rust (b) wl-roots (c) Wayland (d) all three (e) none of the above, it's merely the incidental reality of trying to write code that's compatible/usable across multiple language ecosystems and none of the 3 projects can do much to improve this situation.
- blattimwind 7y ago
  > Does this end up being primarily a negative reflection on the general structure of A core problem in this case is wlroots using a memory management paradigm that isn't easily modeled in Rust. This isn't unexpected per se, since C leaves MM entirely to the developer, while Rust is opinionated.
  - steveklabnik 7y ago
    I would say it's a bit more subtle than that. Rust can express this, but not safely. This is what the end bit is about. The question then becomes, is it worth it if it's largely unsafe? That's a complex question. Unsafe Rust still does give you a lot of advantages, namely that the checked constructs are still safe, even in an unsafe block. Unsafe Rust is slightly more annoying to write than safe Rust, and so that's a downside. It's also possible, and again, this is more in theory since I know nothing about wayland internals, that the safe abstraction was chosen to be a bit too low-level. That is, rather than trying to make the primitive operations safer, designing an external API you'd want users to use, rather than one defined in terms of some of the primitives, may make sense. This has a lot of pros and cons, as you'd imagine. And that's also more work to do.
    - timidger 7y ago
      Author here. Ignoring the social impetus in the Rust community to not use unsafe, I also don't feel like unsafe Rust is something I want to program in all the time. When I program in safe Rust I can be happy once it compiles because I can ignore all of the safety problems that come from C and C++. However in unsafe Rust not only is it much more difficult to express what I want syntatically (the lack of auto deref is very annoying, having to write (*base).value all the time gets very old) and semantically (there is no standard for the unsafe parts of the language - not so much a problem if only smallish parts of this usage is used (because once a standard comes out just that can be updated) but a problem if a whole program is written in it). Unsafe Rust is "good enough" to try to encode these abstractions but I would not use it over C or C++.
quietbritishjim 7y ago
I know very little about Rust or Wayland (I suppose I'm not the target audience) but I got lost very quickly here: > A Wayland “output” is the resource that represents a display device. Commonly this means it handles a computer monitor. This resource could disappear at any time in the life cycle of the application. This is easy enough to imagine: all it takes is a yank of the display’s power cord and the monitor goes away. Surely even if a physical monitor is connected from a computer, the object representing it doesn't instantly go away? If it literally got freed as soon as the user disconnected the monitor then any access to such an object would be dangerous as you could be accessing an object after it's freed, or even another valid display object that got allocated into that space in the mean time. Instead, I would expect an object in that situation to go into some error state, and even that might only be picked up when you perform certain operations. If that were the case, I don't really see how Rust lifetimes are a problem. Since this was the main summary of the problem for laypeople like me, it made the rest of the article quite hard to follow.
- nn3 7y ago
  Yes that's the real solution to the problem. If you have to access something in lots of code, don't make it go away asynchronously. There are many techniques to archive that. Keeping objects around in an error state is one easy way to do that. You never want memory management to be too fine grained. Even if he could implement the fine grained memory management it would likely be impossible to test all the corner cases.
- derefr 7y ago
  Picture a pointer to video memory. Or, simpler, picture a pointer to an SHM section that either side of the SHM IPC conversation can deallocate. From both sides’ perspective, that pointer is probably implemented as both an SHM section, but also an SHM pointer to the section, such that either side can set the SHM pointer to NULL, and then (if they managed to do that) proceed to tell the SHM infrastructure to unmap all mappings of the SHM section. When this sort of structure is used, there are certainly going to be guarantees in place (probably by using some sort of session/transaction functions that compare-and-swap an atomic SHM spinlock controlling the SHM pointer) such that the SHM pointer won’t go NULL [and the thing it points to won’t be deallocated] in the middle of either side writing to it. But those functions aren’t actually consuming the resource and spitting out a new temporary one (i.e. a “handle” to the resource, as the article’s author implemented in their Rust wrapper library originally.) Instead, they’re just functions that block either side from writing to the pointer as long as you’re in them—sort of like disabling interrupts in a critical section. How do you model the ownership of such a C-FFI-runtime-guarded IPC SHM volatile pointer-to-pointer, in Rust? Is there an idiomatic translation for it? Because this sort of thing comes up all the time in the context of kernel handles to buffers, and I would be surprised if the folks writing OSes in Rust haven’t hit on it before. IIRC, there’s an IPC abstraction called a ‘blackboard’ (sort of related to a tuple space) that is the generalization of this SHM model, so it might also help to ask how you’d model an IPC ‘blackboard’ in Rust.
herodotus 7y ago
> Way Cooler is a Wayland compositor that was written in Rust using wlc I know it is not easy, but I wish the author could have started with a paragraph that could help someone like me know whether or not the rest of the article would be something I would like to read. How about something like this (and of course I may some of the facts wrong, but I want to be as constructive as I can): "Wayland is a Windows manager developed as a better alternative to X-Windows. Way Cooler is a tiling Wayland manager, written in Rust, and designed to be easily extendible. In this article I describe my experience in trying to refactor it, for reasons that will be described below. I will also explain why, in certain instances, C was a better choice for this project than Rust."
- ghettoimp 7y ago
  Definitely. So often, whole articles and even much of the ensuing discussion will be using project names and acronyms that I don't know about. A sentence or two of context, or even just links going off to the project/acronym the first time it is used, are wonderfully useful for the wider audience.
  - stagger87 7y ago
    There are links in the first sentence!
- guelo 7y ago
  You weren't the audience of this article. Since he wrote it as a blog post on way-cooler.com he surely was expecting his audience to be people familiar with Way Cooler, Wayland and Rust. He doesn't have a responsibility to dumb it down for you. And anyway it only takes a few minutes for you to get the context. Which you did, good job.
  - deleted 7y ago
    [deleted]
  - dmix 7y ago
    So it's way-cooler.com's fault for not explaining what their library/website is about. Which unfortunately is par for the course on 99% of company blogs, which never explain what the company does without going onto the homepage (which even then is often confusing).
Diggsey 7y ago
I can't speak to every issue which the author might have encountered, but there is a better solution to the lifetime management problem than the two mentioned in the article. Instead of this: fn some_wlroots_callback(output_handle: OutputHandle, surface_handle: SurfaceHandle) { output_handle.run(|output| { surface_handle.run(|surface| { // maybe some more nested layers... }).unwrap() }).unwrap() } One can do this: fn some_wlroots_callback( ctx: CallbackContext, output_handle: OutputHandle, surface_handle: SurfaceHandle ) { let output = ctx.get(output_handle); let surface = ctx.get(surface_handle); } This is safe, because the lifetime of "output" and "surface" can be bound to the lifetime of the "ctx" (whose lifetime is controlled by the library: the library simply has to make sure that "ctx" is not accessible outside of a callback). edit: Realised you can't tell that there's an implicit lifetime in `CallbackContext` here: struct CallbackContext<'a> { ... }; OR type CallbackContext<'a> = &'a CallbackContextImpl;
- timidger 7y ago
  Author here. The problem with that design (which is a great design given what I presented in the article by the way!) is that it doesn't allow you to share handles across callbacks, which is mandatory to do anything interesting. I'm assuming here that you can't use the handles except for that callback context. If you can, then that presents a different problem. https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=1a010f34fdb6886d41145eecc2963280 https://play.rust-lang.org/?version=stable&mode=debug&editio... If you can own a context, even with a lifetime parameter it's possible to leak it using the Box api. That allows you to have access to a &'static CallbackContext. This will break that assumption that it only lives as long as the callback itself.
  - Arnavion 7y ago
    >If you can own a context, even with a lifetime parameter it's possible to leak it using the Box api. The `std::pin::Pin` API prevents you from doing this (callbacks would receive `Pin<&'_ mut CallbackContext>`). >The problem with that design (which is a great design given what I presented in the article by the way!) is that it doesn't allow you to share handles across callbacks But how does sharing handles between callbacks look like? Is it something like "callback FooCreated provided me a Foo handle" and "callback BarCreated requires me to have the Foo from the FooCreated callback" ? So your FooCreated callback needs to save the Foo handle somewhere that it can be reused later? If so, why is it not enough to have a `fn CallbackContext::set_foo_handle(&mut self, Foo)` (or `Pin<&mut self>` based on the above suggestion) ?
mcguire 7y ago
"A Wayland “output” is the resource that represents a display device. Commonly this means it handles a computer monitor. This resource could disappear at any time in the life cycle of the application. This is easy enough to imagine: all it takes is a yank of the display’s power cord and the monitor goes away. [Except it can't; it can only disappear between callbacks?] This is basically the exact opposite of the Rust memory model. Rust likes to own things and give compile-time defined borrows of that memory. This is runtime lifetime management that must be managed in some way." Something leads me to believe that there is something very wrong with the architecture of wlroots-rs. The output should be attached to a callback parameter or something, maybe? I don't know enough about Wayland to say, but something ain't right.
- timidger 7y ago
  Author here. This output is attached to a callback parameter, but you can takes this resource from the callback in C (because it's just a pointer you copy around) and use it in other callbacks. Eventually a "special" callback that will trigger to indicate that the data the pointer refer to will be cleaned up and you need to remove all of your references to that resource because otherwise they will be dangling.
scoutt 7y ago
> When the benefit at the end of the day is just so I don’t have to write C, that doesn’t really make it worth it. This line not only summarizes the content of the article, but I'm afraid it may also describe the actual situation of porting C/C++ code to Rust.
vectorEQ 7y ago
good ol C :). Thanks for sharing these experiences. Happy to see some example of the places where rust still falls a little short / is a bit too restrictive for system programming. a lot of people deny this, but there's plenty of situations where you just want plain old C for it's straightforwardness to implement your own design. if you want to do it the 'rust' way, you find yourself restricted in all kinds of ways. if you can live with those restrictions it's great, but sometimes you just get stuck. Good luck on the rebuild in C!
4bpp 7y ago
How does Rust handle these problems in the context of file I/O? I'm sure there is an axiomatically idiomatic (official) implementation of it as part of the language distribution, and at the same time file handles seem conceptually very similar to display handles and should have similar failure patterns (space can run out, a disk can fail, a plug-and-play disk can be yanked out at any time...).
- steveklabnik 7y ago
  That is all here: https://doc.rust-lang.org/stable/std/fs/struct.File.html https://doc.rust-lang.org/stable/std/fs/struct.File.html These failure patterns are handled by each method, for example, pub fn open<P: AsRef<Path>>(path: P) -> Result<File> that Result will return an error if the file can't be opened, etc. Let's say you want to write some bytes, that's fn write(&mut self, buf: &[u8]) -> Result<usize> This also returns Result, so if you've opened the file, but the disk is now out of space, this will return an error, etc.
mempko 7y ago
I have had the view that safety is typically not the most important goal of a project (user satisfaction is). I have often said that safety can get in the way of writing software that is useful. It's great to see a real-life case with Rust from someone who obviously tried very hard but ultimately had safety get in the way of writing the software they wanted. Hopefully, Rust will evolve to interface with the "unsafe" world in a more ergonomic way.
simag 7y ago
Interesting post that show some current limitations of Rust, especially when not being able to control the design entirely. Rust's ownership system makes it harder to deal with the kind of dynamics wlroots exposes... It seems to me that the issue here is that there is no consensus (e.g library) on how Rust should deal with those issues, so if you want to just to provide bindings to a dynamic system you end up writing middle-ware that deals with that kind of dynamics in a Rust idiomatic fashion (taking advantage of ownership) instead of focusing on the task at hand. It somehow destroys Rust's promise of productivity. A Rust project of mine[1], in an early stage and currently paused, deals with those issues (probably in a way similar to the Handles described in the post) by providing Service smart pointers (i.e, Svc<T>) and a component-framework that helps deal with the dynamics of a service being unregistered (e.g, a monitor unplugged, or a dynamic library unloaded) ; that component framework helps you be sure that if your struct contains a Svc<Foo>, that service is still available when you're called, or the component reaches an invalid state. In general, it seems to be an area where Rust's ecosystem is still very early and would benefit from more input (such as this post) and consensus. Regarding the callback-hell issue, that "dehandle" macro looks very much like async/await, and it looks like it could be implemented either in terms of async/await (still unstable) or generators (even more unstable). Hopefully similar projects will be more likely to succeed as Rust and its ecosystem mature. [1] https://github.com/magnet/socrates-rs https://github.com/magnet/socrates-rs
Lowkeyloki 7y ago
Everything the author and the commentors here have said is completely legit. I'm not looking to start a war. And I completely acknowledge that what I'm about to say might be wrong as I haven't seen the author's code or the code of wlroots. But I have to wonder if the design decisions of wlroots itself may be dubious. If it's this hard to manage memory safely when you're trying to wrap the API in a language that demands you're kept accountable for memory safety....
ac130kz 7y ago
Does wlroots itself present good code quality or it is basically a horrible wrapper to support every Xorg use case?
- Sir_Cmpwn 7y ago
  wlroots is generally highly regarded in the Wayland community, both in terms of technical design and code quality. There's a reason that every Wayland project which started since wlroots has used wlroots, and those who didn't at first eventually rewrote their code to use wlroots.
  - AsyncAwait 7y ago
    I get the sense that most adopted wlroots because Wayland is a complex protocol. There's no alternative to wlroots so part of the reason everybody uses it is because there's no choice if you don't want to start from nothing.
    - Sir_Cmpwn 7y ago
      Wayland is a simple protocol. The complicated part is everything else, like graphics and input and X11 compatibility.
      - AsyncAwait 7y ago
        Right, but that still means that getting to a point most end-users would consider 'usable' is quite complex. Especially because the documentation around these things can be lacking. P.S. I like Wayland and use it everywhere for the record.
  - doubleunplussed 7y ago
    Is there any indication that mutter and kwin will rewrite to use wlroots? That would be glorious.
    - Sir_Cmpwn 7y ago
shmerl 7y ago
A pity wlroots itself isn't written in Rust.
- ddevault 7y ago
  C is the lingra franca of programming. By writing wlroots in C, we make it easy for a half dozen projects to make bindings to other programming languages. Rust is the only one that seems to have failed, and it's not surprising given the constraints of the language. wlroots brings together over a dozen different libraries and interfaces which are implemented in C. You'd have to repeat this process a dozen times over, running into the same problems which caused Timidger to abandon wlroots-rs, only more so. And for what? wlroots works great and didn't require shaving a thosuand yaks.
  - steveklabnik 7y ago
    I know you know this, but for the benefit of others, you're talking about the C ABI, not about the language itself. Rust can expose a C ABI as well, and you'd get all those same bindings benefits. > wlroots brings together over a dozen different libraries and interfaces which are implemented in C. This is a great argument that it should be written in C, though.
  - shmerl 7y ago
    > By writing wlroots in C, we make it easy for a half dozen projects to make bindings to other programming languages. Rust is the same in this regard, you can bind it to any other language. And it doesn't have the horrible downsides of C. So why not use it? > wlroots brings together over a dozen different libraries and interfaces which are implemented in C. That's an issue as well, and this of course runs deep. But starting somewhere should be still possible.
    - Sir_Cmpwn 7y ago
      >So why not use it? Because I don't like it, and I do like C. Rust is not the second coming of Christ, it's a programming language and has many shortcomings and tradeoffs.
kccqzy 7y ago
This code fn some_wlroots_callback(output_handle: OutputHandle, surface_handle: SurfaceHandle) { output_handle.run(|output| { surface_handle.run(|surface| { // maybe some more nested layers... }).unwrap() }).unwrap() } is just screaming continuation monad. You see the same code pattern in early Node.js code (sometimes leading to callback hell). In the JavaScript world, the problem was solved using Promises, and then async/await syntax. But more fundamentally, this is an instance of the continuation monad at work. The continuation monad transformer is defined as newtype ContT r m a = ContT { runContT :: (a -> m r) -> m r } and is nothing more than just a function that takes a callback. If this were Haskell, one could just write stuff = runContT $ do output <- ContT (run outputHandle) surface <- ContT (run surfaceHandle) -- and then maybe some more nested layers -- etc Granted, continuation code can easily be misused to produce an incomprehensible mess in Haskell (its full generality can be compared with goto), but with Rust's FnOnce trait, the scope for misuse is considerably reduced.
- pjmlp 7y ago
  JavaScript and Haskell get to enjoy the productivity of using a tracing GC though.
  - kccqzy 7y ago
    This doesn't really have much to do with the runtime. Certainly with a GC, programming would be easier, but I'm really talking about abstractions within the language. Rust already has language-integrated support for the Result monad. Rust doesn't have a general `do` syntax, but it has `?` which has made programming with the Result monad much easier. Can we think about the continuation monad and arrive at a new syntax that can ease this style of programming?
    - uryga 7y ago
      i don't really know Rust apart from some curious onlooking, but afaik expressing monads (i.e. the Monad typeclass) in Rust is tricky – something to do with the lifetimes of closures and the objects they close over, i think. ([Idiomatic Monads in Rust] probably touches on that). it might be possible to just build it into the language a la Result or async/await though. [Idiomatic Monads in Rust] https://varkor.github.io/blog/2019/03/28/idiomatic-monads-in-rust.html https://varkor.github.io/blog/2019/03/28/idiomatic-monads-in...
josteink 7y ago
Personally I think Rust is a great language. That said it may not be great for everyone nor a great fit for every problem. Sometimes trying and admitting failure is a perfectly rational and valid option.
pcwalton 7y ago
I may be missing something, but what's wrong with the first example? The fact that you can leak an object shouldn't really matter for memory safety if it's just a handle to an object that the server manages. Yeah, it could go wrong, but it won't be unsafe. Object handles are essentially file descriptors, right? In general I think there's a tendency to overcomplicate safety features in Rust. The solution to an overly-complicated system isn't to throw the whole notion of safety out the window: it's to look at exactly what the complexity is buying you. If intricate combinations of Rust features are one extreme of the safety spectrum and C is the other extreme, there's frequently a happy design medium somewhere in the middle. Edit: Looks like oconnor663 over on Reddit had a similar but more specific proposal, which probably works: https://www.reddit.com/r/rust/comments/biq864/comment/em2kipe https://www.reddit.com/r/rust/comments/biq864/comment/em2kip...
- Arnavion 7y ago
  Yes, the same concept is discussed in https://news.ycombinator.com/item?id=19779243 https://news.ycombinator.com/item?id=19779243
newnewpdro 7y ago
I only skimmed the article, but the description of outputs and their asynchronous lifecycle making things complicated just made me think of what's become known as the ECS pattern in game development. Your outputs sound just like entities in a game. They come and go as they please, and you often have multiple references to them from myriad places since entities may interact with many parts of the game. I suspect if you investigated the established techniques for implementing ECS-style games in rust, you might find some simpler and established solutions for your troubles.

283 points by Bl4ckb0ne 7y ago

Interesting to see someone's reasons for moving back from rust to C. Normally it's the other way round. Speaking as a C developer who has dabbled in Rust, It's always good to see the thoughts of someone who has spent a fair amount of time with the language who can give it a fair assessment.

steveklabnik 7y ago

A good post, thanks for sharing. > I want to make one (mildly controversial) thing clear: rewriting a library for the sake of only using Rust is not good engineering. Strong agree. > A literal rewrite of a project to Rust is not interesting, it’s not useful, it just causes churn and splits ecosystems. Time would be better spent either working with existing solutions that already have the effort put in to make them correct or to come up with new green-field projects. This... I'm not so sure about. It really depends on what your objective is. For example, if your goal is to learn, you're not going to cause churn, and you're not going to split ecosystems. Working on project you already know well is a good way to learn, because you can focus on the language, not the project. This also isn't exactly a re-write, in my mind. I mean it is, and it isn't. This is because... > The biggest problem when wrapping wlroots was defining the ownership model of the objects that wlroots exposes. The pain here isn't a re-write, it's an integration with an existing system. That's a good reason to not use something different! I also think this is interesting because it demonstrates something that's often said in discussions about Rust, but mostly in the abstract, and that's that Rust's rules influence the design of your system. It will guide you away from designs where the ownership of components is unclear. To many people, this is a benefit, but it can often cause struggles when learning. And, it can often cause struggles in situations like this: where you can't really re-write some external component to fit in the rules. That being said, there should be a way to do the ownership part that makes sense, but I don't know wlroots well enough to comment. That being said > Currently there is 11 THOUSAND lines of Rust in wlroots-rs. All of this code is just wrapper code, it doesn’t do anything but memory management. is also super legit. Managing this kind of thing is a pain. I can certainly understand not wanting to do it.

ChuckMcM 7y ago
I agree and was going to say the same. That there was a huge challenge in writing code to manage memory ownership, is not surprising because memory ownership is so fluid and adhoc in C and its derivatives. And when you are asked to make that ownership explicit, if the original designers hadn't been thinking about it, you get a lot of cases which they would not have considered. I would like to see a compositor written in Rust, I think it would make for a robust window system. I also know that starting from the position of "this will be written in Rust" would force the issue of memory ownership to the fore and result in a different architecture from the start.
- ajross 7y ago
  Not a wayland, wlroots or rust expert, but dollars to donuts says that it's not "memory" management[1] at issue, but resource allocation on the other side of the graphics driver. Vertex arrays, textures, framebuffers, shaders et. al. all need some kind of allocation strategy, can in the modern world often be shared between process contexts, and they really don't fit well into the metaphors of either C or Rust. But where in C you can just do it anyway, Rust is likely to flip out over your wrapper object abstractions. [1] I mean sure, technically it is memory management, but you know what I mean.
  - klodolph 7y ago
    This is also something that causes some pain when using C++ with RAII. Logically it makes some amount of sense that you have e.g. a C++ wrapper for a texture in OpenGL, and it knows to call glDeleteTextures in its destructor. HOWEVER, the destructor can now only be called if the OpenGL context is active, and the texture may be deleted if the context is lost (which can happen). At some point anybody who's tried to wrap an OpenGL texture with a C++ class either decides to accept that RAII doesn't completely work here, or refactors it so that you manage something like handles to textures, which is a bit silly because you are really at that point managing handles to handles to textures.
    - ska 7y ago

lucideer 7y ago

Question (for any more knowledgable readers here) from someone with a somewhat shallow understanding of the topics discussed: Does this end up being primarily a negative reflection on the general structure of: (a) Rust (b) wl-roots (c) Wayland (d) all three (e) none of the above, it's merely the incidental reality of trying to write code that's compatible/usable across multiple language ecosystems and none of the 3 projects can do much to improve this situation.

blattimwind 7y ago
> Does this end up being primarily a negative reflection on the general structure of A core problem in this case is wlroots using a memory management paradigm that isn't easily modeled in Rust. This isn't unexpected per se, since C leaves MM entirely to the developer, while Rust is opinionated.
- steveklabnik 7y ago
  I would say it's a bit more subtle than that. Rust can express this, but not safely. This is what the end bit is about. The question then becomes, is it worth it if it's largely unsafe? That's a complex question. Unsafe Rust still does give you a lot of advantages, namely that the checked constructs are still safe, even in an unsafe block. Unsafe Rust is slightly more annoying to write than safe Rust, and so that's a downside. It's also possible, and again, this is more in theory since I know nothing about wayland internals, that the safe abstraction was chosen to be a bit too low-level. That is, rather than trying to make the primitive operations safer, designing an external API you'd want users to use, rather than one defined in terms of some of the primitives, may make sense. This has a lot of pros and cons, as you'd imagine. And that's also more work to do.
  - timidger 7y ago
    Author here. Ignoring the social impetus in the Rust community to not use unsafe, I also don't feel like unsafe Rust is something I want to program in all the time. When I program in safe Rust I can be happy once it compiles because I can ignore all of the safety problems that come from C and C++. However in unsafe Rust not only is it much more difficult to express what I want syntatically (the lack of auto deref is very annoying, having to write (*base).value all the time gets very old) and semantically (there is no standard for the unsafe parts of the language - not so much a problem if only smallish parts of this usage is used (because once a standard comes out just that can be updated) but a problem if a whole program is written in it). Unsafe Rust is "good enough" to try to encode these abstractions but I would not use it over C or C++.

quietbritishjim 7y ago

I know very little about Rust or Wayland (I suppose I'm not the target audience) but I got lost very quickly here: > A Wayland “output” is the resource that represents a display device. Commonly this means it handles a computer monitor. This resource could disappear at any time in the life cycle of the application. This is easy enough to imagine: all it takes is a yank of the display’s power cord and the monitor goes away. Surely even if a physical monitor is connected from a computer, the object representing it doesn't instantly go away? If it literally got freed as soon as the user disconnected the monitor then any access to such an object would be dangerous as you could be accessing an object after it's freed, or even another valid display object that got allocated into that space in the mean time. Instead, I would expect an object in that situation to go into some error state, and even that might only be picked up when you perform certain operations. If that were the case, I don't really see how Rust lifetimes are a problem. Since this was the main summary of the problem for laypeople like me, it made the rest of the article quite hard to follow.

nn3 7y ago
Yes that's the real solution to the problem. If you have to access something in lots of code, don't make it go away asynchronously. There are many techniques to archive that. Keeping objects around in an error state is one easy way to do that. You never want memory management to be too fine grained. Even if he could implement the fine grained memory management it would likely be impossible to test all the corner cases.
derefr 7y ago
Picture a pointer to video memory. Or, simpler, picture a pointer to an SHM section that either side of the SHM IPC conversation can deallocate. From both sides’ perspective, that pointer is probably implemented as both an SHM section, but also an SHM pointer to the section, such that either side can set the SHM pointer to NULL, and then (if they managed to do that) proceed to tell the SHM infrastructure to unmap all mappings of the SHM section. When this sort of structure is used, there are certainly going to be guarantees in place (probably by using some sort of session/transaction functions that compare-and-swap an atomic SHM spinlock controlling the SHM pointer) such that the SHM pointer won’t go NULL [and the thing it points to won’t be deallocated] in the middle of either side writing to it. But those functions aren’t actually consuming the resource and spitting out a new temporary one (i.e. a “handle” to the resource, as the article’s author implemented in their Rust wrapper library originally.) Instead, they’re just functions that block either side from writing to the pointer as long as you’re in them—sort of like disabling interrupts in a critical section. How do you model the ownership of such a C-FFI-runtime-guarded IPC SHM volatile pointer-to-pointer, in Rust? Is there an idiomatic translation for it? Because this sort of thing comes up all the time in the context of kernel handles to buffers, and I would be surprised if the folks writing OSes in Rust haven’t hit on it before. IIRC, there’s an IPC abstraction called a ‘blackboard’ (sort of related to a tuple space) that is the generalization of this SHM model, so it might also help to ask how you’d model an IPC ‘blackboard’ in Rust.

herodotus 7y ago

> Way Cooler is a Wayland compositor that was written in Rust using wlc I know it is not easy, but I wish the author could have started with a paragraph that could help someone like me know whether or not the rest of the article would be something I would like to read. How about something like this (and of course I may some of the facts wrong, but I want to be as constructive as I can): "Wayland is a Windows manager developed as a better alternative to X-Windows. Way Cooler is a tiling Wayland manager, written in Rust, and designed to be easily extendible. In this article I describe my experience in trying to refactor it, for reasons that will be described below. I will also explain why, in certain instances, C was a better choice for this project than Rust."

ghettoimp 7y ago
Definitely. So often, whole articles and even much of the ensuing discussion will be using project names and acronyms that I don't know about. A sentence or two of context, or even just links going off to the project/acronym the first time it is used, are wonderfully useful for the wider audience.
- stagger87 7y ago
  There are links in the first sentence!
guelo 7y ago
You weren't the audience of this article. Since he wrote it as a blog post on way-cooler.com he surely was expecting his audience to be people familiar with Way Cooler, Wayland and Rust. He doesn't have a responsibility to dumb it down for you. And anyway it only takes a few minutes for you to get the context. Which you did, good job.
- deleted 7y ago
  [deleted]
- dmix 7y ago
  So it's way-cooler.com's fault for not explaining what their library/website is about. Which unfortunately is par for the course on 99% of company blogs, which never explain what the company does without going onto the homepage (which even then is often confusing).

Diggsey 7y ago

I can't speak to every issue which the author might have encountered, but there is a better solution to the lifetime management problem than the two mentioned in the article. Instead of this: fn some_wlroots_callback(output_handle: OutputHandle, surface_handle: SurfaceHandle) { output_handle.run(|output| { surface_handle.run(|surface| { // maybe some more nested layers... }).unwrap() }).unwrap() } One can do this: fn some_wlroots_callback( ctx: CallbackContext, output_handle: OutputHandle, surface_handle: SurfaceHandle ) { let output = ctx.get(output_handle); let surface = ctx.get(surface_handle); } This is safe, because the lifetime of "output" and "surface" can be bound to the lifetime of the "ctx" (whose lifetime is controlled by the library: the library simply has to make sure that "ctx" is not accessible outside of a callback). edit: Realised you can't tell that there's an implicit lifetime in `CallbackContext` here: struct CallbackContext<'a> { ... }; OR type CallbackContext<'a> = &'a CallbackContextImpl;

timidger 7y ago
Author here. The problem with that design (which is a great design given what I presented in the article by the way!) is that it doesn't allow you to share handles across callbacks, which is mandatory to do anything interesting. I'm assuming here that you can't use the handles except for that callback context. If you can, then that presents a different problem. https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=1a010f34fdb6886d41145eecc2963280 https://play.rust-lang.org/?version=stable&mode=debug&editio... If you can own a context, even with a lifetime parameter it's possible to leak it using the Box api. That allows you to have access to a &'static CallbackContext. This will break that assumption that it only lives as long as the callback itself.
- Arnavion 7y ago
  >If you can own a context, even with a lifetime parameter it's possible to leak it using the Box api. The `std::pin::Pin` API prevents you from doing this (callbacks would receive `Pin<&'_ mut CallbackContext>`). >The problem with that design (which is a great design given what I presented in the article by the way!) is that it doesn't allow you to share handles across callbacks But how does sharing handles between callbacks look like? Is it something like "callback FooCreated provided me a Foo handle" and "callback BarCreated requires me to have the Foo from the FooCreated callback" ? So your FooCreated callback needs to save the Foo handle somewhere that it can be reused later? If so, why is it not enough to have a `fn CallbackContext::set_foo_handle(&mut self, Foo)` (or `Pin<&mut self>` based on the above suggestion) ?

mcguire 7y ago

"A Wayland “output” is the resource that represents a display device. Commonly this means it handles a computer monitor. This resource could disappear at any time in the life cycle of the application. This is easy enough to imagine: all it takes is a yank of the display’s power cord and the monitor goes away. [Except it can't; it can only disappear between callbacks?] This is basically the exact opposite of the Rust memory model. Rust likes to own things and give compile-time defined borrows of that memory. This is runtime lifetime management that must be managed in some way." Something leads me to believe that there is something very wrong with the architecture of wlroots-rs. The output should be attached to a callback parameter or something, maybe? I don't know enough about Wayland to say, but something ain't right.

timidger 7y ago
Author here. This output is attached to a callback parameter, but you can takes this resource from the callback in C (because it's just a pointer you copy around) and use it in other callbacks. Eventually a "special" callback that will trigger to indicate that the data the pointer refer to will be cleaned up and you need to remove all of your references to that resource because otherwise they will be dangling.

scoutt 7y ago

> When the benefit at the end of the day is just so I don’t have to write C, that doesn’t really make it worth it. This line not only summarizes the content of the article, but I'm afraid it may also describe the actual situation of porting C/C++ code to Rust.

vectorEQ 7y ago

good ol C :). Thanks for sharing these experiences. Happy to see some example of the places where rust still falls a little short / is a bit too restrictive for system programming. a lot of people deny this, but there's plenty of situations where you just want plain old C for it's straightforwardness to implement your own design. if you want to do it the 'rust' way, you find yourself restricted in all kinds of ways. if you can live with those restrictions it's great, but sometimes you just get stuck. Good luck on the rebuild in C!

4bpp 7y ago

How does Rust handle these problems in the context of file I/O? I'm sure there is an axiomatically idiomatic (official) implementation of it as part of the language distribution, and at the same time file handles seem conceptually very similar to display handles and should have similar failure patterns (space can run out, a disk can fail, a plug-and-play disk can be yanked out at any time...).

steveklabnik 7y ago
That is all here: https://doc.rust-lang.org/stable/std/fs/struct.File.html https://doc.rust-lang.org/stable/std/fs/struct.File.html These failure patterns are handled by each method, for example, pub fn open<P: AsRef<Path>>(path: P) -> Result<File> that Result will return an error if the file can't be opened, etc. Let's say you want to write some bytes, that's fn write(&mut self, buf: &[u8]) -> Result<usize> This also returns Result, so if you've opened the file, but the disk is now out of space, this will return an error, etc.

mempko 7y ago

I have had the view that safety is typically not the most important goal of a project (user satisfaction is). I have often said that safety can get in the way of writing software that is useful. It's great to see a real-life case with Rust from someone who obviously tried very hard but ultimately had safety get in the way of writing the software they wanted. Hopefully, Rust will evolve to interface with the "unsafe" world in a more ergonomic way.

simag 7y ago

Interesting post that show some current limitations of Rust, especially when not being able to control the design entirely. Rust's ownership system makes it harder to deal with the kind of dynamics wlroots exposes... It seems to me that the issue here is that there is no consensus (e.g library) on how Rust should deal with those issues, so if you want to just to provide bindings to a dynamic system you end up writing middle-ware that deals with that kind of dynamics in a Rust idiomatic fashion (taking advantage of ownership) instead of focusing on the task at hand. It somehow destroys Rust's promise of productivity. A Rust project of mine[1], in an early stage and currently paused, deals with those issues (probably in a way similar to the Handles described in the post) by providing Service smart pointers (i.e, Svc<T>) and a component-framework that helps deal with the dynamics of a service being unregistered (e.g, a monitor unplugged, or a dynamic library unloaded) ; that component framework helps you be sure that if your struct contains a Svc<Foo>, that service is still available when you're called, or the component reaches an invalid state. In general, it seems to be an area where Rust's ecosystem is still very early and would benefit from more input (such as this post) and consensus. Regarding the callback-hell issue, that "dehandle" macro looks very much like async/await, and it looks like it could be implemented either in terms of async/await (still unstable) or generators (even more unstable). Hopefully similar projects will be more likely to succeed as Rust and its ecosystem mature. [1] https://github.com/magnet/socrates-rs https://github.com/magnet/socrates-rs

Lowkeyloki 7y ago

Everything the author and the commentors here have said is completely legit. I'm not looking to start a war. And I completely acknowledge that what I'm about to say might be wrong as I haven't seen the author's code or the code of wlroots. But I have to wonder if the design decisions of wlroots itself may be dubious. If it's this hard to manage memory safely when you're trying to wrap the API in a language that demands you're kept accountable for memory safety....

ac130kz 7y ago

Does wlroots itself present good code quality or it is basically a horrible wrapper to support every Xorg use case?

Sir_Cmpwn 7y ago
wlroots is generally highly regarded in the Wayland community, both in terms of technical design and code quality. There's a reason that every Wayland project which started since wlroots has used wlroots, and those who didn't at first eventually rewrote their code to use wlroots.
- AsyncAwait 7y ago
  I get the sense that most adopted wlroots because Wayland is a complex protocol. There's no alternative to wlroots so part of the reason everybody uses it is because there's no choice if you don't want to start from nothing.
  - Sir_Cmpwn 7y ago
    Wayland is a simple protocol. The complicated part is everything else, like graphics and input and X11 compatibility.
    - AsyncAwait 7y ago
      Right, but that still means that getting to a point most end-users would consider 'usable' is quite complex. Especially because the documentation around these things can be lacking. P.S. I like Wayland and use it everywhere for the record.
- doubleunplussed 7y ago
  Is there any indication that mutter and kwin will rewrite to use wlroots? That would be glorious.
  - Sir_Cmpwn 7y ago

shmerl 7y ago

A pity wlroots itself isn't written in Rust.

ddevault 7y ago
C is the lingra franca of programming. By writing wlroots in C, we make it easy for a half dozen projects to make bindings to other programming languages. Rust is the only one that seems to have failed, and it's not surprising given the constraints of the language. wlroots brings together over a dozen different libraries and interfaces which are implemented in C. You'd have to repeat this process a dozen times over, running into the same problems which caused Timidger to abandon wlroots-rs, only more so. And for what? wlroots works great and didn't require shaving a thosuand yaks.
- steveklabnik 7y ago
  I know you know this, but for the benefit of others, you're talking about the C ABI, not about the language itself. Rust can expose a C ABI as well, and you'd get all those same bindings benefits. > wlroots brings together over a dozen different libraries and interfaces which are implemented in C. This is a great argument that it should be written in C, though.
- shmerl 7y ago
  > By writing wlroots in C, we make it easy for a half dozen projects to make bindings to other programming languages. Rust is the same in this regard, you can bind it to any other language. And it doesn't have the horrible downsides of C. So why not use it? > wlroots brings together over a dozen different libraries and interfaces which are implemented in C. That's an issue as well, and this of course runs deep. But starting somewhere should be still possible.
  - Sir_Cmpwn 7y ago
    >So why not use it? Because I don't like it, and I do like C. Rust is not the second coming of Christ, it's a programming language and has many shortcomings and tradeoffs.

kccqzy 7y ago

This code fn some_wlroots_callback(output_handle: OutputHandle, surface_handle: SurfaceHandle) { output_handle.run(|output| { surface_handle.run(|surface| { // maybe some more nested layers... }).unwrap() }).unwrap() } is just screaming continuation monad. You see the same code pattern in early Node.js code (sometimes leading to callback hell). In the JavaScript world, the problem was solved using Promises, and then async/await syntax. But more fundamentally, this is an instance of the continuation monad at work. The continuation monad transformer is defined as newtype ContT r m a = ContT { runContT :: (a -> m r) -> m r } and is nothing more than just a function that takes a callback. If this were Haskell, one could just write stuff = runContT $ do output <- ContT (run outputHandle) surface <- ContT (run surfaceHandle) -- and then maybe some more nested layers -- etc Granted, continuation code can easily be misused to produce an incomprehensible mess in Haskell (its full generality can be compared with goto), but with Rust's FnOnce trait, the scope for misuse is considerably reduced.

pjmlp 7y ago
JavaScript and Haskell get to enjoy the productivity of using a tracing GC though.
- kccqzy 7y ago
  This doesn't really have much to do with the runtime. Certainly with a GC, programming would be easier, but I'm really talking about abstractions within the language. Rust already has language-integrated support for the Result monad. Rust doesn't have a general `do` syntax, but it has `?` which has made programming with the Result monad much easier. Can we think about the continuation monad and arrive at a new syntax that can ease this style of programming?
  - uryga 7y ago
    i don't really know Rust apart from some curious onlooking, but afaik expressing monads (i.e. the Monad typeclass) in Rust is tricky – something to do with the lifetimes of closures and the objects they close over, i think. ([Idiomatic Monads in Rust] probably touches on that). it might be possible to just build it into the language a la Result or async/await though. [Idiomatic Monads in Rust] https://varkor.github.io/blog/2019/03/28/idiomatic-monads-in-rust.html https://varkor.github.io/blog/2019/03/28/idiomatic-monads-in...

josteink 7y ago

Personally I think Rust is a great language. That said it may not be great for everyone nor a great fit for every problem. Sometimes trying and admitting failure is a perfectly rational and valid option.

pcwalton 7y ago

I may be missing something, but what's wrong with the first example? The fact that you can leak an object shouldn't really matter for memory safety if it's just a handle to an object that the server manages. Yeah, it could go wrong, but it won't be unsafe. Object handles are essentially file descriptors, right? In general I think there's a tendency to overcomplicate safety features in Rust. The solution to an overly-complicated system isn't to throw the whole notion of safety out the window: it's to look at exactly what the complexity is buying you. If intricate combinations of Rust features are one extreme of the safety spectrum and C is the other extreme, there's frequently a happy design medium somewhere in the middle. Edit: Looks like oconnor663 over on Reddit had a similar but more specific proposal, which probably works: https://www.reddit.com/r/rust/comments/biq864/comment/em2kipe https://www.reddit.com/r/rust/comments/biq864/comment/em2kip...

Arnavion 7y ago
Yes, the same concept is discussed in https://news.ycombinator.com/item?id=19779243 https://news.ycombinator.com/item?id=19779243

newnewpdro 7y ago

I only skimmed the article, but the description of outputs and their asynchronous lifecycle making things complicated just made me think of what's become known as the ECS pattern in game development. Your outputs sound just like entities in a game. They come and go as they please, and you often have multiple references to them from myriad places since entities may interact with many parts of the game. I suspect if you investigated the established techniques for implementing ECS-style games in rust, you might find some simpler and established solutions for your troubles.