22 ms·

Digital Audio Workstation Front End Development Struggles

• vrglvrglvrgl 4y ago
  [dead]
• porkbeer 4y ago
  Its weird to see the impressions of someone who has apparently never used the equipment used in music production trying to talk intelligently about said devices. So many incorrect assumptions its really painful to read as someone aquainted with audio engineering. Interesting window into how the devs see it nonetheless.
  • johnfn 4y ago
    You should comment on the specific issues you see in the article.
  • astroalex 4y ago
    Huh? The author wasn’t talking about audio engineering, so I don’t see how that is relevant. The GUI programming issues they’re discussing would be true for any realtime application. The fact they are building a DAW is a bit beside the point otherwise.
  • linuxdude314 4y ago
    It seems like the level of experience of the average poster here has been decreasing steadily over time.
    • Cthulhu_ 4y ago
      I'm sure there's a clever term for it, but a lot of developers (and this is probably a timeless thing) think they can dive into any industry and make a difference, or see things from a different point of view and therefore better. I mean it has a basis in reality, loads of people here will have joined companies that were years behind, companies in new industries. Myself I've gone from cable TV to investment banking to public transit to B2B e-commerce to telcoms to the energy industry, but nowhere did I assume to think I knew better. Not when it came to their main industry anyway, software development and practices, sure, that's kind of my jam. When it comes down to it, every job is just data in, data out, localization and date / number formatting. And trying to get my colleagues to not overcomplicate things, like come on, it's just a simple list view, if you're bored by that you're on the wrong assignment.
  • unboxingelf 4y ago
    the comments about the layout of “mixing tracks” not following design guidelines or being a standard list/tree/etc struck me. in traditional audio engineering we call them channel strips and their layout reflects the signal flow - eq knobs are typically at the top, followed by compression, fx, and so on.
    • bradrn 4y ago
      I think he meant that they don’t follow typical design guidelines for graphical user interfaces on a computer. Which they don’t — outside DAWs (and maybe some image editors?), I’ve never seen a GUI arranged like a channel strip, where controls are stacked on top of each other but not necessarily arranged in a perfect grid. A set of controls like e.g. [0] in Ableton is highly non-standard, compared to the kinds of layouts most GUI toolkits provide. [0] https://rekkerd.org/img/201210/ableton_live9.png https://rekkerd.org/img/201210/ableton_live9.png
    • rzzzt 4y ago
      Naming conventions are a mixed bag (hehe) in DAWs, since not everyone who dabbles in them is an audio engineer by trade. - Ableton calls them "mixer controls": https://www.ableton.com/en/manual/mixing/ https://www.ableton.com/en/manual/mixing/ - Bitwig's manual does mention channel strips: https://www.bitwig.com/userguide/latest/the_mix_view/#channel_strip_section https://www.bitwig.com/userguide/latest/the_mix_view/#channe... - Adobe Audition has "track controls": https://helpx.adobe.com/audition/using/multitrack-editor-overview.html#editing_multitrack_sessions_in_the_editor_panel_and_mixer https://helpx.adobe.com/audition/using/multitrack-editor-ove... - Ardour... probably has mixing board tape and felt pen marker emulation as well, I'm late and couldn't be bothered to look up its manual.
  • rickdeckard 4y ago
    I find the article actually refreshingly dry. The author acknowledges and breaks down a large series of interface requirements for a music production tool and approaches it from a pure development point of view. I wonder what are the "many incorrect assumptions" you see. I didn't notice any attempt to judge or discredit any of the requirements, he merely described them in a very sober way. For someone very familiar with audio equipment it was a nice read for me. It's like reading a summary of user expectations to switches and knobs in a car, from a pure GUI development point of view.
• deleted 4y ago
  [deleted]
• andrewmcwatters 4y ago
  I actually know this space fairly well, probably better than most, because being an implementor of a UI compositor raises some difficult technical challenges with respect to UI tree backing layers and draw invalidation. I'm still not actually sure what the state of the art on these matters is after having researched this for years, and I suspect even what we do today across software like web browsers, video game UI is wildly inefficient for modern rich user-interface requirements. For instance, you could tailor GUI software to strictly draw to specific frame buffer/backing layer regions and you could manually invalidate and redraw when the developer specified, but that's generally far too implementation specific for most UI developers to actually follow. A master UI implementor wouldn't have too much trouble with this, but most people don't have a decade of graphics API level experience on top of designing world-class user-interfaces. So you have some ambiguous UI tree you need to draw and redraw when things move or animate. Today's state of the art is to multithread draw the space to tiles using virtual graphics commands that later get converted into target graphic API calls and those tiles translate to the corresponding graphics API backing layer abstraction, some sort of frame buffer. When the UI tree changes state, those leaves are invalidated all the way up/down the tree to the root at which point the compositor redraws what's necessary and combines the layers into what gets displayed on the screen. There's a bit of work to separate disparate UI element bounds from layers which require acceleration, but those nodes are flagged for the requirement usually by explicit properties. This is really complex work, and actually I think only web browsers today do it. I'm not sure even operating systems do, because usually they don't operate with the same challenges that web browsers do where you have total UI flexibility, or as many people using the platform as web developers. But it's also the only platform that has this architecture. So if you're building desktop software, you can try to emulate this architecture and end up with some fairly good performance. Even just using nested frame buffers gets you really far. But if you're writing front-ends on the web, you're stuck between the browser's compositor or rejecting it entirely and writing your own stack in a graphics layer that gets pushed to a canvas. None of this is good, it's all trade-offs, and I suspect the state of affairs will continue for years, because we don't have enough people doing compositing work. It's exceptionally specialized work. As an addendum, the DOM isn't slow. This is just parroting. It's total crap. People who say this have no idea what they're talking about. What's slow is everything GPU related. That's not even the CSSOM, either. It's specifically the part where pixels get rasterized. But you don't have lots of people who work on compositors telling you that. You have a bunch of speculators thinking the DOM is slow without actually measuring anything. Painting and drawing to backing layers is super expensive. Pixel fill is expensive. Changing document subtree node properties is not. You absolutely need efficient invalidation in UI drawing. I have no idea what this author is talking about.
• ruuda 4y ago
  It's weird that the article mentions Makepad, but somehow fails to observe that Makepad's primary demo application is actually a software synthesizer. Go watch some of Rik Arends' talks! Most recently this one from RustNL last week: https://www.youtube.com/watch?v=9Q4yNlbfiYk https://www.youtube.com/watch?v=9Q4yNlbfiYk (first talk of the day).
  • rikarendsmp 4y ago
    It's ok, we have been pretty quiet. Our primary launch goal is to have a full visual designer/IDE, not really push the UI framework stand-alone. However progress is happening and we are getting closer to that point as you can see in the talk above.
• chaosprint 4y ago
  DAW is so challeging! But in terms of audio effect plugins, I think EGUI is usable already. Here is a Dattorro reverb VST plugin written in Rust with egui and glicol_synth: https://youtu.be/DLFO4dXzKsg https://youtu.be/DLFO4dXzKsg Since EGUI can be used almost every where, I also made an experimental front end for Glicol music language with EGUI: https://glicol-egui.netlify.app/ https://glicol-egui.netlify.app/ There are many features missing, compared with HTML/CSS: https://glicol.org/ https://glicol.org/ The biggest issue with EGUI is that the style seems to be quite fixed. The default style looks great for some customisation would be more ideal in some context. although we are not there yet, it's great to see more and more attempts for audio /gui in rust. And I really feel that Rust is the best choice for the audio backend now.
  • andrewmcwatters 4y ago
    I don't think you can meaningfully use immediate-mode GUIs for anything related to compositing work. If you're creating debug UIs or basic UIs at best, you're fine. That might actually work out alright for DAWs, but if you want sophisticated user-interfaces where you can add effects to things, you need to start retaining graphics data, and so many people advocating for these old school immediate-mode GUI libraries just don't get that.
    • truckerbill 4y ago
      Can you explain this point a bit more? I've always hated retained mode GUI systems and im-guis seem so much nicer to work with. But I've not made something huge with them. What do you mean by compositing in this context?
    • pkphilip 4y ago
      Immediate mode gui - psuedocode where volumeSlider is a slider used for setting the audio volume and spectrograph includes a bar which displays the volume visually: function volumeSlider_onChange() { audioEngine.setVolume(volumeSlider.value); spectrograph.setVolume(volumeSlider.value); } function stopButton_onClick() { audioEngine.getCurrentTrack().stop(); } function audioEngine_statusCallback(Track track) { spectrograph.update(track.status); } To do something like this in a declarative UI, you will need some sort of a state machine, that state machine will then need to have a bunch of state variables, callbacks/hooks, providers/notifiers etc. And you have to have this managed for each variable. That is, there will be a LOT more code that needs to be run before updates flow from the GUI to the engine or from the engine back to the GUI. Why would you think that declarative UI will be more efficient?
• lagniappe 4y ago
  oh hey this is a topic i know of somewhat by chance :) i recently challenged myself to make a daw in the fewest lines of code and i came up with a cli based tracker that essentially does this: - load track.csv - each column corresponds to 1.wav, 2.wav, 3.wav, 4.wav etc - for each line, parse the cell of each column for characteristics and play the sound according to those characteristics (i just did on/off with an x, but you could do volume or duration) - delay before the next line for BPM what i ended up was a fun little toy csv-tracker. i was convinced it would be too stupid to be fun beyond the task of making it, but as soon as i saw the 'notes' scrolling line by line.. ooo-weeee what a rush of nostalgia 10/10 would recommend as a fun exercise. Edit: to actually respond to the article though, because this sounded a bit like storytime, since the author was asking (possibly rhetorically), i think the author -should- stick with Rust. im not saying this due to any fanboy tendencies, 90%+ of what i write is in Go, including the tracker I mentioned above. what makes me say that Rust is a good fit is because though it is difficult to produce code at a high velocity and with more simplicity in Rust, audio is one of the problem domains where realtime computation is a must, and Rust as compared to Go for example would excel in capable hands due to the way Rust manages resources and leans more toward realtime tasks. We're really splitting hairs here, but be that as it may, i know that it wouldn't take long to be thinking about tweaking the GC if i were doing a 'serious' daw in Go. evidently i was not the first with this idea https://www.youtube.com/watch?v=RFdCM2kHL64 https://www.youtube.com/watch?v=RFdCM2kHL64
• jancsika 4y ago
  What we need is a AAA game where the main character fiddles around on an in-game DAW in some abandoned recording studio to unlock a cool zombie-blasting weapon. That will guarantee that any widget redrawing therein happens as efficiently and responsive as possible. Maybe put it in the first level so that audio engineers can easily get to it and just hang out in that part of the game to do their work, glitch free.
  • Klaster_1 4y ago
    https://store.steampowered.com/app/243220/FRACT_OSC/ https://store.steampowered.com/app/243220/FRACT_OSC/
  • Cthulhu_ 4y ago
    I do feel like there should be more crossover between video game & GUI development; most game engines are crossplatform nowadays, GUIs in games are nothing new (and probably reinvented all the time), and everything is GPU rendered already. I've yet to see any desktop app that isn't a video game built in e.g. Unreal or Unity. I did see a colleague build a website widget (iirc it was about placing furniture or a TV in a living room to see what it looks like) using Unity though, that was pretty cool.
    • suby 4y ago
      I have seen a handful of desktop software apps built in Godot over the years. For example, https://bauxite.itch.io/bitmapflow https://bauxite.itch.io/bitmapflow is an app which I've gotten value out of built in Godot.
      • raincole 4y ago
        My favorite Godot app is Material Maker (https://www.materialmaker.org/ https://www.materialmaker.org/). I downloaded it when Substance Designer was acquired by Adoboe. I'd say it actually has better usability than Designer.
    • raincole 4y ago
      Because Unity's UI old system (ugui) is actually not that good? From my experience using web+Electron is just so, so much easier. By the way, SimCity's UI is done by a custom version of WebKit. https://twitter.com/MaxisScott/status/310835756107177984 https://twitter.com/MaxisScott/status/310835756107177984
• jupp0r 4y ago
  It's kind of funny how the author dismisses web technologies while complaining about redundant rerenders in Rust GUI libraries. This is all solved in browsers, both on the rendering engine and on the web framework library levels. Yes you need to pay attention to performance, but the premise that "web is slow" has been disproven so many times already. Just because it's really easy to write apps that are slow (and thus these apps being ubiquitous) doesn't mean that you can't write performant apps if you are so motivated.
  • andrewmcwatters 4y ago
    The amount of effort it takes to create high-performance applications in a web environment is significant, though, and the benefits you get simply working in a faster language in a desktop environment are lost on most web developers, I think, simply because most of them don't actually do any desktop or graphics API work. They simply wouldn't know. Also UI compositors are not a solved space. If they were, people would absolutely not being using Electron. But no one is saying, "Hey I have a CSS 2.1 compliant rasterizer and compositor that you can use in your C++ or Rust environment!" are they?
    • rahkiin 4y ago
      Please, if someone has just that, let me know. It would be excellent for games. I do believe there are implementations of flexbox in rust.
      • andrewmcwatters 4y ago
        Yeah, it's a problem space I'd like to try to tackle again some day. Yoga et al. aren't enough. I think people are dying for a web-based compositor in other host language environments, but don't know how to elucidate this concept because they don't know anything else.
      • fit2rule 4y ago
        [dead]
    • 4y ago
• grumpyprole 4y ago
  > This performance is not a problem if the app is small or the app already redraws every frame like a video game. Seems to me like immediate mode UIs were dismissed rather too quickly. Why not update a DAW UI like a video game? It has similar realtime requirements.
  • andrewmcwatters 4y ago
    Immediate mode GUIs have significant limitations. You can probably lie to your end users by saying the API is immediate mode styled, but actually retained under the hood (and I think to some degree, this already happens today with cached state), but I think that defeats the point.
    • modeless 4y ago
      "Immediate mode" is a description of the API, not the implementation. It's not a lie to cache state under the hood. In fact it's sensible and doesn't defeat the point at all.
      • andrewmcwatters 4y ago
        This is incoherent. The implementation is shaped by API limitations. Immediate mode APIs that hold data on the GPU are not immediate, they're by definition retained.
        • modeless 4y ago
          The people who actually make immediate mode GUIs disagree with you. https://github.com/ocornut/imgui/wiki/About-the-IMGUI-paradigm#half-of-a-definition https://github.com/ocornut/imgui/wiki/About-the-IMGUI-paradi...
          • andrewmcwatters 4y ago
            ocornut's own bulletpoints are incoherent. He can call it whatever he wants, it doesn't mean it's meaningful. The fact that he's designed a retained mode library with an immediate mode API doesn't mean that it doesn't have literally the same limitations, because he doesn't expose the abstractions to you to actually do meaningful retained work. I don't care what made up paradigm someone else has invented. Anyone who was doing immediate mode graphics programming before shader based pipelines knows exactly what it is, because you would have used the APIs by standards organizations themselves.
• devjab 4y ago
  > Both Javascript and the DOM are slow, there's no changing that. The DOM isn't slow. Adding or removing a DOM node is literally a few point swaps, which isn't much more than setting a property on an OOP language object. What is slow is "layout". When Javascript changes something and hands control back to the browser it invokes its CSS recalc, layout, repaint and finally recomposition algorithms to redraw the screen. The layout algorithm is quite complex and it's synchronous, which makes it stupidly easy to make thing slow. Frameworks like React with virtual DOMs help you by redrawing on every state change, which will typically both speed things up significantly and also carry you through a lot of things that would break if you were using non virtual DOM frameworks like JQuery. You can get smooth and quick animations with JavaScript and the DOM, however, but only if you build things to limit what your application can do in a frame to operations which can be performed on the GPU. I can understand if that's not exactly an option for the author of this article, but I'm not convinced you couldn't get Electron (or similar) to work as a GUI framework for a DAW. I suspect Electron specifically might actually be a pretty good framework for DAW's, considering what Microsoft is doing with VSC. I also think that the downside of frameworks like Electron is that you need to be, not only Microsoft, but the Microsoft VSC team (and not the Microsoft Teams team) specifically, to bend them to your way. That being said, I think this will likely be one of those things where you'd need to create a GUI framework (or bend one to your will) to really get DAWs to work for you. Because I think the author is spot on about the "state" of GUI frameworks in general.
  • sirsinsalot 4y ago
    > I also think that the downside of frameworks like Electron is that you need to be, not only Microsoft, but the Microsoft VSC team (and not the Microsoft Teams team) specifically, to bend them to your way. I am not sure this is true, Discord as one example is pretty good. As is Slack's desktop app* *By "good" I mean, perform the task in a stable way and appear fairly 'native' in terms of UX (but not UI design of course)
    • skrebbel 4y ago
      It could be argued that in this sense, the Discord frontend team “is” a VS Code team (and not a MS Teams team)
• z3t4 4y ago
  You would like to use a canvas element for custom ui. A canvas is a hardware accelerated 2d library (for example Skia) where you call functions to draw lines and curves. To make a decibel gauge you would have the background face image in a buffer, then only redraw the hand line on top of it for each frame.
• franky47 4y ago
  I worked in that field for 8 years for a somewhat known plugin editor. We used JUCE a lot at the beginning, but ended up slowly moving to an OpenGL-backed renderer for performance-critical parts (eg: VU meters and other audio-to-UI feedback), and JUCE for the rest of UI scaffolding (layout, controls, menus etc). Same thing went for the other parts of a plugin architecture: we wrote our own VST/AU/AAX interfaces with the DAW to overcome some of the issues we encountered with JUCE. After I left the company, my former teammates rewrote the whole UI framework to a pure OpenGL implementation. I'm not sure if there's even a drop of JUCE in the codebase anymore. Note: this was before the PACE acquisition (though we had good contacts with the PACE team, their founders being brilliant engineers). So my advice to people starting in this field: use whatever is available now, JUCE is a treasure trove for beginners, then slowly optimise where needed.
• wg0 4y ago
  I wonder (and admire) what DaVinvi Resolve is using for it's UI. It seems very polished but is also very performant and is even cross platform.
  • tayistay 4y ago
    They're using Qt. All the Qt libraries are there inside the app's package on macOS.
    • wg0 4y ago
      Amazing. Didn't know Qt is so capable in the right hands. Doesn't even look like typical Qt app.
• zorr 4y ago
  I have a feeling desktop development is going to slowly revert back to classic OOP paradigms just like the web world detoured from MPA -> SPA -> SPA+SSR -> back to something that looks a lot like MPA again. Classic toolkits like GTK, Qt, UIKit/AppKit, Swing and JavaFX use OOP to solve some of the problems this article talks about. However, this OOP model seems to be somewhat incompatible with Rust's strict memory safety rules. At least I think it is because I haven't seen any big efforts for a Rust OOP toolkit like this. Most of the existing Rust toolkits are immediate mode or ECS or something "reactive" that looks like React. While I understand the idea of moving away from "unsafe" languages like C/C++ to something like Rust, I wonder if Rust is the right language for building UI applications? Maybe a better architecture is to use Rust for the core logic as a library to be consumed by a higher level language that meshes well with these OOP widget tree and event handling paradigms.
  • toastal 4y ago
    I think we need other UI models instead of everything mature being object-oriented-oriented. Doing UI work in FP (or FRP) is great in many aspects until you need to integrate with these OO models like the DOM, et.al. Direct integration or a first-class VDOM-like model would be a nice step. There’s a tangent issue with all of popular game engines built around objects.
    • lolrustards 4y ago
      Even Lisp programmers decide to pragmatically use an OOP model when doing GUIs. The idea of using FP for everything is about as sensible as the paradigm of using OOP for everything.
      • Capricorn2481 4y ago
        You say that like CL programmers are inherently functional, which has never been true. They have always embraced OOP when needed
      • kazinator 4y ago
        Less, I would say; because using OOP for everything is straightforward, just obtuse, verbose and possibly slow. You don't have to solve paper-worthy academic puzzles to use OOP everywhere. Using OOP everywhere has been done and probably continues to in some shops. There was a time in the 1990's when nobody was fired for using OOP everywhere. In Common Lisp, absolutely every value you can create has a class: the class-of function is meaningful for any value. You can replace all the defun forms in a Common Lisp program with defgeneric and defmethod.
• ogoffart 4y ago
  Slint dev here. I'm a bit disappointed to see that Slint is considered a non-starter. In fact, one of our users is currently working on exactly that (a VST plugin for an audio application) and they are about to release it very soon. Unfortunately, I can't share a link until it's officially released. The author seems to dismiss declarative UI without explaining why. From my perspective, declarative is the best way to describe UI. Regarding the limited support for custom widgets, I disagree. In Slint, it's incredibly easy to create custom widgets. A widget is simply a component that can be created and utilized. While it's true that our desktop integration is still a work in progress, we have already implemented the core functionalities needed for building such UIs. We are actively developing features based on user needs and prioritize implementing features that our users require immediately. If you're interested in using Slint, please reach out to us, and we'll ensure that we implement what is necessary for your product.
  • chaosprint 4y ago
    Do you have plan for wgpu?
    • tronical 4y ago
      We're working on interop, so that when you select Slint's Skia renderer on top of Vulkan or Metal, you can import a wgpu texture if wgpu's Metal or Vulkan adapter was chosen. What kind of feature set do you need?
  • zerr 4y ago
    Declarative UI, even when implemented coherently within the same langauge like in Flutter and MAUI/Xamarin, so there is no need for an ugly bridge between two worlds, still leads to an unreadable nested hell. Surprisingly, an imperative GUI creation code is much more easier to read and modify.
    • deleted 4y ago
      [deleted]
    • mort96 4y ago
• davedx 4y ago
  I've worked on big complex UI projects that needed to run on embedded hardware, using different tech. Some of my observations over the years, if it helps: * The DOM isn't "slow" per se, but using the DOM for this kind of application is usually the wrong path to take. As soon as I need something even slightly off the well beaten path of web applications I tend to reach for canvas based solutions. * Libraries like EFL are quite nice to work with but also quite low level. By default that means C, but there are bindings for other languages. * Some of the things you describe (like rendering spectrographs) I'm not convinced are as CPU intensive as you think they are? Have you benchmarked? Winamp was doing these kind of visualizations back in the 90's. I kind of doubt if you really need to use shaders to render a typical DAW if you're smart with which parts need immediate mode rendering and which can use retained mode. Using render targets etc seems like a smart way to do a DAW, as most parts are actually quite static and only move around the screen. * I have a feeling - just a feeling - that a lot of people choose Rust for projects because it's one of the newer shinier languages, but as you've discovered, maybe that's not necessarily smart. I would choose Rust for its concurrency paradigm - which might be useful for a DAW! - but for building a UI I would actually personally choose something JavaScript-based. The productivity is just so much better than most other languages, for building UI's. This is not just my experience - I worked for some time at a large public listed US company who used JavaScript for all their big UI projects on embedded hardware, what they changed over time was the underlying rendering engine, but they kept JS because 1) hiring devs 2) productivity. Good luck, it's a super ambitious project, but I'm sure it's amazing fun to work on! :)
  • jerrre 4y ago
    Regarding drawing analyzers, spectrographs: I feel it actually has become a lot more expensive to draw individual pixels to a screen with the cpu compared to 20yrs ago. Perhaps just because the resolutions are higher, but maybe the architecture (hardware or os) have just changed.
    • cannam 4y ago
      With CPU, totally. As an example I can look at my own Sonic Visualiser application, largely written 15-18 years ago and entirely CPU-driven. Relative to then, it's now horrible on contemporary Macs for example - it feels far slower than it did a decade ago. It just isn't what the hardware expects. (There may be an element of toolkit-platform impedance and simple poor design on my part - it uses Qt and feels quicker on other platforms - and I don't want to argue the details here, but I think the basic principle that you really want to avoid CPU in the frame update is sound. Preparing things on a non-time-critical path via CPU should be another matter however, there's quite a lot of capacity there.)
• deleted 4y ago
  [deleted]
• andybak 4y ago
  > Ideally I want to support loading user-generated themes. I really hope this just means "alternative colour palettes". Otherwise they're about to make one of the hardest UI programming tasks I can imagine exponentially harder. I've never understood the passion for themeability. It's like going into a restaurant and expecting to participate in the cooking of your meal. As much nostalgia as I have for the heyday of Winamp skins, most of them were awful and had terrible usability. It was a garish novelty akin to "Pimp My Ride"
  • alkonaut 4y ago
    Completely agree. I'd even go so far as to say that beyond pure accessibility and compat concerns (color blind modes or high contrast erc) even that is going too far with customization.
  • dagw 4y ago
    It's like going into a restaurant and expecting to participate in the cooking of your meal. Lots of people expect and do this all the time. I want this cooked like this, no this, this replaced with that etc. Hell, some restaurants just put out all the food they have on a big table and say "go ahead and put together your own meal with whatever you want, we don't care", and that is many peoples favourite restaurant experience.
    • andybak 4y ago
      Are you disagreeing with my views on theming or just my poor choice of metaphor? ;-)
      • dagw 4y ago
        I personally agree with you, but I know that I'm not necessarily in the majority. Much like many people seem to prefer buffet restaurants to restaurants with a fixed menu.
    • Ygg2 4y ago
      Parent obviously meant you'd be in the kitchen, cooking. Not barking orders.
• bambax 4y ago
  I don't know much (or anything, really) about DAW development, but as a Reaper user I know you can make a cross-platform DAW that performs extremely well on very little ressources, including displaying many instances of all the things the author is talking about (spectrograms, audio files, zoomable midi clips, etc.) in real time, on a Raspberry Pi, while playing audio and driving physical synthesis plugins.
  • spacechild1 4y ago
    True! Fun fact: Reaper uses its own UI framework. Parts of it are open source: https://www.cockos.com/wdl/ https://www.cockos.com/wdl/
• alkonaut 4y ago
  Isn't Bitwig using a C++ backend with a Java frontend? It it were me I'd probably choose between biting the bullet and going all rust, or if I was planning on interop anyway, then go for something a bit more high level like AvaloniaUI (C#) or a thin java solution on top like Bitwig or the Jetbrains platforms.
  • pjmlp 4y ago
    Yes they are.
• squarefoot 4y ago
  I can't recommend a toolkit or a set of widgets because ages ago I came to the conclusion that there's no way to use traditional ones for a good DAW interface, and something very specific to the task should be developed, which is a pain in the ass. For spectrograms, I could suggest a look at Jaaa's source. It draws directly using wrappers on X11 libraries and it's fast and efficient also on low end hardware. https://wiki.linuxaudio.org/apps/all/jaaa https://wiki.linuxaudio.org/apps/all/jaaa http://kokkinizita.linuxaudio.org/linuxaudio/ http://kokkinizita.linuxaudio.org/linuxaudio/
• mkl 4y ago
  For the relative mouse movements, can't you just move the (hidden) mouse pointer to the centre of the screen every time you get a movement event? It will be very hard to hit the edge that way.
• nologic01 4y ago
  After decades of development the GUI space feels more chaotic than ever. There are several dimensions that can become critical bottleneck in advanced use cases (performance, platform independence, ease of development, library/tool availability and maturity etc) and - as this post highlights quite clearly - it is non-trivial to chart a path. Some of the difficulty is definitely intrinsic to the problem space. Nobody said that arbitrary complex interfaces have a license to run at lightning speeds with minimal coding on arbitrary and/or minimal hardware. But I do have the feeling that in part the difficulty is because the different approaches (both historical and current) have been effectively islands, self-contained with no standardization or separation of concerns. E.g. I would at least hope that we would by now have an accepted universal way to describe GUI's (declarative). I think this might be one reason for the popularity of web based approaches: while html/css was made for documents not GUI's, it is at least something everybody agrees on.
• deleted 4y ago
  [deleted]
• gizajob 4y ago
  You could just buy Ableton for $500 and save yourself the dev hassle. It’s already a solved problem. Why reinvent the wheel?
  • JSavageOne 4y ago
    What a toxic and stupid comment. Hey guys Windows already exists, why build another operating system? /s
    • chris-orgmenta 4y ago
      Agree, but ironically I find your first sentence to be auto-descriptive / equally against guidelines.
    • gizajob 4y ago
      Yeah but it's a rule of programming laziness and hubris as outlined by ESR and Larry Wall, that you don't need to keep inventing tools that already exist. There's better things to programme. This dev isn't going to build a better Ableton on his own, and this site now simply lists all the reasons he can't, not problems to solve on his own. It will take magnitudes more than $500 of his time to replicate anything close to Ableton's functionality. I wasn't being toxic at all, just pragmatic. Pragmatism is part of knowing when to programme, and when not to.
      • EyeSayUntoThee 4y ago
        Ableton isn't open source. There are also several other DAWs and Ableton isn't some pinnacle of software engineering. Take Larry and ESR and anyone else with some pinches of salt, because accepting some proprietary implementations of DAWs as a tombstone on the FOSS development of DAWs is rather reductionist to the point of being totally wrong. Why would you bother posting such a comment? Your notion that all questions regarding how to do the manifold tasks a DAW are resolved is specious, and you couldn't possibly prove it to be true because it's false. Ableton has managed to make a product that solves the DAW questions in one particular way, and it's a black-box to boot, so that's not in the public domain, as Meadowlark is. The technical questions entailed in Rust trying to mate with GUIs are EXTREMELY UNSOLVED AND RELEVANT RIGHT NOW! Let me tell you, state, state, state. I can't even propose that you work for Ableton as they are classier than this! Their product is a good and useful DAW for many people, you bet. Your point is?
• alestsurko 4y ago
  For 5 years or so, I'm watching for GUI infrastructure for my imaginary audio applications. And recently I started to develop an audio application in Rust. What I learned is that there is no a single best library in any language. Also, you shouldn't rely on anything as if it even a very popular thing at the moment it can become unmaintained very quickly. So it's better to choose a satisfying solution in what available currently and abstract it as much as possible to be able to switch to something else in the future. I also did a very simple benchmarks for GUI libraries I was interested in (instantiating 10 000 rectangles, styling them and doing layout) and the leader on macOS was the browser (Tauri) with Slint at the second place. But, unfortunately, Tauri is unusable for such kind of application because of IPC. If you need to visualize your audio data, you need to continuously send it to GUI. The fastest way of communication for that would only be WebRTC, because of the sandboxing. Or you would need to go all WASM, but then you don't have access to a lot of other features you needed. As a suggestion, take another look at Qt. At very least MuseScore and Ardour use it. Also, at my opinion the most mature Rust GUI library yet is egui, not Iced. FYI, Audulus uses immediate mode GUI based on nanovg (https://github.com/audulus/vger#why https://github.com/audulus/vger#why). And they develop their own Rust GUI library BTW (https://github.com/audulus/rui https://github.com/audulus/rui)
  • willismichael 4y ago
    > As a suggestion, take another look at Qt. At very least MuseScore and Ardour use it. Ardour uses GTK 2, and only minimally at that: https://discourse.ardour.org/t/gtk-2-has-been-deprecated-is-there-a-roadmap-for-removal/105214/5 https://discourse.ardour.org/t/gtk-2-has-been-deprecated-is-...
• syntheweave 4y ago
  The problem is not really in the toolkits, it's in looking at how the performance emerges from the design. Layout is the fundamental reason why GUI updates can be expensive: if you reduce it to a more static form, layout becomes cheaper, so recomputing it frequently becomes viable, and you can start caching more results without causing trouble. What's the reason not to use static layout? Configuration. But configuration is something the user doesn't want to do a lot of either. If that's what they spend their time on in the GUI, they aren't making forward progress on the task. So there's actually a self-alignment: a GUI that works well is one that can stay relatively "flat" in its performance profile, too. Trying to address this at toolkit level isn't abstract enough, because the things you do want to change, you most likely will see as either "this always updates every frame" or "this is a user action that triggers a full recompile". That isn't something that a general purpose framework can provide - it can't know that sometimes you have a window that needs to completely recompute its layout, and other times, it will never change or even move position. It's going to provide a vocabulary that favors one or the other, but you have to provide support when you need the more exceptional of the two. IME I usually have done fine pushing the IMGUI approach farther. The problem it tends to face is with something like a scrolling list of elements of variable size: making that list automatically line up cleanly and efficiently is quite a lot of effort. But...do you need that design? Will fixed size elements work? Or pagination? That's the kind of thing where it's tempting to say that you do need all the features, but it can quickly fall away from any specific use-case and turn into an aesthetic preference.
  • Capricorn2481 4y ago
    > But configuration is something the user doesn't want to do I guess you don't make music, which is what this post is about. If people didn't want configuration, they would use Audacity and just record. How are you going to paginate a synths oscillator? Volume meters?
• lolrustards 4y ago
  Title should be changed to Rust GUI Development Struggles.
• criddell 4y ago
  There’s also the option of dumping the cross-platform requirement. Which platform is most important to the author? Then narrow down further by deciding on which configuration of that platform matters. Define those things, then build the best DAW possible. In my mind, this doesn’t rule out web-based stuff because I view the web as its own distinct platform.
• signaru 4y ago
  There's a relatively new C++ GUI library literally called "Elements". Not sure how it works though, but the way it looks, and the music background of its creator makes it appear that DAW applications might have at least been considered. https://github.com/cycfi/elements https://github.com/cycfi/elements Edit: It does mention VST and AU in the introduction: > "It should not own the event loop and should be able to co-exist with components within a plugin host such as VST and AU."
• davidgerard 4y ago
  Tom Ellard from industrial band Severed Heads wrote "A Completely Biased Guide to Digital Audio Workstations", which has some pointed comments on DAW interfaces: https://nilamox.com/a-completely-biased-guide-to-daws/ https://nilamox.com/a-completely-biased-guide-to-daws/
• bandika 4y ago
  It’s a hard problem, and I don’t think there is a single solution for a DAW UI. On the one hand there are a lot of UI elements that can be built with traditional widget kits, like the toolbars, panels, even knobs and sliders just custom widgets, you can even build a channel strip that way. On the other hand there are the “contents” or “views” like track view, clips, automation, piano roll, eq etc. I believe those are best to be handled as 2D scene graphs with direct GPU rendering. Even then, each of those DAW specific views presents their own set of challenges which are very involved to address. So I would definitely choose a toolkit that allows both traditional widget trees as well as scene graph style rendering (for eg Egui + wgpu in rust, or Qml in C++). There are other challenges too, I’ve always found synchronizing the ui, the rendering and the audio thread so that everything feels responsive is very hard.
• gattis 4y ago
  Not the coolest answer but I feel like QT is gonna be hard to beat here. It has a ton of this functionality already, signals are not the enemy for something this complicated and time-driven, it runs on anything and performance is great, and its pretty powerful/flexible when you need to whip up custom graphics stuff. QML and JS are really nice to have. I wrote a modular synthesizer GUI with it (https://github.com/ohmbre/ohmstudio.git https://github.com/ohmbre/ohmstudio.git). Took a gamble on QT5 because I wasn't a big fan of QT4 or below, and don't much like KDE, but it turned out perfect. Just never came across anything and said "ew well that's gonna be hard with the tools I have." There was always something handy. GUI never got in the way of audio performance, as long as its on its own thread and using ring buffers etc, you're fine.
  • phkahler 4y ago
    I'm not QT or any other gui toolkit is well suited for this. Every widget is custom and many require high framerate / low latency. It's probably best to treat it like a video game and just render everything every frame. Only use gui events to affect the underlying data model, don't trigger any gui updates or rendering from them. In this light you're looking for fast software or OpenGL rendering as the key feature in the toolkit, but you still want menus and toolbars.
    • carlmr 4y ago
      Which you can do with QT. It's a bit unwieldy, but it's performant and flexible.
      • phkahler 4y ago
        >> Which you can do with QT. It's a bit unwieldy, but it's performant and flexible. Not doubting that it can be done with QT, but I don't think QT is really optimal either. SDL is ideal for the rendering, but has none of the regular toolbars and such, and IIRC no multi-window support. I'd actually recommend GTK for this one, since it is lighter weight and the application doesn't need a ton of actual toolkit functionality.
    • 4y ago
• otikik 4y ago
  Should have gone console-based, keyboard controlled
• berryg 4y ago
  Maybe this is the way forward: build a Rust SDK which does all the high-performance heavy lifting. On top of it you create bindings (using uniffi) to your favourite front-end language. And you create a frontend focusing just on the front-end stuff. This was presented at FOSDEM 2023 by Matthew Hodgson on building Matrix 2.0. 15 minutes into the video Matthew explains the redesign of the Matrix clients for mobile. His conclusion: maybe this is ultimate stack to build mobile applications. https://fosdem.org/2023/schedule/event/matrix20/ https://fosdem.org/2023/schedule/event/matrix20/
• etimberg 4y ago
  The author should look at how EDA (electronic design automation) tools are built. They have many of the same challenges but they also have a lot of solutions
• lbayes 4y ago
  Building user experiences is inherently some of the most difficult software engineering there is. It's not difficult because the existing tools are poorly made (though most of them are), it's difficult because the core problem is extremely complex (in the Rich Hickey [1] sense of the word, e.g., braided, intertwined, complected, etc.). In order to build a user experience, we start with some information that can often (though certainly not always) appear both simple and easy (i.e., unbraided and comprehensible). But then we must create a projection of that information onto a 2D (or 3D) plane and manage it's changing state over time. This is where everything falls apart. We're usually trying to manage a very wide, very deep, loosely defined tree of state and we're trying to reflect an arbitrary number of user interactions and mutations instantly, and over time. Exactly zero of our tools (programming languages, UI toolkits, Operating Systems, etc.) have strong support for managing change-over-time in a way that makes this type of problem fast, stable or enjoyable to present, use, verify or mutate. I love that the OP showed the triangle of Performance, Adaptability and Velocity. I've been framing a similar concept with teams since 2011 which I refer to as, "Fast, Stable and Delightful." It's my thesis that we need to balance these three requirements in both our activities and in the output that we produce. This balance is incredibly difficult to achieve and harder still to maintain. Different organizations tend to focus internal power in one of these 3 nodes (e.g., Google on Stable, Facebook on Moving Fast, and Apple on Delight). Note that the desire to move "Fast," or even to be "Stable" rarely achieves the desired result. The only organization I've seen manage these values effectively is Apple and I believe this is because people there have understood for decades that speed and stability are prerequisites for delight, but that neither speed, nor stability (alone or together) are sufficient to make something that gives humans the sense that someone, somewhere actually cares about how they feel. I love how the OP is exploring this problem space and how they are digging into the existing solutions and trying to define the problem with some rigor. This is fertile ground for exploring new ideas and new approaches. Please do not discourage this activity! [1] Simple Made Easy: https://www.youtube.com/watch?v=SxdOUGdseq4 https://www.youtube.com/watch?v=SxdOUGdseq4
• p0nce 4y ago
  For the audio plugin side of things, I wrote a bit about the kinds of optimizations we do: https://dplug.org/tutorials/Dplug%20Tutorials%2011%20-%20UI%20Feedback%20optimizations%20in%20Lens.pdf https://dplug.org/tutorials/Dplug%20Tutorials%2011%20-%20UI%... People have come to expect much more UI feedback than before, and this has a CPU cost.
• PaulHoule 4y ago
  I'd be concerned about the lack of garbage collection in Rust. It's little appreciated what a breakthrough garbage collection was for code reuse: it was probably more important for code reuse in Java than object-orientation itself. If you don't have garbage collection the API between a library and an application has to address memory allocation and freeing. If the relationship between the library and application is simple this is not a problem, but if library needs to share buffers with the application and particularly if the library constructs complex data structures that are connected with pointers, it is a difficult problem because often the library is not sure if the application is done with a buffer or vice versa... But the garbage collector is! (Ironically this is a case where "abstraction", "encapsulation" and such are dangerous ideas because the problem the garbage collector addresses is a global problem, not a local problem. As a global problem can be correctly addressed globally, this is an appropriate use of "abstraction", but if your doctrine is that "abstractions are local" you have baked failure into your mental model at the very beginning.) It's not accidental that OO languages became popular when GUIs did because OO is a good fit for the GUI domain. Memory allocation matters a lot, particularly if you are building complex applications for creatives. If you're just writing mobile apps and overcomplicated web forms, you can say that "the panel belongs to the form", "the input box belongs to the panel", ... The idea of borrowing becomes highly problematic when you have complex applications where the user can add and remove property sheets and other controls to the UI because you now have a graph relationship between components, not a tree relationship. If you have garbage collection and other tools appropriate to graph relationships you can make it look easy, approach it with the wrong tools and you will always be pushing bubbles around under the rug.
  • EyeSayUntoThee 4y ago
    Audio is literally a major domain where GC is a big no-go... (even a GC running in another GUI thread can be an issue if it's a stop-the-world operation) You won't find a single audio backend in a GC language. GC is obviated by Rust anyway (it's native RAII pattern involves "ownership" which allows "borrowing", things are immutable by default...) and there are ARC and Box implementations (check out Box particularly) You could always slap something like Boehm on, lol... These things need to be solved by the Rust community, regardless, so it's full-steam ahead, and GC is not something tenable when there are ways to control the timing of resource freeing that won't glitch audio playback, which GC would do. GC is for when you have no idea about the types and lifetimes of the objects you create.... when you know these things, why wouldn't you setup proper shrink-wrapped alloc/de-alloc in a time-coordinated fashion?
• Strom 4y ago
  The article claims that Druid doesn't use damage tracking for rendering, but this is not true, or at least not reasonably so. Druid does track damage regions and pipes this all the way to the OS. The only exception is macOS, where the regions are still tracked but not fully connected to the OS. There is an abandoned PR that even adds the connection. That said, not using Druid for a DAW project is a fine choice. Doubly so for a new project, as it was correctly pointed out in the article that Druid is now in maintenance mode.
• tensor 4y ago
  I'm curious as to why the author thinks that the fltk-rs rust bindings to fltk are "practically unusable." I've been using them and while there is a definite learning curve they seem complete, stable, and performant. Also the author is extremely helpful and responsive!
• pkphilip 4y ago
  My recommendation: try Delphi/Lazarus. It is trivial to develop new GUI components in Delphi/Lazarus and there are already very good GUI component libraries for Lazarus available as packs.
• talkingtab 4y ago
  I have recently been using threejs (as in React-three-fiber). In the process I found myself wondering why I have been using html/css/react all this time. And if you think that is a dumb thought, let me tell you I do too. But it just won't go away. I do understand the useFrame() issue, but still ...
• elevaet 4y ago
  This post really makes me appreciate the engineering that led to the completely usable and powerful DAWs that ran on the comparatively weak hardware of the late 90s, early 2000s like Cakewalk, Cubase, Logic.
  • weinzierl 4y ago
    Reaper, let's not forget Reaper. Fully blown DAW with brilliant UI in like 15 MB. Cross-platform too.
    • elevaet 4y ago
      Absolutely, such a great project! I understand one of the guys behind the project is doing it more or less out of passion as he made his money from WinAmp back in the day?
  • deckard1 4y ago
    and without GPUs, mind you. And a lot of that trickery probably came down to: let's write everything in assembly. Fruity Loops was absurdly fast in the early 2000s and was written in Delphi and assembly, IIRC. In 2023 DAWs should not be that difficult to make fast. It's a solved problem. Multi-channel audio processing is one thing that modern CPUs should not even break a sweat at. I believe this article has uncovered the basics that have made every DAW complex from the beginning, but seems to be focused on efficiency with your bog standard off-the-shell GUI library. That's the mistake, IMO. None of the DAWs I know of use, for example, standard GTK. Even the ones that do use a GUI toolkit use it minimally, for the file pickers and configuration. Not as the primary interface for the DAW itself. You need to go lower level and remove the layers of cruft. I mean hell, adding language bindings on top of everything is going to add yet another layer of inefficiency.
    • elevaet 4y ago
      Oh yes, thanks for the reminder about Fruity Loops, another great piece of software from that era. Not a DAW, but I really miss CoolEdit
• jcelerier 4y ago
  Some comments on the issues mentioned at the beginning of the article as I had the same questions in https://ossia.io https://ossia.io: - DAWs have a lot of toolbars and panels (browser panel, timeline panel, piano roll panel, fx rack panel, mixer panel, audio editor panel, automation editor panel, settings panel, etc.). for this Qt always performed brillantly for me. I went from QDockWidget to QSplitter-based layout though. - Some widgets like decibel meters and other visualizers are constantly being animated, meaning the GUI library needs to efficiently redraw the screen every frame. all the CPU-based toolkits will only redraw what changed though - In addition, visualizers can be expensive to render on the CPU (especially spectrograms/spectrometers). Ideally you should use custom shaders to render them on the GPU. fair, but then you may pay the cost of a GPU -> CPU transfer which isn't always free - Clips on the timeline are notoriously expensive to render. There needs to be some way to cache the contents of clips into a texture (Either directly or by making use of the GUI library's "damage tracking" which I'll get into later.) Audio clips are the biggest culprit, because rendering waveforms requires the CPU to first do a linear search through the source material for peak values, and then render the waveform pixel-by-pixel (or even better use custom shaders to send commands to the GPU). AFAIK a lot of DAWs perform a full scan when the file is loaded and save the result in a database or in a file next to it (e.g. Reaper's .reapeaks, Ableton's .asd...) so that you don't need to perform a complete rescan. In ossia.io I use three different algorithms / display methods depending on the zoom level used: at "far away" zoom it uses the minmax of the audio slices, at intermediary zoom it draws lines and when getting closer, it starts drawing individual samples. https://github.com/ossia/score/blob/master/src/plugins/score-plugin-media/Media/Sound/WaveformComputer.cpp https://github.com/ossia/score/blob/master/src/plugins/score... - Automation clips can contain a bunch of bezier curves, which are slow to render. Convert those to line segments with an approximation setting that looks good enough and it'll be ten times faster (keep the bezier for your data model of course). https://github.com/ossia/score/blob/master/src/plugins/score-plugin-curve/Curve/Segment/Power/PowerSegment.cpp#L57 https://github.com/ossia/score/blob/master/src/plugins/score... https://github.com/ossia/score/blob/master/src/plugins/score-plugin-curve/Curve/Segment/CurveSegmentView.cpp#L157 https://github.com/ossia/score/blob/master/src/plugins/score... - Piano roll clips can contain lots of little rectangles in order to display a "minimap" of the MIDI notes inside of it. oh damn yes, I spent so much time on this and it still needs so much optimizing... if someone wants to give a shot at it :D https://github.com/ossia/score/blob/master/src/plugins/score-plugin-midi/Midi/MidiView.cpp https://github.com/ossia/score/blob/master/src/plugins/score... https://github.com/ossia/score/blob/master/src/plugins/score-plugin-midi/Midi/MidiNoteView.cpp https://github.com/ossia/score/blob/master/src/plugins/score... - On top of all this, clips can contain text labels which can also be expensive to render. Yep, made myself a few "cached text" Qt items over time as the builtin cache wasn't satisfactory https://github.com/ossia/score/blob/master/src/lib/score/graphics/TextItem.hpp https://github.com/ossia/score/blob/master/src/lib/score/gra... - The fact that a timeline is zoom-able also makes it harder to cache the rendering of clips. If the timeline changed its zoom level, all visible clips pretty much have to redraw all of their contents. yep - Piano rolls can also be expensive to render if there is a bunch of MIDI notes, especially if there are text labels on the notes. yep - If the user clicks on a folder in a sample browser containing hundreds or even thousands of files, allocating a label widget for each file in the browser list will be very expensive. Something like the list factory in GTK is needed here. yep, Qt's also able to cache this. Though for instance for Qt's QFileSystemModel I carry a small patch to disable any kind of sorting when there's more than a few hundred thousand files (which happens for large media libraries): https://github.com/jcelerier/qtbase/commit/9909c3c7902cf7a2b52d24209f0d7511d4956f7c https://github.com/jcelerier/qtbase/commit/9909c3c7902cf7a2b... and also worked a bit with upstream Qt to get it to improve (for instance it was regenerating regexes ALL THE TIME when filtering for specific file extensions) - We want to reserve as much CPU as possible for the actual audio processing. Ideally the GUI shouldn't take up more than one or two CPU threads. - On some platforms, we also need to make sure there's actually enough CPU left for 3rd-party plugins to render their GUIs. yep
• gravitronic 4y ago
  I've built a modular midi / audio / video platform on the web, and encountered most of the pain points mentioned in the article. https://sequencer.party https://sequencer.party Some points: - you need to move all sequencing and audio / midi work off the main thread. I use WebAudioModules and sequence all midi in the audio thread as well. - you need to move all other heavy lifting off the main thread, for example for the multiplayer features I use YJS.. YJS runs in it's own worker to not stall the main thread - you don't update the DOM, you very carefully redraw parts of SVGs or better, canvases.
• sproketboy 4y ago
  [dead]
• BillyDM 4y ago
  The writer of the blogpost here. The popularity of this post has really caught me at an awkward time. I want to clarify some things: https://billydm.github.io/blog/clarifying-some-things/ https://billydm.github.io/blog/clarifying-some-things/
  • pkphilip 4y ago
    If you are interested in some help with the GUI, I can help
• EyeSayUntoThee 4y ago
  What about using the graphic bits from WDL? https://www.cockos.com/wdl/ https://www.cockos.com/wdl/
• chuchana 4y ago
  The DAW the post refers to: https://meadowlark.app https://meadowlark.app
  • chuchana 4y ago
    And its "design document": https://github.com/MeadowlarkDAW/Meadowlark/blob/main/DESIGN_DOC.md https://github.com/MeadowlarkDAW/Meadowlark/blob/main/DESIGN...