3 ms·

I've worked on big complex UI projects that needed to run on embedded hardware, using different tech. Some of my observations over the years, if it helps: * Th

by davedx 3y ago

I've worked on big complex UI projects that needed to run on embedded hardware, using different tech. Some of my observations over the years, if it helps: * The DOM isn't "slow" per se, but using the DOM for this kind of application is usually the wrong path to take. As soon as I need something even slightly off the well beaten path of web applications I tend to reach for canvas based solutions. * Libraries like EFL are quite nice to work with but also quite low level. By default that means C, but there are bindings for other languages. * Some of the things you describe (like rendering spectrographs) I'm not convinced are as CPU intensive as you think they are? Have you benchmarked? Winamp was doing these kind of visualizations back in the 90's. I kind of doubt if you really need to use shaders to render a typical DAW if you're smart with which parts need immediate mode rendering and which can use retained mode. Using render targets etc seems like a smart way to do a DAW, as most parts are actually quite static and only move around the screen. * I have a feeling - just a feeling - that a lot of people choose Rust for projects because it's one of the newer shinier languages, but as you've discovered, maybe that's not necessarily smart. I would choose Rust for its concurrency paradigm - which might be useful for a DAW! - but for building a UI I would actually personally choose something JavaScript-based. The productivity is just so much better than most other languages, for building UI's. This is not just my experience - I worked for some time at a large public listed US company who used JavaScript for all their big UI projects on embedded hardware, what they changed over time was the underlying rendering engine, but they kept JS because 1) hiring devs 2) productivity. Good luck, it's a super ambitious project, but I'm sure it's amazing fun to work on! :)

jerrre 3y ago
Regarding drawing analyzers, spectrographs: I feel it actually has become a lot more expensive to draw individual pixels to a screen with the cpu compared to 20yrs ago. Perhaps just because the resolutions are higher, but maybe the architecture (hardware or os) have just changed.
- cannam 3y ago
  With CPU, totally. As an example I can look at my own Sonic Visualiser application, largely written 15-18 years ago and entirely CPU-driven. Relative to then, it's now horrible on contemporary Macs for example - it feels far slower than it did a decade ago. It just isn't what the hardware expects. (There may be an element of toolkit-platform impedance and simple poor design on my part - it uses Qt and feels quicker on other platforms - and I don't want to argue the details here, but I think the basic principle that you really want to avoid CPU in the frame update is sound. Preparing things on a non-time-critical path via CPU should be another matter however, there's quite a lot of capacity there.)
  - deleted 3y ago
    [deleted]
  - deleted 3y ago
    [deleted]
  - p0nce 3y ago
    Unfortunately modern macOS renders everything at 5k and downsample, the rendering could do pretty significant stuff (like, extend bitdepth to 16-bit, color lookup to AppleRGB, trim bitdepth, with threads!) behind your back when using non-Metal, and it is buggier than before too. This stuff is way more expensive that CPU rendering itself.
  - klodolph 3y ago
    The “fast path” for macOS exists. I don’t know what is happening in Qt-land, but if you want to throw pixels at the screen really fast, you can do it through Core Animation. You can feed it a buffer of pixel data. My experience is that this is extremely fast.
    - davedx 3y ago
      Good to know about. This was also my intuition, that there must still exist ways to blast pixel data quickly, but it's not the "happy path" in modern graphics API's.
- davedx 3y ago
  Interesting. To be fair it's not something I've experimented with personally - I'm just extrapolating based on pure CPU clock speeds. I have definitely encountered the issue that it's hard to draw pixels using many modern graphics API's, but I would have thought there was a way to make it fast. I guess it depends on what you're drawing and what the primitives are too. Interesting topic, and another reason making a DAW must be a fun project!
- riceart 3y ago
  It isn’t really though. If this were true rendering text would be super slow - but it isn’t. Text shaping and rendering is done on the CPU on all the major platforms and is simply highly optimized. And people throw 15 to 20 years ago like it was a long time ago - but OSX Jaguar is more than 20 years old now (that’s the version that brought the current design of 2D desktop rendering with a compositing window manager to Mac OS X)
  - jcelerier 3y ago
    > It isn’t really though. If this were true rendering text would be super slow - but it isn’t. I wouldn't say so, I regularly profile my system or apps and text rendering is definitely one of the things that come up the most
    - riceart 3y ago
      > text rendering is definitely one of the things that come up the most Ok this doesn’t dispute what I’m saying at all and I would think you would know better. Do you have profiles from 15 or so years ago that show text rendering was a substantially smaller percentage of time. And if you do - ensure it is comparing apples to apples with subpixel (or at least greyscale) hinting and full featured shaping. Neither of these are related to blitting/blending performance but they are relatively expensive. Modern text rendering is super expensive (but by modern we’re talking on the order of 15 to 20 years, not yesterday). People are making claims that CPU 2D rendering has gotten more sluggish in that time frame - though nothing fundamental has changed other than the prevalence of “Retina” in some circles - but that isn’t something that has scaled past CPU performance. This all works because historically these systems have been painstaking about only drawing when needed what’s needed. If anything has changed it’s that - software in general for a number of reasons has gotten much sloppier about this. And in the case of Mac OS specifically - they continue to throw all kinds of shit in the CPU rendering pipeline. If you just do things yourself and blit to a Metal texture, nothing is fundamentally slower. I contend that most of what people are claiming are some fundamental hardware changes (that haven’t happened - things are faster across the board but otherwise it’s the same basic stuff) is due to less effective application software methods. For 2D desktop apps the GPU is not the factor some people seem to think it is.
bradrn 3y ago
> for building a UI I would actually personally choose something JavaScript-based The author does address this at [0]. Their main arguments are that web-based stuff is slow and requires lots of memory, especially when making something which runs outside a web browser. [0] https://billydm.github.io/blog/daw-frontend-development-struggles/#the-web-based-toolkits https://billydm.github.io/blog/daw-frontend-development-stru...
- davedx 3y ago
  Yeah so one tech stack I worked with was EFL with JavaScript bindings. That was fast and compact, even the JS stuff.
ricardobayes 3y ago
I'd think C++ and imgUI would be a good fit for this work but many other approaches could work fine.
duped 3y ago
I've done some spectrogram stuff in the past, and the bottleneck (eg in JUCE where you'd want to do this kind of thing) is in the path rendering. It's pretty cheap to compute a big FFT, even throwing it on separate thread if you care enough for that. Rendering an anti-aliased path that looks decent on a small display panel with the basic vector path tools however isn't great. Not saying it can't be done, because people do it. But making something that compares to the latest/greatest and is fast isn't obvious.
pier25 3y ago
> I worked for some time at a large public listed US company who used JavaScript for all their big UI projects Could you elaborate? Did you use a custom framework or maybe something like React or Svelte? Maybe pure vanilla?
pier25 3y ago
I agree web stuff is really the best way to develop UIs. Good luck making responsive stuff in C++ for example. The paradigm of HTML, CSS, and JS is extremely powerful and even allows you to use canvas, webgpu, wasm. There are multiple commercial projects that use web dev paradigm for GUIs: https://coherent-labs.com/ https://coherent-labs.com/ https://ultralig.ht/ https://ultralig.ht/ https://sciter.com/ https://sciter.com/

by davedx 3y ago

jerrre 3y ago

Regarding drawing analyzers, spectrographs: I feel it actually has become a lot more expensive to draw individual pixels to a screen with the cpu compared to 20yrs ago. Perhaps just because the resolutions are higher, but maybe the architecture (hardware or os) have just changed.

cannam 3y ago
With CPU, totally. As an example I can look at my own Sonic Visualiser application, largely written 15-18 years ago and entirely CPU-driven. Relative to then, it's now horrible on contemporary Macs for example - it feels far slower than it did a decade ago. It just isn't what the hardware expects. (There may be an element of toolkit-platform impedance and simple poor design on my part - it uses Qt and feels quicker on other platforms - and I don't want to argue the details here, but I think the basic principle that you really want to avoid CPU in the frame update is sound. Preparing things on a non-time-critical path via CPU should be another matter however, there's quite a lot of capacity there.)
- deleted 3y ago
  [deleted]
- deleted 3y ago
  [deleted]
- p0nce 3y ago
  Unfortunately modern macOS renders everything at 5k and downsample, the rendering could do pretty significant stuff (like, extend bitdepth to 16-bit, color lookup to AppleRGB, trim bitdepth, with threads!) behind your back when using non-Metal, and it is buggier than before too. This stuff is way more expensive that CPU rendering itself.
- klodolph 3y ago
  The “fast path” for macOS exists. I don’t know what is happening in Qt-land, but if you want to throw pixels at the screen really fast, you can do it through Core Animation. You can feed it a buffer of pixel data. My experience is that this is extremely fast.
  - davedx 3y ago
    Good to know about. This was also my intuition, that there must still exist ways to blast pixel data quickly, but it's not the "happy path" in modern graphics API's.
davedx 3y ago
Interesting. To be fair it's not something I've experimented with personally - I'm just extrapolating based on pure CPU clock speeds. I have definitely encountered the issue that it's hard to draw pixels using many modern graphics API's, but I would have thought there was a way to make it fast. I guess it depends on what you're drawing and what the primitives are too. Interesting topic, and another reason making a DAW must be a fun project!
riceart 3y ago
It isn’t really though. If this were true rendering text would be super slow - but it isn’t. Text shaping and rendering is done on the CPU on all the major platforms and is simply highly optimized. And people throw 15 to 20 years ago like it was a long time ago - but OSX Jaguar is more than 20 years old now (that’s the version that brought the current design of 2D desktop rendering with a compositing window manager to Mac OS X)
- jcelerier 3y ago
  > It isn’t really though. If this were true rendering text would be super slow - but it isn’t. I wouldn't say so, I regularly profile my system or apps and text rendering is definitely one of the things that come up the most
  - riceart 3y ago
    > text rendering is definitely one of the things that come up the most Ok this doesn’t dispute what I’m saying at all and I would think you would know better. Do you have profiles from 15 or so years ago that show text rendering was a substantially smaller percentage of time. And if you do - ensure it is comparing apples to apples with subpixel (or at least greyscale) hinting and full featured shaping. Neither of these are related to blitting/blending performance but they are relatively expensive. Modern text rendering is super expensive (but by modern we’re talking on the order of 15 to 20 years, not yesterday). People are making claims that CPU 2D rendering has gotten more sluggish in that time frame - though nothing fundamental has changed other than the prevalence of “Retina” in some circles - but that isn’t something that has scaled past CPU performance. This all works because historically these systems have been painstaking about only drawing when needed what’s needed. If anything has changed it’s that - software in general for a number of reasons has gotten much sloppier about this. And in the case of Mac OS specifically - they continue to throw all kinds of shit in the CPU rendering pipeline. If you just do things yourself and blit to a Metal texture, nothing is fundamentally slower. I contend that most of what people are claiming are some fundamental hardware changes (that haven’t happened - things are faster across the board but otherwise it’s the same basic stuff) is due to less effective application software methods. For 2D desktop apps the GPU is not the factor some people seem to think it is.

bradrn 3y ago

> for building a UI I would actually personally choose something JavaScript-based The author does address this at [0]. Their main arguments are that web-based stuff is slow and requires lots of memory, especially when making something which runs outside a web browser. [0] https://billydm.github.io/blog/daw-frontend-development-struggles/#the-web-based-toolkits https://billydm.github.io/blog/daw-frontend-development-stru...

davedx 3y ago
Yeah so one tech stack I worked with was EFL with JavaScript bindings. That was fast and compact, even the JS stuff.

ricardobayes 3y ago

I'd think C++ and imgUI would be a good fit for this work but many other approaches could work fine.

duped 3y ago

I've done some spectrogram stuff in the past, and the bottleneck (eg in JUCE where you'd want to do this kind of thing) is in the path rendering. It's pretty cheap to compute a big FFT, even throwing it on separate thread if you care enough for that. Rendering an anti-aliased path that looks decent on a small display panel with the basic vector path tools however isn't great. Not saying it can't be done, because people do it. But making something that compares to the latest/greatest and is fast isn't obvious.

pier25 3y ago

> I worked for some time at a large public listed US company who used JavaScript for all their big UI projects Could you elaborate? Did you use a custom framework or maybe something like React or Svelte? Maybe pure vanilla?

pier25 3y ago

I agree web stuff is really the best way to develop UIs. Good luck making responsive stuff in C++ for example. The paradigm of HTML, CSS, and JS is extremely powerful and even allows you to use canvas, webgpu, wasm. There are multiple commercial projects that use web dev paradigm for GUIs: https://coherent-labs.com/ https://coherent-labs.com/ https://ultralig.ht/ https://ultralig.ht/ https://sciter.com/ https://sciter.com/