5 ms·

They can already do this with the desktop composition software they use today. All the windows are virtualized onto backing layers that you can draw anywhere an

They can already do this with the desktop composition software they use today. All the windows are virtualized onto backing layers that you can draw anywhere and add effects to. It’s how window shadows work, and how certain window effects are done. They just haven’t done it.

ncr100 3y ago
So Apple could have done this, but did not. Why? (Speculation and leaks welcome) - [Profit on basic innovation] Did they want to wait and see how their customers would adopt VisionOS's native free-floating windows, so as to avoid cognitive overload by commingling with MacOS windows? - [Benevolence to fellow competitors] Did they not want to takeover the existing market of virtualized VR desktops?
- theultdev 3y ago
  Most likely because it wasn't polished, so get it out now in a polished and limited state and now you have a fancy update to tout when it is done.
- realslimjd 3y ago
  I'm reasonably certain it's a combination bandwidth and tech issue. The Vision Pro is effectively using AirPlay to mirror the whole screen. If you used AirPlay to mirror each window as a whole screen, you'd run out of bandwidth pretty quickly. The windowing system in MacOS, Quartz Compositor, also isn't built to stream window information. Right now it has a big built in assumption that any windows its displaying are on a screen it also controls. It was probably too big a lift across teams to also re-write the graphics stack for MacOS for the launch of the Vision Pro. Hopefully they get it working in the future, but neither of these problems are easy to solve.
  - NoPedantsThanks 3y ago
    [dead]
  - ncr100 3y ago
    Makes a lot of sense, future will solve it someday.
cududa 3y ago
I'm aware. I've worked in the space. It isn't as simple as you're making it out.
- refulgentis 3y ago
  Yes, it is. In fact, there's already a project for it on GitHub. https://github.com/saagarjha/Ensemble https://github.com/saagarjha/Ensemble 480 lines total, including comments, headers, whole shebang.
  - chaos_emergent 3y ago
    > Ensemble is currently "pre-alpha": it's really more of a demo at this point. There's a lot of things I need to work on and until then it is unlikely I will be taking any code contributions.... The code is definitely not designed for general-purpose use yet, so don't expect much of it :)
  - threeseed 3y ago
    Tech demos are often easy to put together. It's all of the edge cases and UX refinements that takes time.
    - refulgentis 3y ago
      Yeah, sure. I'm gonna go ahead and say Apple probably could have found a way to ship this over the visionOS dev cycle.
  - saagarjha 3y ago
    Some of the code is factored out into individual libraries. For example the networking and serialization code is separate and it is likely that screen recording will get pulled out too at some point.
- AshamedCaptain 3y ago
  Sorry but almost 10 years ago I could do this on Xorg where the worst problem is that compositors cannot redirect input (so you had to kludge new events from scratch). I cannot imagine it would take more than _half an hour_ for someone with macOS display compositor experience to implement it.
brookst 3y ago
Yes? The first iphone didn’t have copy/paste. Apple will always prioritize critical scenarios over nice to have. None of these things are technically difficult, it’s just time. I’m willing to believe they released too early, but at some point you have to start learning from real users.
gorkish 3y ago
> They just haven’t done it. Literally nobody has done it. It's beyond ridiculous that you can't already show or duplicate an application window on any display you want and allow it to be controlled from anywhere it is visible. Searching for ways to do this lead one into extremely niche software ecosystems. Please is there any collaboration app out there that makes it seamless to toss windows around like everyone actually wants?
- joshspankit 3y ago
  I feel like multiple people did it back in the original X11 days, and almost certainly when compiz was the new hotness
  - jonhohle 3y ago
    Nearly 20 years ago there was an OS X screensaver that would capture all the window buffers and then float them around the screen as they rotated various ways. Another app would save a snapshot of all windows as Photoshop layers with appropriate transparency (before blur was added). X11 was always network transparent, though I don’t think windows could easily be moved from one client to another. Even classic Mac OS, which wasn’t designed to be “rootless” could be run that way could have its windows mixed with OS X windows in the classic days. As others have pointed out, there is already a PoC app that is doing it, so it seems if Apple wants to, it’s completely in their power. However, does this match their vision? (pun intended) Time will tell.
- jandrese 3y ago
  Isn't this how ordinary spanning monitors works? It might be slightly awkward with AR goggles since the relative orientation of the displays will be constantly changing as your head moves, and what happens to a window you have half on and half off of the Macbook's screen when you look away? Or do you want to have the application jump between devices, like appearing on your fridge when you go for a drink? With the old X11 protocol and a daemon in the middle this was possible but the use cases were extremely limited and the security issues made it a pain in the ass to actually use. With distros moving away from X11 this is only going to get harder, and you have to ask yourself how much you really want it. This would mean the goggles would be basically just a dumb display for the Mac. It would also be weird to try to move an AR app onto your Mac.

ncr100 3y ago

So Apple could have done this, but did not. Why? (Speculation and leaks welcome) - [Profit on basic innovation] Did they want to wait and see how their customers would adopt VisionOS's native free-floating windows, so as to avoid cognitive overload by commingling with MacOS windows? - [Benevolence to fellow competitors] Did they not want to takeover the existing market of virtualized VR desktops?

theultdev 3y ago
Most likely because it wasn't polished, so get it out now in a polished and limited state and now you have a fancy update to tout when it is done.
realslimjd 3y ago
I'm reasonably certain it's a combination bandwidth and tech issue. The Vision Pro is effectively using AirPlay to mirror the whole screen. If you used AirPlay to mirror each window as a whole screen, you'd run out of bandwidth pretty quickly. The windowing system in MacOS, Quartz Compositor, also isn't built to stream window information. Right now it has a big built in assumption that any windows its displaying are on a screen it also controls. It was probably too big a lift across teams to also re-write the graphics stack for MacOS for the launch of the Vision Pro. Hopefully they get it working in the future, but neither of these problems are easy to solve.
- NoPedantsThanks 3y ago
  [dead]
- ncr100 3y ago
  Makes a lot of sense, future will solve it someday.

cududa 3y ago

I'm aware. I've worked in the space. It isn't as simple as you're making it out.

refulgentis 3y ago
Yes, it is. In fact, there's already a project for it on GitHub. https://github.com/saagarjha/Ensemble https://github.com/saagarjha/Ensemble 480 lines total, including comments, headers, whole shebang.
- chaos_emergent 3y ago
  > Ensemble is currently "pre-alpha": it's really more of a demo at this point. There's a lot of things I need to work on and until then it is unlikely I will be taking any code contributions.... The code is definitely not designed for general-purpose use yet, so don't expect much of it :)
- threeseed 3y ago
  Tech demos are often easy to put together. It's all of the edge cases and UX refinements that takes time.
  - refulgentis 3y ago
    Yeah, sure. I'm gonna go ahead and say Apple probably could have found a way to ship this over the visionOS dev cycle.
- saagarjha 3y ago
  Some of the code is factored out into individual libraries. For example the networking and serialization code is separate and it is likely that screen recording will get pulled out too at some point.
AshamedCaptain 3y ago
Sorry but almost 10 years ago I could do this on Xorg where the worst problem is that compositors cannot redirect input (so you had to kludge new events from scratch). I cannot imagine it would take more than _half an hour_ for someone with macOS display compositor experience to implement it.

brookst 3y ago

Yes? The first iphone didn’t have copy/paste. Apple will always prioritize critical scenarios over nice to have. None of these things are technically difficult, it’s just time. I’m willing to believe they released too early, but at some point you have to start learning from real users.

gorkish 3y ago

> They just haven’t done it. Literally nobody has done it. It's beyond ridiculous that you can't already show or duplicate an application window on any display you want and allow it to be controlled from anywhere it is visible. Searching for ways to do this lead one into extremely niche software ecosystems. Please is there any collaboration app out there that makes it seamless to toss windows around like everyone actually wants?

joshspankit 3y ago
I feel like multiple people did it back in the original X11 days, and almost certainly when compiz was the new hotness
- jonhohle 3y ago
  Nearly 20 years ago there was an OS X screensaver that would capture all the window buffers and then float them around the screen as they rotated various ways. Another app would save a snapshot of all windows as Photoshop layers with appropriate transparency (before blur was added). X11 was always network transparent, though I don’t think windows could easily be moved from one client to another. Even classic Mac OS, which wasn’t designed to be “rootless” could be run that way could have its windows mixed with OS X windows in the classic days. As others have pointed out, there is already a PoC app that is doing it, so it seems if Apple wants to, it’s completely in their power. However, does this match their vision? (pun intended) Time will tell.
jandrese 3y ago
Isn't this how ordinary spanning monitors works? It might be slightly awkward with AR goggles since the relative orientation of the displays will be constantly changing as your head moves, and what happens to a window you have half on and half off of the Macbook's screen when you look away? Or do you want to have the application jump between devices, like appearing on your fridge when you go for a drink? With the old X11 protocol and a daemon in the middle this was possible but the use cases were extremely limited and the security issues made it a pain in the ass to actually use. With distros moving away from X11 this is only going to get harder, and you have to ask yourself how much you really want it. This would mean the goggles would be basically just a dumb display for the Mac. It would also be weird to try to move an AR app onto your Mac.