10 ms·

Using a Linux phone as a secondary monitor

172 points by cunidev 4y ago

ur-whale 4y ago
Great hack. Would be even neater if I could stream music to the phone's sound system from my desktop.
- ben-schaaf 4y ago
  Pulseaudio lets you turn any device into a speaker over Bluetooth or IP.
  - aidenn0 4y ago
    Snapchat is good too, and already has Android clients
    - radus 4y ago
      *snapcast: https://github.com/badaix/snapcast https://github.com/badaix/snapcast
      - aidenn0 4y ago
        grr... that was from my phone and autocorrect changed it; too late to edit now.
  - michaelmrose 4y ago
    Tried this with an android client and experienced substantial latency despite working just fine between 2 computers in the same lan. Mpd seemed to work better.
- seba_dos1 4y ago
  That's super easy already. You can make the phone act as a USB sound card, or you can use PulseAudio's network transparency (see https://puri.sm/posts/speak-to-me/ https://puri.sm/posts/speak-to-me/), or you can broadcast an RTP stream, or you can use Bluetooth audio...
squarefoot 4y ago
That low latency is an impressive achievement, makes me want to attempt to convert all my old tablets to a real Linux install (no chroot jails etc), but the risk of bricking them is really high. I wish there was some tool to probe the hardware under Android, possibly against a distributed database, to know at least if the device is similar to a more known one (it could be the case among no name cheap Chinese tablets) and if there are chances of success before a blind attempt.
- usrn 4y ago
  Most SOCs have a mode you can put them into with just the hardware switches that runs firmware in the mask ROM which waits for a bootloader over USB. It's pretty hard to actually brick modern devices.
  - mschuster91 4y ago
    Yeah, but most non-brand manufacturers don't release the tools, driver packages or the full firmware images at all - meaning you have to rely on questionable third party sites, hackers and leakers to obtain them to recover your device. I don't even want to know how many people ended up with malware from shoddy SEO-spam domains advertising Samsung's ODIN tool. If Google had forced everyone to use the aboot boot loader and to allow rooting, we wouldn't be in this damn mess of people using all kinds of highly questionable stuff.
    - bitwize 4y ago
      The carriers wouldn't allow that, though. The carriers only permitted the iPhone because Apple guaranteed it would be locked down. I am, of course, talking about the USA. I believe EU law forbids cellular carriers from being so thickheaded.
- xen2xen1 4y ago
  "distrubted database". What XDA isn't good enough?
- imtringued 4y ago
bluedays 4y ago
Wish that I could use my iPad as a second screen. I’m even tempted to get a mac just for this feature
- spicybright 4y ago
  There's definitely ways you can do that. Look into programs that use HDMI dummy plugs. It's a little stub that plugs into an HDMI port trick the OS into rendering a separate screen. Then there are a lot of apps that let you display/interact with that virtual screen, usually through some kind of browser. Acts exactly like any monitor you plug in, regardless of OS. It seems silly to need hardware for this, but it's a very clean/stable way to do this without mucking with config files or using weird hacks.
  - raihansaputra 4y ago
    You don't even need dummy plugs with macOs now. You can use BetterDummy instead
    - spicybright 4y ago
      Finally! I'll have to try that out.
- 867-5309 4y ago
  while you're at it, get another Tesla for a second windscreen!
- diffeomorphism 4y ago
  > Wish that I could use my iPad as a second screen. No wishing necessary. Before apple sherlocked the idea with sidecar there have been multiple apps that let you do that and are (somewhat) cross-platform (Luna display, duet, ...). These apps are still around and much better than the apple one for specialized use cases like art, they cost money though. Or, if you don't need high speed, just go the simple vnc route, e.g. https://kbumsik.io/using-ipad-as-a-2nd-monitor-on-linux https://kbumsik.io/using-ipad-as-a-2nd-monitor-on-linux
askz 4y ago
I actually achieved the same, with benefits. The hack is here: https://github.com/any1/wayvnc/issues/7#issuecomment-657092231 https://github.com/any1/wayvnc/issues/7#issuecomment-6570922... Benefits are: two way interaction :)
throw93232 4y ago
It is pretty easy to do that with X11 and VNC.
- bitwize 4y ago
  No. It's not. Fast video playback and low latency was a requirement, and you can't do that over the wire with X11 or VNC. Pipewire is the solution for this. This is why SirCmpwn was right, and the anti-Wayland crowd needs to just shut up. Wayland+Pipewire is the future of remote desktop. Together they solve issues X11 by its nature cannot solve.
  - yjftsjthsd-h 4y ago
    What about X makes it fundamentally incapable of sending pixels off device at low latency?
    - bitwize 4y ago
      The X protocol is extremely chatty, does not compress images, and uses TCP. It's virtually impossible to make any sort of latency guarantees with that sort of protocol. Pipewire can use several compressed formats and the OP used UDP to stream the media off Pipewire to the remote display.
      - yjftsjthsd-h 4y ago
        Yeah, I know running X11 over the wire doesn't hold up, but that's not what's being done here with Wayland either; since we're locally grabbing pixels off the virtual display and then flinging them over the wire via custom UDP protocol, X11 vs pipewire only matters for the initial handover.
        seba_dos1 4y ago
        The compositor may be able to transfer image data to PipeWire in zero-copy fashion, which may then be consumed by hardware video encoder without any copy on the path. I'm not sure whether you'd be able to do that using Xorg as display server.
kache_ 4y ago
Imagine if you could do this with any phone tablet. Why can't you?
- askz 4y ago
  you actually can, with vnc and sway https://github.com/any1/wayvnc/issues/7#issuecomment-6570922 https://github.com/any1/wayvnc/issues/7#issuecomment-6570922...
  - silicon2401 4y ago
    does that with any phone, or just any tablet? It would be interesting to use old phones for media control or something
    - diffeomorphism 4y ago
      Any phone, tablet, fridge, toaster, ... that can run a vnc app.
mxuribe 4y ago
This is great! In my mind, i wish for KDE Connect to eventually evolve to enable something like this...KDE Connect is already a wonderful piece of software!
- ognarb 4y ago
  You mean this https://invent.kde.org/network/kdeconnect-kde/-/merge_requests/427 https://invent.kde.org/network/kdeconnect-kde/-/merge_reques...? :)
  - mxuribe 4y ago
    Yay!!! :-)
rcarmo 4y ago
I like the UDP approach. I tried doing something similar to multicast video streams to Raspberry Pi 1s quite a while ago (back when it was impossible to run a modern browser on them and we wanted live dashboards across the building), and it can be very efficient.
thomond 4y ago
> Virtual display handling: creating a fake video output. On Xorg, some GPU-specific hacks exist (probably Intel-only) Nvidia users are out of luck in my experience.
luke-stanley 4y ago
I didn't see Xpra mentioned, but Xpra does desktop and window streaming with fast video encoding using H.265 (or H.264 and others options), and it has an experimental UDP transport - though I haven't needed it yet. In the Windows world, gamers use Parsec which matches those two network transport and encoding design decisions (which I am sure are the right direction). To use Xpra as part of a secondary monitor solution, you might need to define a "dummy" window region for it, and I suspect X server can be configured to allow this but I don't know how yet. I'm interested in any solutions for Linux secondary monitor devices too, especially more simple ones. Edit: kbumsik link below uses xrandr to add a new display region and then uses x11vnc with a "clip" argument. I couldn't see such an argument for xrandr.
- cunidev 4y ago
  Tried Xpra ~2 years ago, but unfortunately it would not work for me, and it was Xorg-only at least at the time
  - kzrdude 4y ago
    Still is X-only.
- megous 4y ago
  This would work nicely on Pinephone, because it has very efficient accelerated decoding of H.264 codec. gstreamer using kmssink consumes very little power decoding H.264 video. It would likely run at ~2W with display on, so for about 5 hours on battery (at reasonable screen brightness)
deleted 4y ago
[deleted]
butz 4y ago
What about standard, probably proprietary, solutions, like Miracast ("Wireless Display")? I wanted to use my old tablet with Windows as secondary monitor, but no luck when running Linux on primary PC. Best option is only mirroring desktop with GNOME Network Displays. It would be great if there would be out of the box support for wireless displays on Linux.
- samstave 4y ago
  Im surprised Figma doesn't allw for this... only because their app dev kit with phone mirroring desktop is pretty sweet... I could see how it coud be adapted for this purpose
- cunidev 4y ago
  Not only Miracast, but even Chromecast had >700ms latency in my experience :(
KerrickStaley 4y ago
A related idea I had a while ago: I think it'd be cool if you could use a cell phone as a mouse. You'd have a mount that holds the back of the phone slightly off the table, and the phone would track its position using the IMU and maybe motion tracking with the camera. The phone display could show a subregion of the monitor and you could click by tapping the phone screen.
- themodelplumber 4y ago
  A little bit different but I hear the Sony VAIO VN-CX1 clearing its throat somewhere in the periphery...
- lordofgibbons 4y ago
  You can already do this with KDE Connect on your phone. And you don't need to be running KDE for it to work on your desktop - it works with Gnome too.
  - KerrickStaley 4y ago
    Could you give a link to docs on that? I visited https://kdeconnect.kde.org/ https://kdeconnect.kde.org/ but didn't see this feature mentioned.
    - sudosysgen 4y ago
      I dont know if its in the docs but it definitely works (just tested it)
    - pronoiac 4y ago
      I think it's treating the phone screen as a touchpad, not moving the phone like a mouse - https://userbase.kde.org/KDEConnect#Virtual_Input https://userbase.kde.org/KDEConnect#Virtual_Input
- kzrdude 4y ago
  Does this include using the phone as a touchpad? That would be cool and useful.
KingVase 4y ago
Very nice setup. I wish it were possible with KDE Connect.
jrumbut 4y ago
Linux phone is getting so exciting! I wish there was a way to dual boot or have something like a live CD to get into it in a low risk way.
- megous 4y ago
  There's a way to multi-boot: https://xnux.eu/p-boot-demo/ https://xnux.eu/p-boot-demo/ (using a Pinephone bootloader) You can also multi-boot using kexec (using Linux to boot other Linux image) or just by booting into initramfs that has a GUI program for selecting how to continue boot (which root filesystem to start from), etc.
follower 4y ago
If you haven't already checked it out, I'd suggest investigating use of `evdi` ("Extensible Virtual Display Interface") for the virtual display aspect of this: * https://github.com/DisplayLink/evdi https://github.com/DisplayLink/evdi * https://github.com/rhofour/evdi-vnc https://github.com/rhofour/evdi-vnc

172 points by cunidev 4y ago

ur-whale 4y ago

Great hack. Would be even neater if I could stream music to the phone's sound system from my desktop.

ben-schaaf 4y ago
Pulseaudio lets you turn any device into a speaker over Bluetooth or IP.
- aidenn0 4y ago
  Snapchat is good too, and already has Android clients
  - radus 4y ago
    *snapcast: https://github.com/badaix/snapcast https://github.com/badaix/snapcast
    - aidenn0 4y ago
      grr... that was from my phone and autocorrect changed it; too late to edit now.
- michaelmrose 4y ago
  Tried this with an android client and experienced substantial latency despite working just fine between 2 computers in the same lan. Mpd seemed to work better.
seba_dos1 4y ago
That's super easy already. You can make the phone act as a USB sound card, or you can use PulseAudio's network transparency (see https://puri.sm/posts/speak-to-me/ https://puri.sm/posts/speak-to-me/), or you can broadcast an RTP stream, or you can use Bluetooth audio...

squarefoot 4y ago

That low latency is an impressive achievement, makes me want to attempt to convert all my old tablets to a real Linux install (no chroot jails etc), but the risk of bricking them is really high. I wish there was some tool to probe the hardware under Android, possibly against a distributed database, to know at least if the device is similar to a more known one (it could be the case among no name cheap Chinese tablets) and if there are chances of success before a blind attempt.

usrn 4y ago
Most SOCs have a mode you can put them into with just the hardware switches that runs firmware in the mask ROM which waits for a bootloader over USB. It's pretty hard to actually brick modern devices.
- mschuster91 4y ago
  Yeah, but most non-brand manufacturers don't release the tools, driver packages or the full firmware images at all - meaning you have to rely on questionable third party sites, hackers and leakers to obtain them to recover your device. I don't even want to know how many people ended up with malware from shoddy SEO-spam domains advertising Samsung's ODIN tool. If Google had forced everyone to use the aboot boot loader and to allow rooting, we wouldn't be in this damn mess of people using all kinds of highly questionable stuff.
  - bitwize 4y ago
    The carriers wouldn't allow that, though. The carriers only permitted the iPhone because Apple guaranteed it would be locked down. I am, of course, talking about the USA. I believe EU law forbids cellular carriers from being so thickheaded.
xen2xen1 4y ago
"distrubted database". What XDA isn't good enough?
imtringued 4y ago

bluedays 4y ago

Wish that I could use my iPad as a second screen. I’m even tempted to get a mac just for this feature

spicybright 4y ago
There's definitely ways you can do that. Look into programs that use HDMI dummy plugs. It's a little stub that plugs into an HDMI port trick the OS into rendering a separate screen. Then there are a lot of apps that let you display/interact with that virtual screen, usually through some kind of browser. Acts exactly like any monitor you plug in, regardless of OS. It seems silly to need hardware for this, but it's a very clean/stable way to do this without mucking with config files or using weird hacks.
- raihansaputra 4y ago
  You don't even need dummy plugs with macOs now. You can use BetterDummy instead
  - spicybright 4y ago
    Finally! I'll have to try that out.
867-5309 4y ago
while you're at it, get another Tesla for a second windscreen!
diffeomorphism 4y ago
> Wish that I could use my iPad as a second screen. No wishing necessary. Before apple sherlocked the idea with sidecar there have been multiple apps that let you do that and are (somewhat) cross-platform (Luna display, duet, ...). These apps are still around and much better than the apple one for specialized use cases like art, they cost money though. Or, if you don't need high speed, just go the simple vnc route, e.g. https://kbumsik.io/using-ipad-as-a-2nd-monitor-on-linux https://kbumsik.io/using-ipad-as-a-2nd-monitor-on-linux

askz 4y ago

I actually achieved the same, with benefits. The hack is here: https://github.com/any1/wayvnc/issues/7#issuecomment-657092231 https://github.com/any1/wayvnc/issues/7#issuecomment-6570922... Benefits are: two way interaction :)

throw93232 4y ago

It is pretty easy to do that with X11 and VNC.

bitwize 4y ago
No. It's not. Fast video playback and low latency was a requirement, and you can't do that over the wire with X11 or VNC. Pipewire is the solution for this. This is why SirCmpwn was right, and the anti-Wayland crowd needs to just shut up. Wayland+Pipewire is the future of remote desktop. Together they solve issues X11 by its nature cannot solve.
- yjftsjthsd-h 4y ago
  What about X makes it fundamentally incapable of sending pixels off device at low latency?
  - bitwize 4y ago
    The X protocol is extremely chatty, does not compress images, and uses TCP. It's virtually impossible to make any sort of latency guarantees with that sort of protocol. Pipewire can use several compressed formats and the OP used UDP to stream the media off Pipewire to the remote display.
    - yjftsjthsd-h 4y ago
      Yeah, I know running X11 over the wire doesn't hold up, but that's not what's being done here with Wayland either; since we're locally grabbing pixels off the virtual display and then flinging them over the wire via custom UDP protocol, X11 vs pipewire only matters for the initial handover.
      - seba_dos1 4y ago
        The compositor may be able to transfer image data to PipeWire in zero-copy fashion, which may then be consumed by hardware video encoder without any copy on the path. I'm not sure whether you'd be able to do that using Xorg as display server.

kache_ 4y ago

Imagine if you could do this with any phone tablet. Why can't you?

askz 4y ago
you actually can, with vnc and sway https://github.com/any1/wayvnc/issues/7#issuecomment-6570922 https://github.com/any1/wayvnc/issues/7#issuecomment-6570922...
- silicon2401 4y ago
  does that with any phone, or just any tablet? It would be interesting to use old phones for media control or something
  - diffeomorphism 4y ago
    Any phone, tablet, fridge, toaster, ... that can run a vnc app.

mxuribe 4y ago

This is great! In my mind, i wish for KDE Connect to eventually evolve to enable something like this...KDE Connect is already a wonderful piece of software!

ognarb 4y ago
You mean this https://invent.kde.org/network/kdeconnect-kde/-/merge_requests/427 https://invent.kde.org/network/kdeconnect-kde/-/merge_reques...? :)
- mxuribe 4y ago
  Yay!!! :-)

rcarmo 4y ago

I like the UDP approach. I tried doing something similar to multicast video streams to Raspberry Pi 1s quite a while ago (back when it was impossible to run a modern browser on them and we wanted live dashboards across the building), and it can be very efficient.

thomond 4y ago

> Virtual display handling: creating a fake video output. On Xorg, some GPU-specific hacks exist (probably Intel-only) Nvidia users are out of luck in my experience.

luke-stanley 4y ago

I didn't see Xpra mentioned, but Xpra does desktop and window streaming with fast video encoding using H.265 (or H.264 and others options), and it has an experimental UDP transport - though I haven't needed it yet. In the Windows world, gamers use Parsec which matches those two network transport and encoding design decisions (which I am sure are the right direction). To use Xpra as part of a secondary monitor solution, you might need to define a "dummy" window region for it, and I suspect X server can be configured to allow this but I don't know how yet. I'm interested in any solutions for Linux secondary monitor devices too, especially more simple ones. Edit: kbumsik link below uses xrandr to add a new display region and then uses x11vnc with a "clip" argument. I couldn't see such an argument for xrandr.

cunidev 4y ago
Tried Xpra ~2 years ago, but unfortunately it would not work for me, and it was Xorg-only at least at the time
- kzrdude 4y ago
  Still is X-only.
megous 4y ago
This would work nicely on Pinephone, because it has very efficient accelerated decoding of H.264 codec. gstreamer using kmssink consumes very little power decoding H.264 video. It would likely run at ~2W with display on, so for about 5 hours on battery (at reasonable screen brightness)

deleted 4y ago

[deleted]

butz 4y ago

What about standard, probably proprietary, solutions, like Miracast ("Wireless Display")? I wanted to use my old tablet with Windows as secondary monitor, but no luck when running Linux on primary PC. Best option is only mirroring desktop with GNOME Network Displays. It would be great if there would be out of the box support for wireless displays on Linux.

samstave 4y ago
Im surprised Figma doesn't allw for this... only because their app dev kit with phone mirroring desktop is pretty sweet... I could see how it coud be adapted for this purpose
cunidev 4y ago
Not only Miracast, but even Chromecast had >700ms latency in my experience :(

KerrickStaley 4y ago

A related idea I had a while ago: I think it'd be cool if you could use a cell phone as a mouse. You'd have a mount that holds the back of the phone slightly off the table, and the phone would track its position using the IMU and maybe motion tracking with the camera. The phone display could show a subregion of the monitor and you could click by tapping the phone screen.

themodelplumber 4y ago
A little bit different but I hear the Sony VAIO VN-CX1 clearing its throat somewhere in the periphery...
lordofgibbons 4y ago
You can already do this with KDE Connect on your phone. And you don't need to be running KDE for it to work on your desktop - it works with Gnome too.
- KerrickStaley 4y ago
  Could you give a link to docs on that? I visited https://kdeconnect.kde.org/ https://kdeconnect.kde.org/ but didn't see this feature mentioned.
  - sudosysgen 4y ago
    I dont know if its in the docs but it definitely works (just tested it)
  - pronoiac 4y ago
    I think it's treating the phone screen as a touchpad, not moving the phone like a mouse - https://userbase.kde.org/KDEConnect#Virtual_Input https://userbase.kde.org/KDEConnect#Virtual_Input
kzrdude 4y ago
Does this include using the phone as a touchpad? That would be cool and useful.

KingVase 4y ago

Very nice setup. I wish it were possible with KDE Connect.

jrumbut 4y ago

Linux phone is getting so exciting! I wish there was a way to dual boot or have something like a live CD to get into it in a low risk way.

megous 4y ago
There's a way to multi-boot: https://xnux.eu/p-boot-demo/ https://xnux.eu/p-boot-demo/ (using a Pinephone bootloader) You can also multi-boot using kexec (using Linux to boot other Linux image) or just by booting into initramfs that has a GUI program for selecting how to continue boot (which root filesystem to start from), etc.

follower 4y ago

If you haven't already checked it out, I'd suggest investigating use of `evdi` ("Extensible Virtual Display Interface") for the virtual display aspect of this: * https://github.com/DisplayLink/evdi https://github.com/DisplayLink/evdi * https://github.com/rhofour/evdi-vnc https://github.com/rhofour/evdi-vnc