3 ms·

Scrpy is awesome and one of those rare pieces of software that has a clear and powerful thing to do and does it so well you get to forget it's even there. Essen

Scrpy is awesome and one of those rare pieces of software that has a clear and powerful thing to do and does it so well you get to forget it's even there. Essentially you get the feeling that this is how it should work. But at the same time I find it a bit sad example of how much effort to and how much corner case fiddling such conceptually very simple tasks have to take. Caricatyring here, but it is essentially just blitting pixels over a network connection. I don't think it should be that hard a task with a more reasonable underlying stack. Plan 9 (and probably many other sadly forgotten systems) showed how something like scrcpy can fall out more or less automatically from a better designed system. Maybe I'm underestimating some fundamental complications, but I can't help but feel something has gone very wrong with how we do things with computers. Maybe something like worse is better on piles of previous worse is better.

jayd16 3y ago
I think you're taking for granted just how performance intensive this is. For example, "Just a blit" is a full screen draw worth of pixel operations on a hardware platform that is known to have pixel draw at a premium.
- actionfromafar 3y ago
  I don't understand this comment. It's a marvel of course how far we have come, but what today is so intensive? The (old!) Apple iPad cable runs BSD Unix and does real time video encoding.
  - refulgentis 3y ago
    Correct, you're violently agreeing, _they embed a whole *NIX computer in the cable to make it happen_. They were replying to someone saying why does everyone make it so complicated. Refer to my comment for the math on why its intensive
    - actionfromafar 3y ago
      Are you violently agreeing with me now? :-)
  - saagarjha 3y ago
    It did video encoding because the cable wasn’t capable of streaming full quality video at the required bandwidth. Not because they wanted to make a complicated dongle.
- jampekka 3y ago
  Of course there are a lot of implementation details that need to be done for the bits to move and the pixels to light up. But the scrcpy code is not about this, it's all hidden behind the libraries and in the hardware. What scrcpy has to do is micromanage and hack the APIs on the both sides for this all to happen in the guts of the systema. I don't see a reason why all this couldn't be done with something like /dev/remote/window < /dev/screen Of course there would be a lot of magic happening at the implementation level, but it's not like you type in all the HTTP protocol and negotiate the compression algorithm manually when you go to a website.
refulgentis 3y ago
Quick math on blitting pixels: - lets assume 1920x1080: 2073600 pixels. - 4 bytes a pixel = 2073600 * 4 = 8294400 = 8.2 MB / frame. - 8.2 MB frame * 60 fps = 480 MB/sec = 3.2 gbps = an order of magnitude higher than even your average techie's internet entire pipe. That's for 1080p. The market baseline for resolution, 4K, 4x's that. 240 fps (market baseline for gamers) adds another 4x. ~50 gbps As long as you're willing to accept intellectually that _sometimes_ there will be video-like content played, where more than 10% of the pixels change in a frame, you now _need_ to assume some form of compression algorithm is applied on every frame, i.e. do video encoding.
- nine_k 3y ago
  This is why VNC has been the solution for this during many years, and still remains so. Scrcpy is different because it does not require you to.install anything on the phone. It uses the built-in debugging tools (which you need to just activate in a few clicks).
  - charcircuit 3y ago
    >Scrcpy is different because it does not require you to.install anything on the phone. When you run scrcpy it copies over the server app which runs on the device. The debugging tools are not used for doing the screen capturing.
    - marwis 3y ago
      Do you know why when using WiFi it has to send data over ADB TCP connection rather than self managed UDP?
- saagarjha 3y ago
  Yep, exactly. Sending video over the network with low latency is not a trivial task. There’s a reason we have video and image formats that do heavy compression and inter-frame analysis to reduce bandwidth.
- jampekka 3y ago
charcircuit 3y ago
What part of streaming a screen on Android do you find complex? You pretty much just setup MediaEncoder and let it do its thing. For the input to the encoder you create a VirtualDisplay that mirrors the default Display that is backed by the input Surface of the encoder. For the output of the encoder you can just send the output buffer over network. The PC side isn't complex either. You take those packets from the phone, run it through a decoder that draws the frames to a texture which you can show in a window.
- saagarjha 3y ago
  Bandwidth.
- jampekka 3y ago
  If it's so simple, why does it take 30k or so lines of code for scrcpy to do it?
  - charcircuit 3y ago
    Because there are a lot of features beyond just streaming a screen capture, it supports many versions of Android, and it is a polished utility. The actually screen recording part is simple since you are just using existing Android APIs to have hook up the diplay server to a media encoder and the media encoder to the network. https://github.com/Genymobile/scrcpy/blob/a3cdf1a6b86ea22786e1f7d09b9c202feabc6949/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java https://github.com/Genymobile/scrcpy/blob/a3cdf1a6b86ea22786...
    - jampekka 3y ago
      That is reasonablish as video APIs go. But it's quite a small part of all scrcpy has to do to mirror the screen over the network. Most of that code would be covered with something like "gst-launch pipewiresrc ! encodebin" and the whole mirroring with adding a few elements. Well, in theory at least; gstreamer has the habbit of getting quite fiddly oftentimes. But I don't see any fundamental problems why this too couldn't work with simple composable tools.

jayd16 3y ago

I think you're taking for granted just how performance intensive this is. For example, "Just a blit" is a full screen draw worth of pixel operations on a hardware platform that is known to have pixel draw at a premium.

actionfromafar 3y ago
I don't understand this comment. It's a marvel of course how far we have come, but what today is so intensive? The (old!) Apple iPad cable runs BSD Unix and does real time video encoding.
- refulgentis 3y ago
  Correct, you're violently agreeing, _they embed a whole *NIX computer in the cable to make it happen_. They were replying to someone saying why does everyone make it so complicated. Refer to my comment for the math on why its intensive
  - actionfromafar 3y ago
    Are you violently agreeing with me now? :-)
- saagarjha 3y ago
  It did video encoding because the cable wasn’t capable of streaming full quality video at the required bandwidth. Not because they wanted to make a complicated dongle.
jampekka 3y ago
Of course there are a lot of implementation details that need to be done for the bits to move and the pixels to light up. But the scrcpy code is not about this, it's all hidden behind the libraries and in the hardware. What scrcpy has to do is micromanage and hack the APIs on the both sides for this all to happen in the guts of the systema. I don't see a reason why all this couldn't be done with something like /dev/remote/window < /dev/screen Of course there would be a lot of magic happening at the implementation level, but it's not like you type in all the HTTP protocol and negotiate the compression algorithm manually when you go to a website.

refulgentis 3y ago

Quick math on blitting pixels: - lets assume 1920x1080: 2073600 pixels. - 4 bytes a pixel = 2073600 * 4 = 8294400 = 8.2 MB / frame. - 8.2 MB frame * 60 fps = 480 MB/sec = 3.2 gbps = an order of magnitude higher than even your average techie's internet entire pipe. That's for 1080p. The market baseline for resolution, 4K, 4x's that. 240 fps (market baseline for gamers) adds another 4x. ~50 gbps As long as you're willing to accept intellectually that _sometimes_ there will be video-like content played, where more than 10% of the pixels change in a frame, you now _need_ to assume some form of compression algorithm is applied on every frame, i.e. do video encoding.

nine_k 3y ago
This is why VNC has been the solution for this during many years, and still remains so. Scrcpy is different because it does not require you to.install anything on the phone. It uses the built-in debugging tools (which you need to just activate in a few clicks).
- charcircuit 3y ago
  >Scrcpy is different because it does not require you to.install anything on the phone. When you run scrcpy it copies over the server app which runs on the device. The debugging tools are not used for doing the screen capturing.
  - marwis 3y ago
    Do you know why when using WiFi it has to send data over ADB TCP connection rather than self managed UDP?
saagarjha 3y ago
Yep, exactly. Sending video over the network with low latency is not a trivial task. There’s a reason we have video and image formats that do heavy compression and inter-frame analysis to reduce bandwidth.
jampekka 3y ago

charcircuit 3y ago

What part of streaming a screen on Android do you find complex? You pretty much just setup MediaEncoder and let it do its thing. For the input to the encoder you create a VirtualDisplay that mirrors the default Display that is backed by the input Surface of the encoder. For the output of the encoder you can just send the output buffer over network. The PC side isn't complex either. You take those packets from the phone, run it through a decoder that draws the frames to a texture which you can show in a window.

saagarjha 3y ago
Bandwidth.
jampekka 3y ago
If it's so simple, why does it take 30k or so lines of code for scrcpy to do it?
- charcircuit 3y ago
  Because there are a lot of features beyond just streaming a screen capture, it supports many versions of Android, and it is a polished utility. The actually screen recording part is simple since you are just using existing Android APIs to have hook up the diplay server to a media encoder and the media encoder to the network. https://github.com/Genymobile/scrcpy/blob/a3cdf1a6b86ea22786e1f7d09b9c202feabc6949/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java https://github.com/Genymobile/scrcpy/blob/a3cdf1a6b86ea22786...
  - jampekka 3y ago
    That is reasonablish as video APIs go. But it's quite a small part of all scrcpy has to do to mirror the screen over the network. Most of that code would be covered with something like "gst-launch pipewiresrc ! encodebin" and the whole mirroring with adding a few elements. Well, in theory at least; gstreamer has the habbit of getting quite fiddly oftentimes. But I don't see any fundamental problems why this too couldn't work with simple composable tools.