5 ms·

This is really cool! The "video" quality is pretty bad though. From what I can see, the app streams base64 encoded JPEGs via WebSockets and draws them in a Can

This is really cool! The "video" quality is pretty bad though. From what I can see, the app streams base64 encoded JPEGs via WebSockets and draws them in a Canvas. The overhead of sending complete images costs a lot of bandwidth. You may want to look into jsmpeg for live streaming: https://github.com/phoboslab/jsmpeg https://github.com/phoboslab/jsmpeg

weiyin 12y ago
Thanks for the suggestion, that looks like a cool library. A future goal is to be able to provide higher resolution images to fully show off the high DPI displays.
kolencherry 12y ago
Why not use WebRTC for streaming the video?
- TD-Linux 12y ago
  You don't even need to go that complicated. A WebM stream over HTTP is pretty easy and would work well for this.
  - weiyin 12y ago
    we did some testing using compressed video, but ended up finding single frames to be more performant. also, the setup can handle much higher framerate and quality but we've kept things fairly lean for the demo :) can easily handle 15fps w/ high quality images. thanks again for the questions and comments.
  - cbhl 12y ago
    It's worth noting that Chrome Remote Desktop uses VP8 (which is the video codec used in WebM): http://googleblog.blogspot.com/2012/05/next-step-in-chrome-os-journey.html http://googleblog.blogspot.com/2012/05/next-step-in-chrome-o... If you're having performance issues with it, you just have the video codec tuned wrong.
    - weiyin 12y ago
      definitely a possibility :) we'll check that out, thanks for the heads up.
TheMakeA 12y ago
We've actually experimented with jsmpeg for this type of thing. The problem we ran into was that jsmpeg seemed to require every frame from ffmpeg. If the browser or network slowed, the feed would fall behind the simulation, leading to several seconds of latency when interacting. We'd love to use it and appreciate your hard work though. Do you have any ideas on how that problem can be solved?
- phoboslab 12y ago
  There are two sides to this problem. For jsmpeg (or even in the server), it would be possible to simply wait for the next intra frame (a frame that doesn't require previous frames to decode) on certain conditions and throw away all other frames in between. This will give the decoder and network time to catch up. By default ffmpeg emits one intra frame per second, iirc. The more difficult part is probably detecting this lag. I'm not sure how this could be done. Maybe the node websocket server (ws) can detect congestion somehow.
  - TheMakeA 12y ago
    Thanks. Maybe our ffmpeg settings weren't creating intra-frames somehow (or we happened to drop every single one) because the stream would never recover. I'll probably try it again at some point soon. We really want jsmpeg to work.
AndroidWebEmu 12y ago
There's a javascript VNC client that was created during a LinkedIn Intern Hackday that could solve the problem if a different approach is used on the backend. Each instance would need a virtual framebuffer. The js vnc client is a little buggy, but with some cleanup it worked quite well for a similar project that involved making the Android emulator available in a browser.
userbinator 12y ago
base64 encoded JPEGs MJPEG is supported in all browsers I know of, and would cut down on bandwidth by 33% over base64. (If you really want to use WebSockets, a more dense encoding than base64 could also help - look into base85, for example.)
- niutech 12y ago
  MJPEG is good, but keep in mind that Chrome doesn't support playback of MJPEG streams as a main resource, it has to be in an <img> tag.
  - userbinator 12y ago
    Looks like that functionality used to be there, was broken, fixed, and then broken again sometime later. The latest bug for it appears to be https://code.google.com/p/chromium/issues/detail?id=308999 https://code.google.com/p/chromium/issues/detail?id=308999 For this application you'd need it to be in an <img> tag anyway since there's the other stuff on the page too, so that's a moot point.
    - niutech 12y ago
      Not only <img>. The MJPEG stream could theoretically be embedded in an <iframe>, thus being affected by this bug.

weiyin 12y ago

Thanks for the suggestion, that looks like a cool library. A future goal is to be able to provide higher resolution images to fully show off the high DPI displays.

kolencherry 12y ago

Why not use WebRTC for streaming the video?

TD-Linux 12y ago
You don't even need to go that complicated. A WebM stream over HTTP is pretty easy and would work well for this.
- weiyin 12y ago
  we did some testing using compressed video, but ended up finding single frames to be more performant. also, the setup can handle much higher framerate and quality but we've kept things fairly lean for the demo :) can easily handle 15fps w/ high quality images. thanks again for the questions and comments.
- cbhl 12y ago
  It's worth noting that Chrome Remote Desktop uses VP8 (which is the video codec used in WebM): http://googleblog.blogspot.com/2012/05/next-step-in-chrome-os-journey.html http://googleblog.blogspot.com/2012/05/next-step-in-chrome-o... If you're having performance issues with it, you just have the video codec tuned wrong.
  - weiyin 12y ago
    definitely a possibility :) we'll check that out, thanks for the heads up.

TheMakeA 12y ago

We've actually experimented with jsmpeg for this type of thing. The problem we ran into was that jsmpeg seemed to require every frame from ffmpeg. If the browser or network slowed, the feed would fall behind the simulation, leading to several seconds of latency when interacting. We'd love to use it and appreciate your hard work though. Do you have any ideas on how that problem can be solved?

phoboslab 12y ago
There are two sides to this problem. For jsmpeg (or even in the server), it would be possible to simply wait for the next intra frame (a frame that doesn't require previous frames to decode) on certain conditions and throw away all other frames in between. This will give the decoder and network time to catch up. By default ffmpeg emits one intra frame per second, iirc. The more difficult part is probably detecting this lag. I'm not sure how this could be done. Maybe the node websocket server (ws) can detect congestion somehow.
- TheMakeA 12y ago
  Thanks. Maybe our ffmpeg settings weren't creating intra-frames somehow (or we happened to drop every single one) because the stream would never recover. I'll probably try it again at some point soon. We really want jsmpeg to work.

AndroidWebEmu 12y ago

There's a javascript VNC client that was created during a LinkedIn Intern Hackday that could solve the problem if a different approach is used on the backend. Each instance would need a virtual framebuffer. The js vnc client is a little buggy, but with some cleanup it worked quite well for a similar project that involved making the Android emulator available in a browser.

userbinator 12y ago

base64 encoded JPEGs MJPEG is supported in all browsers I know of, and would cut down on bandwidth by 33% over base64. (If you really want to use WebSockets, a more dense encoding than base64 could also help - look into base85, for example.)

niutech 12y ago
MJPEG is good, but keep in mind that Chrome doesn't support playback of MJPEG streams as a main resource, it has to be in an <img> tag.
- userbinator 12y ago
  Looks like that functionality used to be there, was broken, fixed, and then broken again sometime later. The latest bug for it appears to be https://code.google.com/p/chromium/issues/detail?id=308999 https://code.google.com/p/chromium/issues/detail?id=308999 For this application you'd need it to be in an <img> tag anyway since there's the other stuff on the page too, so that's a moot point.
  - niutech 12y ago
    Not only <img>. The MJPEG stream could theoretically be embedded in an <iframe>, thus being affected by this bug.