4 ms·

I've done A LOT of research and experimentation in this space over the course of the last 5-ish years, mainly out of sheer curiosity. This sounds very, very sim

I've done A LOT of research and experimentation in this space over the course of the last 5-ish years, mainly out of sheer curiosity. This sounds very, very similar to something I prototyped a while back using x264 + Broadway.js (HTML5 video streaming with low latency is a non-starter). I'm extremely curious, has Paperspace actually developed an entirely new video codec, encoder, and efficient JS decoder for this? I'm a little confused by the statements so far (and the comments are flowing in, so apologies if this was addressed while I was typing): "we are using a video stream", "using a JS renderer", "building a streaming protocol", "using GPU tech originally developed for video game streaming", "a remote desktop protocol that could stream HD video" So, is just the streaming protocol you've developed and not the codec? In my expiriments, simply pushing out h264 NAL units over websockets and passing them to the decoder was a pretty solid start. Add a tiny layer of buffering over that and I imagine it'd be fairly stable. Ultimately, I backed away from h264 for licensing and performance issues. Also, what's the transport into the browser? Websockets? WebRTC data channels? Have you encountered performance issues with Firefox not being able to handle websockets or data channels in web workers? (Which is seemingly coming in FF37.)

DTE 12y ago
awesome questions! There will be tech blog deep dive that goes into more detail, but just to clarify things a bit we are using h264 as the underlying codec but the protocol is a combination things -- h264 packets over websockets is where we started too! Usually you want a "reliable" transport (tcp) for handshake, some transfer, etc and then you can get away with a more fire-and-forget (udp) stream for everything else. We broker different connections for different scenarios. The web version is naturally limited to either webrtc or websockets but for the paperweight it didn't make sense to force a web paradigm (or full webbrowser just to access limited socket types) so we interface directly with the hardware. Haven't encountered big problems with firefox, but browser inconsistencies are definitely something we spend a lot of time working through
- billconan 12y ago
  how do you interface directly with the hardware?
  - rcconf 12y ago
    Because Paperweight is the black circle hardware device they made to stream from Paperspace. There's no point to using a browser on it, just stream directly through the wifi/ethernet port using TCP/IP.
Icer5k 12y ago
I'm curious what performance issues you were seeing in Firefox. I'm doing quite a bit of work with the same stack, and have been able to get sub-10ms decode times with Broadway.JS in all the major browsers. As long as you use transferable objects to pass data around, I haven't had any problems using web workers as decode threads either.
- EvanDotPro 12y ago
  I'm sorry, I actually mixed up two different experiments regarding the performance issues (I've been playing with this stuff for a looongg time and have many different prototypes). Allow my to clarify. :) The performance issues with using Broadway.js were more "general", in that the experience varied wildly, from amazing to unusable depending on the browser/device/etc)... Not necessarily related to Websockets + Webworkers. The Websocket issue on the other hand, was for a more recent experiment where I was playing with the idea of bridging NoVNC over WebRTC data channels. It's not so much a performance "issue" as much as a "possible area for optimization"... Though, every time I'm playing with noVNC or Broadway.js in a FF tab on my i7 laptop, it pretty much renders FF pretty laggy in all other tabs. I imagine offloading as much of the processing to workers as possible would be the best approach to lessen the effect — though, I'm not much of a frontend / JS dev. Here's the noVNC issue: https://github.com/kanaka/noVNC/issues/114 https://github.com/kanaka/noVNC/issues/114 And the bugzilla for FF: https://bugzilla.mozilla.org/show_bug.cgi?id=504553 https://bugzilla.mozilla.org/show_bug.cgi?id=504553
phoboslab 12y ago
Shameless plug: check out jsmpeg[1] - it's an MPEG1 decoder written in JavaScript that's capable of low latency streaming via WebSockets, weighs only ~25kb gzipped, offloads part of the decoding to the GPU via WebGL and runs smoothly on an iPhone4. Of course it has some drawbacks, as MPEG1 is a pretty old codec. The data rate is fairly high and it struggles with high resolution streams. 720p still decodes nicely in realtime on most mobile devices, though. [1] https://github.com/phoboslab/jsmpeg https://github.com/phoboslab/jsmpeg
- DTE 12y ago
  Thanks for sharing! I've definitely bumped into jsmpeg a number of times when researching JS decoders. Did you ever pursue an emscripten/ asm.js version?

DTE 12y ago

awesome questions! There will be tech blog deep dive that goes into more detail, but just to clarify things a bit we are using h264 as the underlying codec but the protocol is a combination things -- h264 packets over websockets is where we started too! Usually you want a "reliable" transport (tcp) for handshake, some transfer, etc and then you can get away with a more fire-and-forget (udp) stream for everything else. We broker different connections for different scenarios. The web version is naturally limited to either webrtc or websockets but for the paperweight it didn't make sense to force a web paradigm (or full webbrowser just to access limited socket types) so we interface directly with the hardware. Haven't encountered big problems with firefox, but browser inconsistencies are definitely something we spend a lot of time working through

billconan 12y ago
how do you interface directly with the hardware?
- rcconf 12y ago
  Because Paperweight is the black circle hardware device they made to stream from Paperspace. There's no point to using a browser on it, just stream directly through the wifi/ethernet port using TCP/IP.

Icer5k 12y ago

I'm curious what performance issues you were seeing in Firefox. I'm doing quite a bit of work with the same stack, and have been able to get sub-10ms decode times with Broadway.JS in all the major browsers. As long as you use transferable objects to pass data around, I haven't had any problems using web workers as decode threads either.

EvanDotPro 12y ago
I'm sorry, I actually mixed up two different experiments regarding the performance issues (I've been playing with this stuff for a looongg time and have many different prototypes). Allow my to clarify. :) The performance issues with using Broadway.js were more "general", in that the experience varied wildly, from amazing to unusable depending on the browser/device/etc)... Not necessarily related to Websockets + Webworkers. The Websocket issue on the other hand, was for a more recent experiment where I was playing with the idea of bridging NoVNC over WebRTC data channels. It's not so much a performance "issue" as much as a "possible area for optimization"... Though, every time I'm playing with noVNC or Broadway.js in a FF tab on my i7 laptop, it pretty much renders FF pretty laggy in all other tabs. I imagine offloading as much of the processing to workers as possible would be the best approach to lessen the effect — though, I'm not much of a frontend / JS dev. Here's the noVNC issue: https://github.com/kanaka/noVNC/issues/114 https://github.com/kanaka/noVNC/issues/114 And the bugzilla for FF: https://bugzilla.mozilla.org/show_bug.cgi?id=504553 https://bugzilla.mozilla.org/show_bug.cgi?id=504553

phoboslab 12y ago

Shameless plug: check out jsmpeg[1] - it's an MPEG1 decoder written in JavaScript that's capable of low latency streaming via WebSockets, weighs only ~25kb gzipped, offloads part of the decoding to the GPU via WebGL and runs smoothly on an iPhone4. Of course it has some drawbacks, as MPEG1 is a pretty old codec. The data rate is fairly high and it struggles with high resolution streams. 720p still decodes nicely in realtime on most mobile devices, though. [1] https://github.com/phoboslab/jsmpeg https://github.com/phoboslab/jsmpeg

DTE 12y ago
Thanks for sharing! I've definitely bumped into jsmpeg a number of times when researching JS decoders. Did you ever pursue an emscripten/ asm.js version?