4 ms·

I almost did a double take when I saw this project. We used this library two years ago in HS for some low-latency VR streaming. And yes, it's a little CPU-inte

I almost did a double take when I saw this project. We used this library two years ago in HS for some low-latency VR streaming. And yes, it's a little CPU-intensive so our smartphones got rather hot. We tried other streaming protocols (such as H.264) but all of them supplied noticeable latency that made our system disorienting. Only JSMPEG was fast enough for our purposes. It's a fantastic library for any low-latency streaming! Highly recommend it. --- Anyways, self-plug for our old project: https://rmj.us/motorized-live-360-video/ https://rmj.us/motorized-live-360-video/. Basically, the smartphone's gryoscope controls a remote video camera that streams a live-feed to the user's headset.

detaro 7y ago
Interesting project! Curious what exactly you tried - H.264 is a codec, and there's a bunch of ways of delivering it to the client (HLS, WebRTC, some have build WebSocket-based streaming, ...), and I'd expect that the main latency is hidden there, not in the decoding?
- ryanmjacobs 7y ago
  Of course. FFMpeg was used consistently on our backend; we cycled through various codecs to find the best one for our purpose and changed the frontend library accordingly. And I think you're right on the encoding latency. I believe that H.264 buffers a little bit before it makes a decision on how to compress the frames, where as MPEG1 doesn't? I could be completely wrong but my gut is telling me that MPEG1 is basically independent, slightly-modified JPEG frames.
  - josteink 7y ago
    > I could be completely wrong but my gut is telling me that MPEG1 is basically independent, slightly-modified JPEG frames. Not that simple, but not that far off. Basically MPEG1 has key-frames (“JPEG” frames) and (forward) delta-frames (diffs). From my understanding H264 has many improvements, including reverse delta-frames. My guess is that disabling those in the encoder will improve real-time capabilities.
    - Orphis 7y ago
      Bingo, that's one of the tuning you can use for real-time communication. There's a lot of ways to tune an encoder, and getting optimal setting for each use case is always tricky.
    - ryanmjacobs 7y ago
      Ah gotcha, I think I was mixing up MPEG1 and MJPEG in my head. I didn't realize that you could disable those features in the encoder. I'll have to look into x264 tunables that lower latency. Might be interesting.
  - wolf550e 7y ago
    libx264 (which is the H.264 software encoder everyone uses) has a mode optimized for zero latency (--tune zerolatency).
maaaats 7y ago
Cool project, similar in spirit as to something I did a few years back as well ( https://github.com/Matsemann/oculus-fpv https://github.com/Matsemann/oculus-fpv ). It's pretty trippy watching real stuff live, but your "head" is somewhere else. Cool that it's even possible to experiment with in the browser now.
- ryanmjacobs 7y ago
  Oh shoot wow that's really cool! Honestly, it's such a weird feeling viewing through that secondary perspective. I'm glad that someone else also came up with the idea. We had a lot of fun by having someone hold the device mounted on a pole, looking down on the viewer, and following them around the room. This provided a weird, quasi-video-gamey third-person perspective, like in GTA or some other.
  - netsharc 7y ago
    A friend once walked towards my office while we were video chatting (he was on his phone). Seeing the video of our office hallway on my screen, and realizing he was walking to my office disorientated me a little. And these guys did that with a car: https://www.youtube.com/watch?v=nIRUavithF8 https://www.youtube.com/watch?v=nIRUavithF8
- otakucode 7y ago
  That experience sounds like 'virtual embodiment'. Researchers are using it for a bunch of different things as it can be quite powerful. So powerful, in fact, that some of the researchers have been advocating for limiting access to the technology and establishing a 'code of ethics' around its use. Basically, how you see the world is tied to an internal mental model of your own body that establishes where each limb is, how big you are in relation to your environment, etc... and virtual embodiment seems able to alter that very easily. With lasting effects.
  - netsharc 7y ago
    One man ported Grand Theft Auto V to VR, and he got really freaked out when he shot a man in-game...
Tade0 7y ago
We experimented with it five years ago trying to reduce the loading time of animations in a mobile game, but with the phones of the time we only managed to exchange waiting for the data with waiting for the devices to process the clips. I guess nowadays this strategy would work much better, even considering that 4G is currently the norm.

detaro 7y ago

Interesting project! Curious what exactly you tried - H.264 is a codec, and there's a bunch of ways of delivering it to the client (HLS, WebRTC, some have build WebSocket-based streaming, ...), and I'd expect that the main latency is hidden there, not in the decoding?

ryanmjacobs 7y ago
Of course. FFMpeg was used consistently on our backend; we cycled through various codecs to find the best one for our purpose and changed the frontend library accordingly. And I think you're right on the encoding latency. I believe that H.264 buffers a little bit before it makes a decision on how to compress the frames, where as MPEG1 doesn't? I could be completely wrong but my gut is telling me that MPEG1 is basically independent, slightly-modified JPEG frames.
- josteink 7y ago
  > I could be completely wrong but my gut is telling me that MPEG1 is basically independent, slightly-modified JPEG frames. Not that simple, but not that far off. Basically MPEG1 has key-frames (“JPEG” frames) and (forward) delta-frames (diffs). From my understanding H264 has many improvements, including reverse delta-frames. My guess is that disabling those in the encoder will improve real-time capabilities.
  - Orphis 7y ago
    Bingo, that's one of the tuning you can use for real-time communication. There's a lot of ways to tune an encoder, and getting optimal setting for each use case is always tricky.
  - ryanmjacobs 7y ago
    Ah gotcha, I think I was mixing up MPEG1 and MJPEG in my head. I didn't realize that you could disable those features in the encoder. I'll have to look into x264 tunables that lower latency. Might be interesting.
- wolf550e 7y ago
  libx264 (which is the H.264 software encoder everyone uses) has a mode optimized for zero latency (--tune zerolatency).

maaaats 7y ago

Cool project, similar in spirit as to something I did a few years back as well ( https://github.com/Matsemann/oculus-fpv https://github.com/Matsemann/oculus-fpv ). It's pretty trippy watching real stuff live, but your "head" is somewhere else. Cool that it's even possible to experiment with in the browser now.

ryanmjacobs 7y ago
Oh shoot wow that's really cool! Honestly, it's such a weird feeling viewing through that secondary perspective. I'm glad that someone else also came up with the idea. We had a lot of fun by having someone hold the device mounted on a pole, looking down on the viewer, and following them around the room. This provided a weird, quasi-video-gamey third-person perspective, like in GTA or some other.
- netsharc 7y ago
  A friend once walked towards my office while we were video chatting (he was on his phone). Seeing the video of our office hallway on my screen, and realizing he was walking to my office disorientated me a little. And these guys did that with a car: https://www.youtube.com/watch?v=nIRUavithF8 https://www.youtube.com/watch?v=nIRUavithF8
otakucode 7y ago
That experience sounds like 'virtual embodiment'. Researchers are using it for a bunch of different things as it can be quite powerful. So powerful, in fact, that some of the researchers have been advocating for limiting access to the technology and establishing a 'code of ethics' around its use. Basically, how you see the world is tied to an internal mental model of your own body that establishes where each limb is, how big you are in relation to your environment, etc... and virtual embodiment seems able to alter that very easily. With lasting effects.
- netsharc 7y ago
  One man ported Grand Theft Auto V to VR, and he got really freaked out when he shot a man in-game...

Tade0 7y ago

We experimented with it five years ago trying to reduce the loading time of animations in a mobile game, but with the phones of the time we only managed to exchange waiting for the data with waiting for the devices to process the clips. I guess nowadays this strategy would work much better, even considering that 4G is currently the norm.