5 ms·

I think technically he’s correct (I haven’t worked on media decoding code, but I understand how common video encoding formats work). If you have a long video wi

I think technically he’s correct (I haven’t worked on media decoding code, but I understand how common video encoding formats work). If you have a long video with only a single key frame at the beginning then to step back you would need to, starting from the beginning of the video, decode every frame up to the previous frame you wanted to jump to in order to apply frame deltas, also assuming you have some sort of frame counter to determine when you’ve reached the target frame. In the worst case this does require a lot of compute, but this is an edge case if you primarily care about common video formats with normal encoding settings. I assume seeking backwards is also painfully slow on videos encoded in this manner, so why stepping back 1 frame is out of the question when compared to seeking backwards, I don’t fully understand, it must have something to do with precise frame counts being unavailable on some hardware decoders for some formats (and there being no good workaround) so you _may_ not actually go back 1 frame. I don’t see any reason it couldn’t be supported for a set of formats with reasonable encoding/decoding settings, and provide some error message for other formats if a user attempts to step back, e.g. reverse frame stepping unavailable for current video due to format/encoding/decoding settings.

sergiotapia 2y ago
that's still a whole lot of yapping that as an end user I don't care about. i can frame scrub forwards and backwards in multiple other apps. right? very weird response from the vlc team in that original thread.
- makeitdouble 2y ago
  The back and forth in itself feels so weird to me, with so many hurt feelings: - the devs expressed in no uncertain terms that they don't want to do it (the first answer is just perfect) - every third comment is about "we know you don't want to do it, but as users why should we care ?" Well, if you don't care about the devs, on what base are you asking them to care about your specific problem ?
  - justin66 2y ago
    > Well, if you don't care about the devs, on what base are you asking them to care about your specific problem ? Caring about the user's requirements is part of the dev's job description. Caring about the dev's... anything is not in the user's job description. (one advantage commercial software has: it really does help when there's an interface between the dev and the user in the form of customer support. or a commercial incentive to actually work on what the user wants.)
    - skydhash 2y ago
      > Caring about the user's requirements is part of the dev's job description. For OSS project, it's better to assume that the user persona for the software is the devs or the maintainers. The dev-user relationship you expect is actually the vendor-client in commercial software.
    - makeitdouble 2y ago
      > job description Money getting involved would indeed simplify the question. Here no money is changing hand, so coming up with an angle that's motivating enough for the devs is IMHO the only option. Either bring up an aspect they're not considering that changes the equation for them, or come up with a solution that isn't plaggued by the issues they are afraid to deal with. That's where I see listening to the devs and caring about their issues to be the only path forward, short of contributing as a dev oneself..
NavinF 2y ago
> video with only a single key frame at the beginning I've literally never seen a video like that in my life, but I'd still expect it to work. Just decode everything starting from frame 1. My desktop can decode H265 at 1,666 fps. I can wait. https://docs.nvidia.com/video-technologies/video-codec-sdk/12.1/nvdec-application-note/index.html https://docs.nvidia.com/video-technologies/video-codec-sdk/1...
- Panzer04 2y ago
  That blows up pretty quickly though? Even a 10 minute video will take in the worst case ~20 seconds to decode at that rate. Not really an excuse not to have it (since most video wont be encoded in such an insane way), but the developers owe no obligation to users to implement it.
  - Hendrikto 2y ago
    Theoretically. Practically, 10+ minute videos with just a single i frame at the beginning do not exist.
  - deleted 2y ago
    [deleted]
hsbauauvhabzb 2y ago
Wouldn’t creating in-memory key frames for every nth frame resolve substantial computations on a frame-by-frame basis?
- infofarmer 2y ago
  you'd have to "rebase" all the other frames to be derived from those
  - coryrc 2y ago
    You wouldn't have to any more than you need 0 through N frames in memory to calculate frame N+1. Whatever your decoding state completes at frame N can be considered a key frame.
    - infofarmer 2y ago
      decoding state can be bulkier than a key frame and opaque to the CPU if hardware acceleration is used I wonder if caching semi-compressed frames would be more efficient in either case (CPU or GPU)
      - hsbauauvhabzb 2y ago
        It doesn’t have to be every frame though. Pre calculate to 25%/50%/75%, then as I’m playing, save key frames for more incremental points, and if I start scrubbing, calculate more around that region. Edit: this doesn’t have to happen synchronously either, it can occur in a background thread or passively.
- DonHopkins 2y ago
  There is no reason to start caching previous frames until AFTER the user has paused and pressed the "back frame" key. Only THEN does it need to rewind to the previous i-frame and re-render and cache frames. And there is no measurable cost to remembering the timestamp of the last i-frame, so you know where to rewind to.
scottlamb 2y ago
> If you have a long video with only a single key frame at the beginning then ...you can't support the scrub bar efficiently either, so no one encodes video that way. Typically to go to a frame you find the last IDR frame before it (and in reasonable encodings those are frequent enough) and decode forward until you get to the frame of interest. Doing that every time the user presses the single frame back button really doesn't seem that bad, and neither does holding onto some extra reference images for at least like 1080p frames. (8k video and such starts getting more expensive but maybe even then start doing all some references after the first press of the frame back button in this GOP or some such.) It's of course work to do, and I'm not super motivated to send them that patch, and there's the question of it it would be merged and maintained indefinitely, but what folks are asking for is technically possible. > I don’t see any reason it couldn’t be supported for a set of formats with reasonable encoding/decoding settings, and provide some error message for other formats if a user attempts to step back, e.g. reverse frame stepping unavailable for current video due to format/encoding/decoding settings. Yeah, this. That's likely more or less what they already do with the scrub bar.
- mtrower 2y ago
  > It's of course work to do, and I'm not super motivated to send them that patch, and there's the question of it it would be merged That's my issue; he calls for people to send patches, but anyone capable of writing such a patch is also probably going to see that he's not positive on the matter, and that his "patches welcome" is really pretty passive aggressive in this instance. At least, that's how it comes off to me. I would expect that, should I submit such a patch, it would simply be rejected on the basis that "it is not a general solution".
  - mdf 2y ago
    There's also a middle ground: Painstakingly describe the solution first, along with its downside of not being general in the same way as some of the existing features (I guess for example seeking back 10 seconds) are not, and ask whether a patch implementing this solution would be welcome before implementing it.
fragmede 2y ago
all that's correct, but it's besides the point since other players are able to do this.
- FabHK 2y ago
  As outlined in some other thread, mpv is not able to stream eg to ChromeCast, unlike VLC. Maybe VLC supports certain things that make the previous-frame thing harder. I suspect it is so, but I don't have insight into the detailed architecture of VLC, unlike I assume the VLC developers. Do you?
  - fragmede 2y ago
    Why is their architecture making it harder on them relevant to the question if it's possible? Because the root question is if it's possible, and there's multiple existence proofs that it's possible. Maybe the VLC developers are just tired. I don't blame them. They had to do a whole refactor to get Chromecast support working, and they got no thanks for that. Or maybe it just wasn't enough thanks and they don't feel like doing another refactor. Chromecast support is quite tricky, I've dug into the protocol. Anyway, I'm not in control of their development, I'm just pointing out that seeking backwards is possible.
    - FabHK 2y ago
      [flagged]
oefrha 2y ago
> I think technically he’s correct (I haven’t worked on media decoding code, but I understand how common video encoding formats work). He’s technically simply wrong (I have worked on media decoding code, hell I’m working on a related project today). His player supports seeking back by 10 seconds or whatnot, but he insists that somehow to implement precise seeking to the previous frame, you need to seek from the very beginning of the video, no two ways about it: > There is not a slight technical difficulty. On a logical level, this feature is algorithmically impossible, except for the extreme: You can decode all frames up to the previous one. But this would be far too slow for anything except really short videos. It’s obvious bullshit, if you encounter one of those pathological videos (that don’t really exist except in his mind or in some test suites) you just give up after a reasonable amount of time, same as how you give up if you can’t seek back ~10s in a reasonable amount of time. And players do give up seeking all the time already, not just on these hypothetical one-I-frame-per-hour videos, but real world videos with messed up pts/dts with no reasonable way to go back a short interval.
eviks 2y ago
Or in the less-than-worst case you could cache the created full frame if no native full frame is encountered for X seconds and then in the worst case you don't have to go to the beginning of the video, but to that cached intermediate full frame?

sergiotapia 2y ago

that's still a whole lot of yapping that as an end user I don't care about. i can frame scrub forwards and backwards in multiple other apps. right? very weird response from the vlc team in that original thread.

makeitdouble 2y ago
The back and forth in itself feels so weird to me, with so many hurt feelings: - the devs expressed in no uncertain terms that they don't want to do it (the first answer is just perfect) - every third comment is about "we know you don't want to do it, but as users why should we care ?" Well, if you don't care about the devs, on what base are you asking them to care about your specific problem ?
- justin66 2y ago
  > Well, if you don't care about the devs, on what base are you asking them to care about your specific problem ? Caring about the user's requirements is part of the dev's job description. Caring about the dev's... anything is not in the user's job description. (one advantage commercial software has: it really does help when there's an interface between the dev and the user in the form of customer support. or a commercial incentive to actually work on what the user wants.)
  - skydhash 2y ago
    > Caring about the user's requirements is part of the dev's job description. For OSS project, it's better to assume that the user persona for the software is the devs or the maintainers. The dev-user relationship you expect is actually the vendor-client in commercial software.
  - makeitdouble 2y ago
    > job description Money getting involved would indeed simplify the question. Here no money is changing hand, so coming up with an angle that's motivating enough for the devs is IMHO the only option. Either bring up an aspect they're not considering that changes the equation for them, or come up with a solution that isn't plaggued by the issues they are afraid to deal with. That's where I see listening to the devs and caring about their issues to be the only path forward, short of contributing as a dev oneself..

NavinF 2y ago

> video with only a single key frame at the beginning I've literally never seen a video like that in my life, but I'd still expect it to work. Just decode everything starting from frame 1. My desktop can decode H265 at 1,666 fps. I can wait. https://docs.nvidia.com/video-technologies/video-codec-sdk/12.1/nvdec-application-note/index.html https://docs.nvidia.com/video-technologies/video-codec-sdk/1...

Panzer04 2y ago
That blows up pretty quickly though? Even a 10 minute video will take in the worst case ~20 seconds to decode at that rate. Not really an excuse not to have it (since most video wont be encoded in such an insane way), but the developers owe no obligation to users to implement it.
- Hendrikto 2y ago
  Theoretically. Practically, 10+ minute videos with just a single i frame at the beginning do not exist.
- deleted 2y ago
  [deleted]

hsbauauvhabzb 2y ago

Wouldn’t creating in-memory key frames for every nth frame resolve substantial computations on a frame-by-frame basis?

infofarmer 2y ago
you'd have to "rebase" all the other frames to be derived from those
- coryrc 2y ago
  You wouldn't have to any more than you need 0 through N frames in memory to calculate frame N+1. Whatever your decoding state completes at frame N can be considered a key frame.
  - infofarmer 2y ago
    decoding state can be bulkier than a key frame and opaque to the CPU if hardware acceleration is used I wonder if caching semi-compressed frames would be more efficient in either case (CPU or GPU)
    - hsbauauvhabzb 2y ago
      It doesn’t have to be every frame though. Pre calculate to 25%/50%/75%, then as I’m playing, save key frames for more incremental points, and if I start scrubbing, calculate more around that region. Edit: this doesn’t have to happen synchronously either, it can occur in a background thread or passively.
DonHopkins 2y ago
There is no reason to start caching previous frames until AFTER the user has paused and pressed the "back frame" key. Only THEN does it need to rewind to the previous i-frame and re-render and cache frames. And there is no measurable cost to remembering the timestamp of the last i-frame, so you know where to rewind to.

scottlamb 2y ago

> If you have a long video with only a single key frame at the beginning then ...you can't support the scrub bar efficiently either, so no one encodes video that way. Typically to go to a frame you find the last IDR frame before it (and in reasonable encodings those are frequent enough) and decode forward until you get to the frame of interest. Doing that every time the user presses the single frame back button really doesn't seem that bad, and neither does holding onto some extra reference images for at least like 1080p frames. (8k video and such starts getting more expensive but maybe even then start doing all some references after the first press of the frame back button in this GOP or some such.) It's of course work to do, and I'm not super motivated to send them that patch, and there's the question of it it would be merged and maintained indefinitely, but what folks are asking for is technically possible. > I don’t see any reason it couldn’t be supported for a set of formats with reasonable encoding/decoding settings, and provide some error message for other formats if a user attempts to step back, e.g. reverse frame stepping unavailable for current video due to format/encoding/decoding settings. Yeah, this. That's likely more or less what they already do with the scrub bar.

mtrower 2y ago
> It's of course work to do, and I'm not super motivated to send them that patch, and there's the question of it it would be merged That's my issue; he calls for people to send patches, but anyone capable of writing such a patch is also probably going to see that he's not positive on the matter, and that his "patches welcome" is really pretty passive aggressive in this instance. At least, that's how it comes off to me. I would expect that, should I submit such a patch, it would simply be rejected on the basis that "it is not a general solution".
- mdf 2y ago
  There's also a middle ground: Painstakingly describe the solution first, along with its downside of not being general in the same way as some of the existing features (I guess for example seeking back 10 seconds) are not, and ask whether a patch implementing this solution would be welcome before implementing it.

fragmede 2y ago

all that's correct, but it's besides the point since other players are able to do this.

FabHK 2y ago
As outlined in some other thread, mpv is not able to stream eg to ChromeCast, unlike VLC. Maybe VLC supports certain things that make the previous-frame thing harder. I suspect it is so, but I don't have insight into the detailed architecture of VLC, unlike I assume the VLC developers. Do you?
- fragmede 2y ago
  Why is their architecture making it harder on them relevant to the question if it's possible? Because the root question is if it's possible, and there's multiple existence proofs that it's possible. Maybe the VLC developers are just tired. I don't blame them. They had to do a whole refactor to get Chromecast support working, and they got no thanks for that. Or maybe it just wasn't enough thanks and they don't feel like doing another refactor. Chromecast support is quite tricky, I've dug into the protocol. Anyway, I'm not in control of their development, I'm just pointing out that seeking backwards is possible.
  - FabHK 2y ago
    [flagged]

oefrha 2y ago

> I think technically he’s correct (I haven’t worked on media decoding code, but I understand how common video encoding formats work). He’s technically simply wrong (I have worked on media decoding code, hell I’m working on a related project today). His player supports seeking back by 10 seconds or whatnot, but he insists that somehow to implement precise seeking to the previous frame, you need to seek from the very beginning of the video, no two ways about it: > There is not a slight technical difficulty. On a logical level, this feature is algorithmically impossible, except for the extreme: You can decode all frames up to the previous one. But this would be far too slow for anything except really short videos. It’s obvious bullshit, if you encounter one of those pathological videos (that don’t really exist except in his mind or in some test suites) you just give up after a reasonable amount of time, same as how you give up if you can’t seek back ~10s in a reasonable amount of time. And players do give up seeking all the time already, not just on these hypothetical one-I-frame-per-hour videos, but real world videos with messed up pts/dts with no reasonable way to go back a short interval.

eviks 2y ago

Or in the less-than-worst case you could cache the created full frame if no native full frame is encountered for X seconds and then in the worst case you don't have to go to the beginning of the video, but to that cached intermediate full frame?