6 ms·

>In summary this is the very basic idea behind a video: a series of pictures / frames running at a given rate. >Therefore we need to introduce some logic to pl

>In summary this is the very basic idea behind a video: a series of pictures / frames running at a given rate. >Therefore we need to introduce some logic to play each frame smoothly. For that matter, each frame has a presentation timestamp (PTS) which is an increasing number factored in a timebase that is a rational number (where the denominator is know as timescale) divisible by the frame rate (fps). Constant framerate is a very dangerous assumption to make. It's true for video recorded by professional cameras, but if you check video recorded by most mobile phones you notice that it's wrong.

gowan 9y ago
to be fair it is possible to do this from the command line with ffmpeg when using concat[1][2]. i did this when creating real time video from screencast in the chrome remote debugger. [1] https://ffmpeg.org/ffmpeg-formats.html#concat https://ffmpeg.org/ffmpeg-formats.html#concat [2] https://trac.ffmpeg.org/wiki/Slideshow https://trac.ffmpeg.org/wiki/Slideshow
jiggunjer 9y ago
In my experience even cameras with constant framerates aren't really constant, if you look at the timestamps the delay between frames typically varies a few (dozen) ms. So on average you might have 30fps, but if you sample a random second you might end up with 28-32 frames.
- colund 9y ago
  29.97?
  - djsumdog 9y ago
    That comes from the old NTSC standard and the history of adding colour in a backwards compatible way: https://www.youtube.com/watch?v=3GJUM6pCpew https://www.youtube.com/watch?v=3GJUM6pCpew In the case that the above posters are mentioning, this probably has less to do with NTSC compatibility and more to do with bad timing hardware or cameras that attempt to do too much in PIO mode instead of using real encoding hardware.
    - mikepurvis 9y ago
      Glad to see that video linked; it's a terrific explanation.
    - jiggunjer 9y ago
      I wouldn't say bad timing means bad hardware though, even expensive scientific cameras suffer from this. You need rather complex electronics to capture at a high and consistent rate. I think normally the encoding is done by software.
lovelearning 9y ago
[1] is a useful list of such assumptions. Found it very handy when programming AV handling. [1]: https://haasn.xyz/posts/2016-12-25-falsehoods-programmers-believe-about-%5Bvideo-stuff%5D.html https://haasn.xyz/posts/2016-12-25-falsehoods-programmers-be...
- jsheard 9y ago
  See also: this talk from a Vimeo engineer about the video encoding horrors they've encountered in the wild. https://www.youtube.com/watch?v=cRSO3RtUOOk https://www.youtube.com/watch?v=cRSO3RtUOOk / https://speakerdeck.com/demuxed/things-developers-believe-about-video-files-proven-wrong-by-user-uploads-derek-buitenhuis https://speakerdeck.com/demuxed/things-developers-believe-ab...
  - zerokernel 9y ago
    Honestly I don't quite know how or why engineers keep working in a "all of this is fundamentally broken always has been and never will be fixed" domain, on a motivational basis.
- ouid 9y ago
  "all frame timestamps are monotonically increasing" wow...
fest 9y ago
This really hits if you try to merge audio which has been recorded separately from video (e.g. Raspberry Pi where at least a few years ago software did not support recording audio synchronously with video). And the worst thing is- it could vary from one device to other because the crystal oscillators used for image sensor clock have slightly different resonant frequency (which is caused by manufacturing tolerances when cutting the crystals).
orev 9y ago
Any tutorial will not get very far if every corner case is spelled out every time. The point of “learning” documents is to give enough concepts and useful information for the reader to have a basis to start building additional knowledge on top of. Getting into the weeds quickly derails this process.
- ouid 9y ago
  what's wrong with presenting everything in the correct abstraction to start with? Whether you a public school math class, or a git tutorial online, you always get the same thing. A list of procedures for using the tools in the most common cases you are likely to encounter. This seems like a good idea in the move fast and break things ideology, but what do we notice about people's skills in the real world? No one knows how git works, and no one knows any math by the time they leave high school. Give people a set of tools and prove to them that it solves every problem in some domain. If they can't solve the problems using the most primitive, complete toolset you can give them, then the case where they solve the problem is the edge case. As an example, often, we will have students write fractions in "lowest terms", then only present students with fractions in which the numerators and denominators only share a few, small common prime factors. But checking prime factors is absolutely the hardest way of solving this problem, and unless you're exhaustive in your search, you can't have any confidence that you actually have solved it. Students are intrinsically aware that this wold be tedious in general, and it gives them anxiety to know that they could have missed something, or not tried sufficiently large divisors. They have no confidence in their tools, and rightly so. The only reason they can solve these problems at all is because the instructor gives them problems which can be solved this way. The origin of the question "when will I use this in the real world?" is the instinct that the tool you have been given is only for edge cases, and cannot be relied upon. I think you are exactly wrong in your assessment. Getting into the weeds is the only way to learn, because almost everything in the world is weeds, some of those weeds just happen to be called crops.
  - dfabulich 9y ago
    > what's wrong with presenting everything in the correct abstraction to start with? Nobody can retain the material that way. Each thing you learn has to be attached to other concepts you already have, and your skills build on one another. Nobody teaches Peano's axioms of arithmetic to kindergarteners; they're still learning how to identify shapes, to compare quantities. When they eventually learn even the simplest proofs, they'll build on the basis of careful attention to detail that they learned while mastering basic arithmetic. Nobody teaches general relativity on the first day of college-level physics class. Students are still learning calculus at that point; even if they already had a calculus course, this is their first opportunity to apply it. And nobody teaches the fundamental proofs of calculus on the first day of calculus class; you can use sloppy language like "infinitesimals" to establish a good intuition for how to use derivatives and integrals. If you tried introducing this material from the bottom up, it wouldn't take. But if I'm wrong, sure, go try it and see how it works.
thearn4 9y ago
This was something I had to find out the hard way in the process of writing this https://github.com/thearn/webcam-pulse-detector https://github.com/thearn/webcam-pulse-detector Without interpolative resampling, it didn't work well for me at all.
userbinator 9y ago
It's true for video recorded by professional cameras, but if you check video recorded by most mobile phones you notice that it's wrong. It's unfortunate that so many tools assume constant framerate, because VFR is a useful compression technique in itself --- for sequences where there's basically no change between frames (few-second black scenes, for example), the encoder can stop continually encoding "no change" and just wait until there is.

gowan 9y ago

to be fair it is possible to do this from the command line with ffmpeg when using concat[1][2]. i did this when creating real time video from screencast in the chrome remote debugger. [1] https://ffmpeg.org/ffmpeg-formats.html#concat https://ffmpeg.org/ffmpeg-formats.html#concat [2] https://trac.ffmpeg.org/wiki/Slideshow https://trac.ffmpeg.org/wiki/Slideshow

jiggunjer 9y ago

In my experience even cameras with constant framerates aren't really constant, if you look at the timestamps the delay between frames typically varies a few (dozen) ms. So on average you might have 30fps, but if you sample a random second you might end up with 28-32 frames.

colund 9y ago
29.97?
- djsumdog 9y ago
  That comes from the old NTSC standard and the history of adding colour in a backwards compatible way: https://www.youtube.com/watch?v=3GJUM6pCpew https://www.youtube.com/watch?v=3GJUM6pCpew In the case that the above posters are mentioning, this probably has less to do with NTSC compatibility and more to do with bad timing hardware or cameras that attempt to do too much in PIO mode instead of using real encoding hardware.
  - mikepurvis 9y ago
    Glad to see that video linked; it's a terrific explanation.
  - jiggunjer 9y ago
    I wouldn't say bad timing means bad hardware though, even expensive scientific cameras suffer from this. You need rather complex electronics to capture at a high and consistent rate. I think normally the encoding is done by software.

lovelearning 9y ago

[1] is a useful list of such assumptions. Found it very handy when programming AV handling. [1]: https://haasn.xyz/posts/2016-12-25-falsehoods-programmers-believe-about-%5Bvideo-stuff%5D.html https://haasn.xyz/posts/2016-12-25-falsehoods-programmers-be...

jsheard 9y ago
See also: this talk from a Vimeo engineer about the video encoding horrors they've encountered in the wild. https://www.youtube.com/watch?v=cRSO3RtUOOk https://www.youtube.com/watch?v=cRSO3RtUOOk / https://speakerdeck.com/demuxed/things-developers-believe-about-video-files-proven-wrong-by-user-uploads-derek-buitenhuis https://speakerdeck.com/demuxed/things-developers-believe-ab...
- zerokernel 9y ago
  Honestly I don't quite know how or why engineers keep working in a "all of this is fundamentally broken always has been and never will be fixed" domain, on a motivational basis.
ouid 9y ago
"all frame timestamps are monotonically increasing" wow...

fest 9y ago

This really hits if you try to merge audio which has been recorded separately from video (e.g. Raspberry Pi where at least a few years ago software did not support recording audio synchronously with video). And the worst thing is- it could vary from one device to other because the crystal oscillators used for image sensor clock have slightly different resonant frequency (which is caused by manufacturing tolerances when cutting the crystals).

orev 9y ago

Any tutorial will not get very far if every corner case is spelled out every time. The point of “learning” documents is to give enough concepts and useful information for the reader to have a basis to start building additional knowledge on top of. Getting into the weeds quickly derails this process.

ouid 9y ago
what's wrong with presenting everything in the correct abstraction to start with? Whether you a public school math class, or a git tutorial online, you always get the same thing. A list of procedures for using the tools in the most common cases you are likely to encounter. This seems like a good idea in the move fast and break things ideology, but what do we notice about people's skills in the real world? No one knows how git works, and no one knows any math by the time they leave high school. Give people a set of tools and prove to them that it solves every problem in some domain. If they can't solve the problems using the most primitive, complete toolset you can give them, then the case where they solve the problem is the edge case. As an example, often, we will have students write fractions in "lowest terms", then only present students with fractions in which the numerators and denominators only share a few, small common prime factors. But checking prime factors is absolutely the hardest way of solving this problem, and unless you're exhaustive in your search, you can't have any confidence that you actually have solved it. Students are intrinsically aware that this wold be tedious in general, and it gives them anxiety to know that they could have missed something, or not tried sufficiently large divisors. They have no confidence in their tools, and rightly so. The only reason they can solve these problems at all is because the instructor gives them problems which can be solved this way. The origin of the question "when will I use this in the real world?" is the instinct that the tool you have been given is only for edge cases, and cannot be relied upon. I think you are exactly wrong in your assessment. Getting into the weeds is the only way to learn, because almost everything in the world is weeds, some of those weeds just happen to be called crops.
- dfabulich 9y ago
  > what's wrong with presenting everything in the correct abstraction to start with? Nobody can retain the material that way. Each thing you learn has to be attached to other concepts you already have, and your skills build on one another. Nobody teaches Peano's axioms of arithmetic to kindergarteners; they're still learning how to identify shapes, to compare quantities. When they eventually learn even the simplest proofs, they'll build on the basis of careful attention to detail that they learned while mastering basic arithmetic. Nobody teaches general relativity on the first day of college-level physics class. Students are still learning calculus at that point; even if they already had a calculus course, this is their first opportunity to apply it. And nobody teaches the fundamental proofs of calculus on the first day of calculus class; you can use sloppy language like "infinitesimals" to establish a good intuition for how to use derivatives and integrals. If you tried introducing this material from the bottom up, it wouldn't take. But if I'm wrong, sure, go try it and see how it works.

thearn4 9y ago

This was something I had to find out the hard way in the process of writing this https://github.com/thearn/webcam-pulse-detector https://github.com/thearn/webcam-pulse-detector Without interpolative resampling, it didn't work well for me at all.

userbinator 9y ago

It's true for video recorded by professional cameras, but if you check video recorded by most mobile phones you notice that it's wrong. It's unfortunate that so many tools assume constant framerate, because VFR is a useful compression technique in itself --- for sequences where there's basically no change between frames (few-second black scenes, for example), the encoder can stop continually encoding "no change" and just wait until there is.