In reality it's the size yes, but there is an inherent delay because of the way B-frames work. You'll need to buffer 2-3 frames worth 33ms each (for NTSC), so 6

In reality it's the size yes, but there is an inherent delay because of the way B-frames work. You'll need to buffer 2-3 frames worth 33ms each (for NTSC), so 66-99ms delay regardless of throughput. Audio doesn't have such a concept, you get 33ms worth of audio, you can play it right away. Of course, you can opt to not have b-frames and negate that issue.