4 ms·

Though QuickTime is only an abstraction of OS provided codecs on Macs, this move once again brings up the question in me: Is there any valid technical reason f

Though QuickTime is only an abstraction of OS provided codecs on Macs, this move once again brings up the question in me: Is there any valid technical reason for browsers to handle media files in any way other than wrapping OS provided codecs?

blocke 16y ago
Are you sure every codec on your system is kept up to date against the latest exploits? Every single one? Which is updated more often... Your browser or your codecs? Browsers are taking over certain functions because the OS side (both OS X and Windows) are a mess. I imagine it makes browsers easier to test too not having even more dependencies over the quirks of whatever dot release of the OS you are running.
- jws 16y ago
  Better yet, why do my codecs have access to anything on my computer other than their input stream and output stream. They probably want to run on their own core anyway, isolate them and forget about them. (Ok, I'll give you an H.264 or MPEG-2 being tied into my video hardware for great performance and low power consumption, but the rest of them…)
  - cube13 16y ago
    I'm going to only cover the desktop/laptop space. This approach actually makes a lot of sense in the smartphone/tablet market, especially with all the multi-core chips coming out. It's true that the bare minimum that a codec needs to do is take an input bytestream and output it to the various output(video and audio) bytestreams. So, for a hardware accelerated video, this is pretty simple-all you need to do is pipe your input stream to the hardware, and that will handle all the magic of getting it directly out to the screens and speakers. However, for non-native hardware accelerated codecs(i.e. not H.264 or MPEG-2), this process involves a lot more hardware and software to actually get something to the screen. The non-native hardware accelerated codecs need access to(at a bare minimum) the sound and video system APIs. Unless the codec developers are certifiably insane, the codec will be using DirectX(for Windows) or OpenGL(for everything else) to handle the actual writing to the screen, especially if you want to use the video hardware for faster decoding. While it's possible for browsers to wrap those API calls, it just makes a lot more effort on all developers involved, and doesn't really get you much security at the expense of both complexity and performance. So it makes sense for any codec to have access to the lower level APIs for displaying things to the screen. Sandboxing the process should be possible for the file inputs, but I'm not sure if it's really possible for file outputs in the cases where the codec is using a system-level API. EDIT: After reading what I wrote, this is a good argument for the browsers themselves wrapping the codecs and placing them into a sandbox. Unfortunately, that will never happen, because of the legal minefield that almost all have right now...
    - nitrogen 16y ago
      Actually, the codecs themselves usually aren't responsible for displaying to the screen via DirectX, OpenGL, etc. In the case of <video>, the browser needs to be able to mix the video into the web page, which means the browser needs the video data in its own memory, or as a pixmap/texture in video card memory. But, most codecs are going to be written in C and assembly language for speed, which brings the potential for buffer overflows and other low-level exploits. Plus, that video data does eventually make it to the kernel and then the video hardware (often via a separate overlay interface like DirectShow, Xv, or VDPAU, though that is probably not the case with web browsers), so a vulnerability at any point along the chain is a serious issue.
antimatter15 16y ago
So they don't have to deal with inconsistent behaviors on different platforms. You can pretty much mess up your os installation anyway you would like, but bundling ffmpeg, libtheora or libvpx and you no longer need to worry about different platforms or configs
- CountSessine 16y ago
  That converts a trivially solved technical problem into an insoluble political problem.
- wtallis 16y ago
  Unless you want hardware acceleration...
  - sid0 16y ago
    What precludes bundled libraries from having support for hardware acceleration? Firefox 4 accelerates full-screen WebM videos on Windows and Mac, for instance.
    - wtallis 16y ago
      You need platform-specific APIs to give you direct access to the relevant hardware, which you then need to be able to use efficiently. I'd rather see applications just feed the bytestream to the OS to decode using the best means available, even if it means support for new codecs is a bit slow. (Although an OS-level codec plugin architecture like QuickTime Components can get you software decoding for new or obscure codecs.) If my GPU has a high-quality, high-speed H.264 decoder, I shouldn't have to worry about which browsers will use it. They should all default to the OS-provided decoding and playback methods unless there are sound technical reasons not to. If I want to use a new codec that's not supported by the fixed-function decoder, I shouldn't have to worry about which web browsers have bothered to include the necessary OpenCL code to decode on my GPU. I should just be able to install one codec that hooks in to the OS's media framework, and all browsers should use that first.
sid0 16y ago
Is there any valid technical reason for browsers to handle media files in any way other than wrapping OS provided codecs? See http://weblogs.mozillazine.org/roc/archives/2009/06/directshow_and.html http://weblogs.mozillazine.org/roc/archives/2009/06/directsh.... The philosophical reasons are even more important though.
DarkShikari 16y ago
So that your browser vendor can force their philosophical decisions about file formats on their users. That's basically it. Security is supposedly an issue, but "clicking on a video file in your browser to play it" is no easier than clicking "open file" in a download prompt -- if there's a vulnerability in your media player, attackers don't need <video> to get to it.
- nitrogen 16y ago
  I think you may be slightly biased here :). A security exploit that requires clicking is a lot less serious than one that happens without user intervention. Since <video> and <object>/<embed> can play automatically, it makes sense to limit the tags' exposure to unknown code from the operating system's installed codecs.
- deleted 16y ago
  [deleted]

blocke 16y ago

Are you sure every codec on your system is kept up to date against the latest exploits? Every single one? Which is updated more often... Your browser or your codecs? Browsers are taking over certain functions because the OS side (both OS X and Windows) are a mess. I imagine it makes browsers easier to test too not having even more dependencies over the quirks of whatever dot release of the OS you are running.

jws 16y ago
Better yet, why do my codecs have access to anything on my computer other than their input stream and output stream. They probably want to run on their own core anyway, isolate them and forget about them. (Ok, I'll give you an H.264 or MPEG-2 being tied into my video hardware for great performance and low power consumption, but the rest of them…)
- cube13 16y ago
  I'm going to only cover the desktop/laptop space. This approach actually makes a lot of sense in the smartphone/tablet market, especially with all the multi-core chips coming out. It's true that the bare minimum that a codec needs to do is take an input bytestream and output it to the various output(video and audio) bytestreams. So, for a hardware accelerated video, this is pretty simple-all you need to do is pipe your input stream to the hardware, and that will handle all the magic of getting it directly out to the screens and speakers. However, for non-native hardware accelerated codecs(i.e. not H.264 or MPEG-2), this process involves a lot more hardware and software to actually get something to the screen. The non-native hardware accelerated codecs need access to(at a bare minimum) the sound and video system APIs. Unless the codec developers are certifiably insane, the codec will be using DirectX(for Windows) or OpenGL(for everything else) to handle the actual writing to the screen, especially if you want to use the video hardware for faster decoding. While it's possible for browsers to wrap those API calls, it just makes a lot more effort on all developers involved, and doesn't really get you much security at the expense of both complexity and performance. So it makes sense for any codec to have access to the lower level APIs for displaying things to the screen. Sandboxing the process should be possible for the file inputs, but I'm not sure if it's really possible for file outputs in the cases where the codec is using a system-level API. EDIT: After reading what I wrote, this is a good argument for the browsers themselves wrapping the codecs and placing them into a sandbox. Unfortunately, that will never happen, because of the legal minefield that almost all have right now...
  - nitrogen 16y ago
    Actually, the codecs themselves usually aren't responsible for displaying to the screen via DirectX, OpenGL, etc. In the case of <video>, the browser needs to be able to mix the video into the web page, which means the browser needs the video data in its own memory, or as a pixmap/texture in video card memory. But, most codecs are going to be written in C and assembly language for speed, which brings the potential for buffer overflows and other low-level exploits. Plus, that video data does eventually make it to the kernel and then the video hardware (often via a separate overlay interface like DirectShow, Xv, or VDPAU, though that is probably not the case with web browsers), so a vulnerability at any point along the chain is a serious issue.

antimatter15 16y ago

So they don't have to deal with inconsistent behaviors on different platforms. You can pretty much mess up your os installation anyway you would like, but bundling ffmpeg, libtheora or libvpx and you no longer need to worry about different platforms or configs

CountSessine 16y ago
That converts a trivially solved technical problem into an insoluble political problem.
wtallis 16y ago
Unless you want hardware acceleration...
- sid0 16y ago
  What precludes bundled libraries from having support for hardware acceleration? Firefox 4 accelerates full-screen WebM videos on Windows and Mac, for instance.
  - wtallis 16y ago
    You need platform-specific APIs to give you direct access to the relevant hardware, which you then need to be able to use efficiently. I'd rather see applications just feed the bytestream to the OS to decode using the best means available, even if it means support for new codecs is a bit slow. (Although an OS-level codec plugin architecture like QuickTime Components can get you software decoding for new or obscure codecs.) If my GPU has a high-quality, high-speed H.264 decoder, I shouldn't have to worry about which browsers will use it. They should all default to the OS-provided decoding and playback methods unless there are sound technical reasons not to. If I want to use a new codec that's not supported by the fixed-function decoder, I shouldn't have to worry about which web browsers have bothered to include the necessary OpenCL code to decode on my GPU. I should just be able to install one codec that hooks in to the OS's media framework, and all browsers should use that first.

sid0 16y ago

Is there any valid technical reason for browsers to handle media files in any way other than wrapping OS provided codecs? See http://weblogs.mozillazine.org/roc/archives/2009/06/directshow_and.html http://weblogs.mozillazine.org/roc/archives/2009/06/directsh.... The philosophical reasons are even more important though.

DarkShikari 16y ago

So that your browser vendor can force their philosophical decisions about file formats on their users. That's basically it. Security is supposedly an issue, but "clicking on a video file in your browser to play it" is no easier than clicking "open file" in a download prompt -- if there's a vulnerability in your media player, attackers don't need <video> to get to it.

nitrogen 16y ago
I think you may be slightly biased here :). A security exploit that requires clicking is a lot less serious than one that happens without user intervention. Since <video> and <object>/<embed> can play automatically, it makes sense to limit the tags' exposure to unknown code from the operating system's installed codecs.
deleted 16y ago
[deleted]