8 ms·

Don’t Copy That Surface

91 points by feross 6y ago

pimlottc 6y ago
What does “surface” mean here? It’s not explained or even used once in the article body.
- nvrspyx 6y ago
  From skimming, it has something to do with media buffers, but I don't really know what that is as a lay person with next to zero web development experience/knowledge.
- pavlov 6y ago
  A surface typically means a 2D image buffer managed by an API. For example, a graphics API like Metal or Direct3D lets you create surfaces that reside either in GPU or CPU memory. Or, an OS-level interprocess API like Apple's IOSurface lets you create shared surfaces that can be accessed by multiple processes.
- magicalist 6y ago
  > What does “surface” mean here? It’s not explained or even used once in the article body. Yeah, seems like a poor choice of title without the explanation. For rendering implementations, "surface" usually refers to a part of memory being rendered into as output or that something else has rendered into that is being used as input, like an image buffer that webcam output is dumped into. One example set of docs: https://docs.microsoft.com/en-us/windows/win32/direct3d9/direct3d-surfaces https://docs.microsoft.com/en-us/windows/win32/direct3d9/dir... I don't know if I've ever really thought about the name. I guess maybe at some point it was more straightforward and a surface was just a part of memory that was going to be blitted directly to screen and so in that sense you're painting a "surface" of that area of the screen? I'd be interested if anyone knows a more detailed etymology.
  - foobiekr 6y ago
    "Canvas" was already taken at the time these APIs were becoming popular.
    - kevingadd 6y ago
      typically terminology-wise I see "canvas" used to refer to a surface targeted by paint operations, vs other surfaces which may be read-only (contain an image you loaded from disk or a snapshot of a canvas at a previous point in time). In D3D terminology I think it's almost always been "surfaces" and "render targets", where the latter is something you can paint into. Surfaces are often broken down into levels, and don't necessarily contain image data (they can contain depth/stencil data for 3D rendering, for example)
      - foobiekr 6y ago
        Yes. In the pre-DirectX games SDK for Windows 95 it was a mix of “surface” and DIB (eg CreateCompatibleDIB). The win32 API was already using “canvas” in many places. Surface actually shows up in OS/2 Presentation Manager as well.
  - brucedawson 6y ago
    Good point. I don't think anybody gave much thought to the title after I came up with it - I suspect it is easy to gloss over during the edit passes. I chose the title so long ago that I can't really remember what I was thinking.
    - toast0 6y ago
      It seems like you were trying to reference 'Don't Copy That Floppy'
- thaumasiotes 6y ago
  https://docs.microsoft.com/en-us/windows/win32/direct3d9/direct3d-surfaces https://docs.microsoft.com/en-us/windows/win32/direct3d9/dir... > A surface represents a linear area of display memory and usually resides in the display memory of the display card, although surfaces can exist in system memory. https://stackoverflow.com/questions/1033445/what-exactly-is-a-surface-in-opengl-es-or-direct3d https://stackoverflow.com/questions/1033445/what-exactly-is-... > broadly speaking, a surface is some 2D image data.
nikanj 6y ago
Wait, someone actually cares about performance these days? I'm not sarcastic here, this was a really cool read evocative of Raymond Chen and the old NT Debugging blog. I'd like to read more blogs where the solution to bad performance isn't just 1) throw more servers at the issue 2) rewrite the service in a different language / framework and hope to not hit the same pathological case again.
- plorkyeran 6y ago
  This post was written by Bruce Dawson and is very similar to his normal blog content: https://randomascii.wordpress.com/ https://randomascii.wordpress.com/
- stefan_ 6y ago
  Making sure surfaces aren't copied needlessly is actually a rather common optimization, and particularly so on mobile devices where great care is taken that all subsystems can share memory - e.g. the display silicon needs to be able to use textures from the video decoding directly for scanout. Not doing so escalates quickly - copying a 4k10bpp buffer 60 times per second takes up ~5Gbit of memory bandwidth.
- brundolf 6y ago
  Performance is incredibly important in web browsers - almost as important as the system itself, but it's a much younger domain so there's more left to be figured out. Chromium and Firefox regularly generate some of the coolest and most interesting optimization-related blog posts out there. Examples: https://hacks.mozilla.org/category/performance/ https://hacks.mozilla.org/category/performance/
- dataflow 6y ago
  > Wait, someone actually cares about performance these days? I think Bruce Dawson is literally the only person I can think of who still publishes anything like this for Windows, but yes, one person does!
Diggsey 6y ago
You can't fault MS for [their documentation](https://docs.microsoft.com/en-us/windows/win32/api/mfobjects/nf-mfobjects-imfmediabuffer-lock#remarks https://docs.microsoft.com/en-us/windows/win32/api/mfobjects...)... > This method may internally allocate some memory, so if the buffer supports the IMF2DBuffer interface, you should use the IMF2DBuffer::Lock2D method to lock the buffer instead. For 2-D buffers, the Lock2DSize method can be more efficient than the Lock method, depending on the MF2DBuffer_LockFlags value you specify. Calling Lock2DSize with MF2DBuffer_LockFlags_Read won’t incur a copy back when the buffer is unlocked and calling it with MF2DBuffer_LockFlags_Write won’t incur a copy from the internal buffer. Calling Lock2DSize with LockFlags_ReadWrite behaves the same as Lock and Lock2D and will incur a both copy from and copy back when unlocked. The general guidance for best performance is to avoid using IMFMediaBuffer and IMF2DBuffer whenever possible and instead use IMF2DBuffer2 with the minimum required lock flags. Note that if the buffer is locked using Lock2D, the Lock method might return MF_E_INVALIDREQUEST.
- LegionMammal978 6y ago
  It looks like that was added in response to this issue. See comment 1 on the Chromium issue[0] and the associated PR on the documentation[1]. [0]: https://bugs.chromium.org/p/chromium/issues/detail?id=1071180 https://bugs.chromium.org/p/chromium/issues/detail?id=107118... [1]: https://github.com/MicrosoftDocs/sdk-api/pull/207 https://github.com/MicrosoftDocs/sdk-api/pull/207
darig 6y ago
Who doesn't want 12 bit color?
wlesieutre 6y ago
Safe to assume this improvement will carry over to Electron? I’ve tried using Discord for video chat and screen sharing on a 4 year old laptop (2.9 ghz dual core i5) and it was not a pleasant experience.
ricardobeat 6y ago
Having a Google Meet tab open still fries even the most powerful MBP 16". The fans become so loud the noise cancelling starts failing for the microphone...

91 points by feross 6y ago

pimlottc 6y ago

What does “surface” mean here? It’s not explained or even used once in the article body.

nvrspyx 6y ago
From skimming, it has something to do with media buffers, but I don't really know what that is as a lay person with next to zero web development experience/knowledge.
pavlov 6y ago
A surface typically means a 2D image buffer managed by an API. For example, a graphics API like Metal or Direct3D lets you create surfaces that reside either in GPU or CPU memory. Or, an OS-level interprocess API like Apple's IOSurface lets you create shared surfaces that can be accessed by multiple processes.
magicalist 6y ago
> What does “surface” mean here? It’s not explained or even used once in the article body. Yeah, seems like a poor choice of title without the explanation. For rendering implementations, "surface" usually refers to a part of memory being rendered into as output or that something else has rendered into that is being used as input, like an image buffer that webcam output is dumped into. One example set of docs: https://docs.microsoft.com/en-us/windows/win32/direct3d9/direct3d-surfaces https://docs.microsoft.com/en-us/windows/win32/direct3d9/dir... I don't know if I've ever really thought about the name. I guess maybe at some point it was more straightforward and a surface was just a part of memory that was going to be blitted directly to screen and so in that sense you're painting a "surface" of that area of the screen? I'd be interested if anyone knows a more detailed etymology.
- foobiekr 6y ago
  "Canvas" was already taken at the time these APIs were becoming popular.
  - kevingadd 6y ago
    typically terminology-wise I see "canvas" used to refer to a surface targeted by paint operations, vs other surfaces which may be read-only (contain an image you loaded from disk or a snapshot of a canvas at a previous point in time). In D3D terminology I think it's almost always been "surfaces" and "render targets", where the latter is something you can paint into. Surfaces are often broken down into levels, and don't necessarily contain image data (they can contain depth/stencil data for 3D rendering, for example)
    - foobiekr 6y ago
      Yes. In the pre-DirectX games SDK for Windows 95 it was a mix of “surface” and DIB (eg CreateCompatibleDIB). The win32 API was already using “canvas” in many places. Surface actually shows up in OS/2 Presentation Manager as well.
- brucedawson 6y ago
  Good point. I don't think anybody gave much thought to the title after I came up with it - I suspect it is easy to gloss over during the edit passes. I chose the title so long ago that I can't really remember what I was thinking.
  - toast0 6y ago
    It seems like you were trying to reference 'Don't Copy That Floppy'
thaumasiotes 6y ago
https://docs.microsoft.com/en-us/windows/win32/direct3d9/direct3d-surfaces https://docs.microsoft.com/en-us/windows/win32/direct3d9/dir... > A surface represents a linear area of display memory and usually resides in the display memory of the display card, although surfaces can exist in system memory. https://stackoverflow.com/questions/1033445/what-exactly-is-a-surface-in-opengl-es-or-direct3d https://stackoverflow.com/questions/1033445/what-exactly-is-... > broadly speaking, a surface is some 2D image data.

nikanj 6y ago

Wait, someone actually cares about performance these days? I'm not sarcastic here, this was a really cool read evocative of Raymond Chen and the old NT Debugging blog. I'd like to read more blogs where the solution to bad performance isn't just 1) throw more servers at the issue 2) rewrite the service in a different language / framework and hope to not hit the same pathological case again.

plorkyeran 6y ago
This post was written by Bruce Dawson and is very similar to his normal blog content: https://randomascii.wordpress.com/ https://randomascii.wordpress.com/
stefan_ 6y ago
Making sure surfaces aren't copied needlessly is actually a rather common optimization, and particularly so on mobile devices where great care is taken that all subsystems can share memory - e.g. the display silicon needs to be able to use textures from the video decoding directly for scanout. Not doing so escalates quickly - copying a 4k10bpp buffer 60 times per second takes up ~5Gbit of memory bandwidth.
brundolf 6y ago
Performance is incredibly important in web browsers - almost as important as the system itself, but it's a much younger domain so there's more left to be figured out. Chromium and Firefox regularly generate some of the coolest and most interesting optimization-related blog posts out there. Examples: https://hacks.mozilla.org/category/performance/ https://hacks.mozilla.org/category/performance/
dataflow 6y ago
> Wait, someone actually cares about performance these days? I think Bruce Dawson is literally the only person I can think of who still publishes anything like this for Windows, but yes, one person does!

Diggsey 6y ago

You can't fault MS for [their documentation](https://docs.microsoft.com/en-us/windows/win32/api/mfobjects/nf-mfobjects-imfmediabuffer-lock#remarks https://docs.microsoft.com/en-us/windows/win32/api/mfobjects...)... > This method may internally allocate some memory, so if the buffer supports the IMF2DBuffer interface, you should use the IMF2DBuffer::Lock2D method to lock the buffer instead. For 2-D buffers, the Lock2DSize method can be more efficient than the Lock method, depending on the MF2DBuffer_LockFlags value you specify. Calling Lock2DSize with MF2DBuffer_LockFlags_Read won’t incur a copy back when the buffer is unlocked and calling it with MF2DBuffer_LockFlags_Write won’t incur a copy from the internal buffer. Calling Lock2DSize with LockFlags_ReadWrite behaves the same as Lock and Lock2D and will incur a both copy from and copy back when unlocked. The general guidance for best performance is to avoid using IMFMediaBuffer and IMF2DBuffer whenever possible and instead use IMF2DBuffer2 with the minimum required lock flags. Note that if the buffer is locked using Lock2D, the Lock method might return MF_E_INVALIDREQUEST.

LegionMammal978 6y ago
It looks like that was added in response to this issue. See comment 1 on the Chromium issue[0] and the associated PR on the documentation[1]. [0]: https://bugs.chromium.org/p/chromium/issues/detail?id=1071180 https://bugs.chromium.org/p/chromium/issues/detail?id=107118... [1]: https://github.com/MicrosoftDocs/sdk-api/pull/207 https://github.com/MicrosoftDocs/sdk-api/pull/207

darig 6y ago

Who doesn't want 12 bit color?

wlesieutre 6y ago

Safe to assume this improvement will carry over to Electron? I’ve tried using Discord for video chat and screen sharing on a 4 year old laptop (2.9 ghz dual core i5) and it was not a pleasant experience.

ricardobeat 6y ago

Having a Google Meet tab open still fries even the most powerful MBP 16". The fans become so loud the noise cancelling starts failing for the microphone...