6 ms·

Screen and asset image scaling for games

65 points by speeder 13y ago

steventhedev 13y ago
There's an upscaling algorithm he missed: http://research.microsoft.com/en-us/um/people/kopf/pixelart/paper/pixel.pdf http://research.microsoft.com/en-us/um/people/kopf/pixelart/...
- speeder 13y ago
  I don't missed it, I ignored it on purpose, for two reasons: One, I did not understood it well enough to try to explain even the basics. Two, I could not find a binary or source I could compile easily to make a example image. Also you can put a "three" there: it is not widely used, while even the recent xBR has a couple of plug-in implementations on emulators for example (and has binaries) EDIT: Lack of binary is also why I did not made sections for 2xSal, SuperEagle and so on... If people help me finding that kind of stuff or creating the needed images, I will gladly update the article.
  - archivator 13y ago
    This looks like a FOSS implementation of Depixelizing Pixel Art - https://github.com/jerith/depixel https://github.com/jerith/depixel
- johnsoft 13y ago
  That's probably my favorite research paper ever. The web link in the PDF is offline, but it looks like they moved everything to the Microsoft site -- http://research.microsoft.com/en-us/um/people/kopf/pixelart/ http://research.microsoft.com/en-us/um/people/kopf/pixelart/. I just spent a good 15 minutes mousing over the image comparisons again
vinkelhake 13y ago
While not really general image scaling, there's a lot of work being done on post-processing in emulators to try to emulate CRT displays (like the aperture grille) or even the quirks of NTSC. http://emulation-general.wikia.com/wiki/Shaders_and_Filters http://emulation-general.wikia.com/wiki/Shaders_and_Filters
- speeder 13y ago
  Heh, it mentions that noone can emulate yet shadow mask CRT. Those are what I mention in my article (TVs that have triangle pixels composed of 3 circles) I never use current CRTs emulation because I guess I never saw TVs that could be emulated, I think all CRT TVs I saw consoles running were of shadow mask type, having almost no scanline (and I was a curious child that used to put amplifying lenses in front of TVs to see their pixels...)
cageface 13y ago
You run into pretty much exactly the same set of problems in the audio DSP realm whenever you have to deal with audio of different sample rates or with variable sample rate algorithms like tape delay emulations. Most typically in that domain you also use windowed sinc filters and there's a ton of literature on the tradeoffs of specific window designs, as well as very fast fixed point implementations etc. It's all pretty interesting stuff and trying to make it run efficiently on modern mobile hardware is a fun challenge.
MProgrammer 13y ago
A great way to deal with scaling assets for mobile is to create your art in a vector format to begin with. For Uncle Slam on iOS we did all the artwork in Illustrator and exported to PNG at three sizes: iPhone 1x, iPhone 2x (and iPad), and iPad 2x. The results are sharp images with the right level of detail and no scaling artifacts. We even took the exact same artwork and made an 8 foot tall banner with it, and it looks great!
- speeder 13y ago
  On Kidoteca some of our art is made that way, but not all of it. Our backgrounds for example frequently feature "digital painting" style, with stuff that is outright impossible to do in vector.
- shachar 13y ago
  You assume here that size doesn't matter, when it might. If the difference is only initial download size in your game, and is a few dozen MBs, then yes I agree. but for some applications, this is content that is loaded in runtime, and since size DOES matter in this regard, some applications will need to upscale images.
jblow 13y ago
Some time ago, I wrote some articles on image scaling that go into a bit more technical depth (with a specific focus on mipmapping): http://number-none.com/product/Mipmapping,%20Part%201/index.html http://number-none.com/product/Mipmapping,%20Part%201/index.... http://number-none.com/product/Mipmapping,%20Part%202/index.html http://number-none.com/product/Mipmapping,%20Part%202/index.... A problem I get into at the end of the second article is that gamma-correction is very important for good image scaling results. However, almost nobody gamma corrects during scaling, even today.
- csense 13y ago
  > gamma-correction is very important for good image scaling results This is a great observation.
- james4k 13y ago
  I might be misunderstanding, but shouldn't gamma correction happen after drawing your scene?
  - jblow 13y ago
    You are thinking of a different gamma-correction step. The reason you gamma-correct during rendering is because monitors expect data to already be gamma-corrected, and it is this expectation that causes the problem I am talking about. Data in a bitmap is not linear with the light-intensity of the colors; it is gamma'd! The simplest kind of mipmapping is a box filter, where you are just averaging 4 pixel values at once into a new pixel value. Thinking just of grayscale pixels, if you add 4 pixels that are each 1.0 (if you are thinking in 8 bits, 1.0 == 255), and divide by 4, you get 1.0 again. If you add two pixels that are 1.0, and two that are 0, you get a value of 0.5. Which would be fine if your bitmap were stored in units that are linear with the actual intensity of light; but they are not, because they are in a weird gamma! What you are going to get is something like pow(0.5, 2.2) which is way too dark. Thus when you don't gamma-correct during mipmapping, bright things lose definition and smudge into dim things way more than actually happens in real-life when you see objects from far away.
zokier 13y ago
One alternative upscaling method (for non-integer ratios) is to first upscale with no interpolation a integer ratio to make a bigger image than the target size. Then the large image is down-scaled (with some suitable algorithm/filtering/interpolation) to final size. Afaik it is what Apple does with it's "Retina" MacBooks and thus I have dubbed it "retina scaling", but there is probably a proper name for the method too. edit: comparison http://imgur.com/a/fC8iQ#1 http://imgur.com/a/fC8iQ#1
- alanctgardner2 13y ago
  In a class I took on image processing, this was offered as the definitive way to do non-integer scaling. Apple certainly didn't invent it.
city41 13y ago
When offering upscaling options, it's great to offer all the things mentioned in the article. But please always offer nearest neighbor as an option too. I want to see the original pixels as close to their original form as possible, and so do many gamers. Pixel art is great, the "blockiness" is part of its charm.
- speeder 13y ago
  You remembered me of some stupid browsers that insist in filtering the image, making some specialized pixel art sites look absolutely horrible.
- Narishma 13y ago
  That's only true for PC games, where monitors did indeed produce blocky pixels. Arcade, console and early computer games that connected to TVs were designed to take advantage of the "deficiencies" of those screens to produce smoothed images, or the illusion of more colours than the systems were capable of, among other effects. So if you want to play those games as they were meant to be, you either output directly (without any upscaling or filters) to an old CRT TV, or you use those NTSC filters or shaders that most emulators have that try to mimic (to various degrees of success) the way an old TV displays images.
  - city41 13y ago
    I disagree, and so do most/all retro gamers nowadays. The golden standard back then were the standard res RGB monitors that arcade games used. Consumer TVs in America at best had composite input and paled terribly in comparison. But we all dreamed of our games looking as crisp as they do in the arcade. Even the NES had an arcade counterpart where you could experience NES games in crisp, beautiful RGB. Standard res RGB monitors are not as crisp as PC monitors. But they are still quite nice and really do let the individual pixels shine through. Most retro gamers now typically hook up their consoles via SCART to a CRT that accepts pure RGB, typically a Sony PVM. I've got all my consoles and my arcade boards running on a PVM and can really easily see the pixels making up the art. here's a few examples I grabbed off the web: http://imgur.com/a/GXHyf http://imgur.com/a/GXHyf They look even better in person. Yeah most emulators provide filters that give you scan lines and all that jazz (none of them look quite right). And of course an RGB monitor does not look at all like an old game running in a modern emulator with no filters. But I would still definitely say you can see and appreciate each individual pixel.
darkmighty 13y ago
The reason linear filters don't work for upscaling, say a Lanczos or Sinc, is that the base supposition made for generating the filters is that the source is "bandlimited" -- that is, all images are samples of some source which can at most vary once every pixel. Problem is, while that works very well for regions that are supposed to be smooth -- textures and stuff -- it fails at edges, since edges have infinite bandwidth. The result is a wash out as you can see for linear/bicubic interpolation. Of course, it does work almost perfectly for downscaling, because in this case we're not creating any edges, just reconstructing the original edges -- an those vary at most once every pixel.
- jblow 13y ago
  Actually, the effect happens just as much during downscaling. It is just not as noticeable, because your artifacts happen at subpixel scale. But they do affect the output colors. The math does not really know the difference between upscaling and downscaling.
Lerc 13y ago
I'll add my 2c with an article that I wrote some years ago. http://www.screamingduck.com/Article.php?ArticleID=35&Show=ABCE http://www.screamingduck.com/Article.php?ArticleID=35&Show=A... It's a look at kind of a interpolation weighted by scale2x.
zackmorris 13y ago
What we did for Return to Dark Castle (which ran at a fixed 640x480, 16 bit color) is give the user the option to scale by nearest neighbor, interpolation, or both (mixed 50% with each other). The "both" option worked surprisingly well, especially since it was a "quick fix"!

65 points by speeder 13y ago

There's an upscaling algorithm he missed: http://research.microsoft.com/en-us/um/people/kopf/pixelart/paper/pixel.pdf http://research.microsoft.com/en-us/um/people/kopf/pixelart/...

speeder 13y ago
I don't missed it, I ignored it on purpose, for two reasons: One, I did not understood it well enough to try to explain even the basics. Two, I could not find a binary or source I could compile easily to make a example image. Also you can put a "three" there: it is not widely used, while even the recent xBR has a couple of plug-in implementations on emulators for example (and has binaries) EDIT: Lack of binary is also why I did not made sections for 2xSal, SuperEagle and so on... If people help me finding that kind of stuff or creating the needed images, I will gladly update the article.
- archivator 13y ago
  This looks like a FOSS implementation of Depixelizing Pixel Art - https://github.com/jerith/depixel https://github.com/jerith/depixel
johnsoft 13y ago
That's probably my favorite research paper ever. The web link in the PDF is offline, but it looks like they moved everything to the Microsoft site -- http://research.microsoft.com/en-us/um/people/kopf/pixelart/ http://research.microsoft.com/en-us/um/people/kopf/pixelart/. I just spent a good 15 minutes mousing over the image comparisons again

vinkelhake 13y ago

While not really general image scaling, there's a lot of work being done on post-processing in emulators to try to emulate CRT displays (like the aperture grille) or even the quirks of NTSC. http://emulation-general.wikia.com/wiki/Shaders_and_Filters http://emulation-general.wikia.com/wiki/Shaders_and_Filters

speeder 13y ago
Heh, it mentions that noone can emulate yet shadow mask CRT. Those are what I mention in my article (TVs that have triangle pixels composed of 3 circles) I never use current CRTs emulation because I guess I never saw TVs that could be emulated, I think all CRT TVs I saw consoles running were of shadow mask type, having almost no scanline (and I was a curious child that used to put amplifying lenses in front of TVs to see their pixels...)

cageface 13y ago

You run into pretty much exactly the same set of problems in the audio DSP realm whenever you have to deal with audio of different sample rates or with variable sample rate algorithms like tape delay emulations. Most typically in that domain you also use windowed sinc filters and there's a ton of literature on the tradeoffs of specific window designs, as well as very fast fixed point implementations etc. It's all pretty interesting stuff and trying to make it run efficiently on modern mobile hardware is a fun challenge.

MProgrammer 13y ago

A great way to deal with scaling assets for mobile is to create your art in a vector format to begin with. For Uncle Slam on iOS we did all the artwork in Illustrator and exported to PNG at three sizes: iPhone 1x, iPhone 2x (and iPad), and iPad 2x. The results are sharp images with the right level of detail and no scaling artifacts. We even took the exact same artwork and made an 8 foot tall banner with it, and it looks great!

speeder 13y ago
On Kidoteca some of our art is made that way, but not all of it. Our backgrounds for example frequently feature "digital painting" style, with stuff that is outright impossible to do in vector.
shachar 13y ago
You assume here that size doesn't matter, when it might. If the difference is only initial download size in your game, and is a few dozen MBs, then yes I agree. but for some applications, this is content that is loaded in runtime, and since size DOES matter in this regard, some applications will need to upscale images.

jblow 13y ago

Some time ago, I wrote some articles on image scaling that go into a bit more technical depth (with a specific focus on mipmapping): http://number-none.com/product/Mipmapping,%20Part%201/index.html http://number-none.com/product/Mipmapping,%20Part%201/index.... http://number-none.com/product/Mipmapping,%20Part%202/index.html http://number-none.com/product/Mipmapping,%20Part%202/index.... A problem I get into at the end of the second article is that gamma-correction is very important for good image scaling results. However, almost nobody gamma corrects during scaling, even today.

csense 13y ago
> gamma-correction is very important for good image scaling results This is a great observation.
james4k 13y ago
I might be misunderstanding, but shouldn't gamma correction happen after drawing your scene?
- jblow 13y ago
  You are thinking of a different gamma-correction step. The reason you gamma-correct during rendering is because monitors expect data to already be gamma-corrected, and it is this expectation that causes the problem I am talking about. Data in a bitmap is not linear with the light-intensity of the colors; it is gamma'd! The simplest kind of mipmapping is a box filter, where you are just averaging 4 pixel values at once into a new pixel value. Thinking just of grayscale pixels, if you add 4 pixels that are each 1.0 (if you are thinking in 8 bits, 1.0 == 255), and divide by 4, you get 1.0 again. If you add two pixels that are 1.0, and two that are 0, you get a value of 0.5. Which would be fine if your bitmap were stored in units that are linear with the actual intensity of light; but they are not, because they are in a weird gamma! What you are going to get is something like pow(0.5, 2.2) which is way too dark. Thus when you don't gamma-correct during mipmapping, bright things lose definition and smudge into dim things way more than actually happens in real-life when you see objects from far away.

zokier 13y ago

One alternative upscaling method (for non-integer ratios) is to first upscale with no interpolation a integer ratio to make a bigger image than the target size. Then the large image is down-scaled (with some suitable algorithm/filtering/interpolation) to final size. Afaik it is what Apple does with it's "Retina" MacBooks and thus I have dubbed it "retina scaling", but there is probably a proper name for the method too. edit: comparison http://imgur.com/a/fC8iQ#1 http://imgur.com/a/fC8iQ#1

alanctgardner2 13y ago
In a class I took on image processing, this was offered as the definitive way to do non-integer scaling. Apple certainly didn't invent it.

city41 13y ago

When offering upscaling options, it's great to offer all the things mentioned in the article. But please always offer nearest neighbor as an option too. I want to see the original pixels as close to their original form as possible, and so do many gamers. Pixel art is great, the "blockiness" is part of its charm.

speeder 13y ago
You remembered me of some stupid browsers that insist in filtering the image, making some specialized pixel art sites look absolutely horrible.
Narishma 13y ago
That's only true for PC games, where monitors did indeed produce blocky pixels. Arcade, console and early computer games that connected to TVs were designed to take advantage of the "deficiencies" of those screens to produce smoothed images, or the illusion of more colours than the systems were capable of, among other effects. So if you want to play those games as they were meant to be, you either output directly (without any upscaling or filters) to an old CRT TV, or you use those NTSC filters or shaders that most emulators have that try to mimic (to various degrees of success) the way an old TV displays images.
- city41 13y ago
  I disagree, and so do most/all retro gamers nowadays. The golden standard back then were the standard res RGB monitors that arcade games used. Consumer TVs in America at best had composite input and paled terribly in comparison. But we all dreamed of our games looking as crisp as they do in the arcade. Even the NES had an arcade counterpart where you could experience NES games in crisp, beautiful RGB. Standard res RGB monitors are not as crisp as PC monitors. But they are still quite nice and really do let the individual pixels shine through. Most retro gamers now typically hook up their consoles via SCART to a CRT that accepts pure RGB, typically a Sony PVM. I've got all my consoles and my arcade boards running on a PVM and can really easily see the pixels making up the art. here's a few examples I grabbed off the web: http://imgur.com/a/GXHyf http://imgur.com/a/GXHyf They look even better in person. Yeah most emulators provide filters that give you scan lines and all that jazz (none of them look quite right). And of course an RGB monitor does not look at all like an old game running in a modern emulator with no filters. But I would still definitely say you can see and appreciate each individual pixel.

darkmighty 13y ago

The reason linear filters don't work for upscaling, say a Lanczos or Sinc, is that the base supposition made for generating the filters is that the source is "bandlimited" -- that is, all images are samples of some source which can at most vary once every pixel. Problem is, while that works very well for regions that are supposed to be smooth -- textures and stuff -- it fails at edges, since edges have infinite bandwidth. The result is a wash out as you can see for linear/bicubic interpolation. Of course, it does work almost perfectly for downscaling, because in this case we're not creating any edges, just reconstructing the original edges -- an those vary at most once every pixel.

jblow 13y ago
Actually, the effect happens just as much during downscaling. It is just not as noticeable, because your artifacts happen at subpixel scale. But they do affect the output colors. The math does not really know the difference between upscaling and downscaling.

Lerc 13y ago

I'll add my 2c with an article that I wrote some years ago. http://www.screamingduck.com/Article.php?ArticleID=35&Show=ABCE http://www.screamingduck.com/Article.php?ArticleID=35&Show=A... It's a look at kind of a interpolation weighted by scale2x.

zackmorris 13y ago

What we did for Return to Dark Castle (which ran at a fixed 640x480, 16 bit color) is give the user the option to scale by nearest neighbor, interpolation, or both (mixed 50% with each other). The "both" option worked surprisingly well, especially since it was a "quick fix"!