10 ms·

Why are video games graphics challenging? Productionizing rendering algorithms

309 points by bartwr 6y ago

gabereiser 6y ago
Such a good write up. It’s mind boggling the amount of problems you have to solve with fakes, hacks, trickery to fool the human player’s eye. What’s worse is that all of these techniques across all of the mentioned areas are evolving and changing as hardware changes. It’s impossible for one person to keep up anymore. The art pipeline is pretty much cookie cutter from films with an added step of simplification of the mesh and baking normals. The rest is up to the engine and graphics programmers to get working as intended, the downstream of this is the design team. It’s gotten easier in the animation department. Easier in the texturing department. Easier due to painting and PBR, but it’s gotten exponentially more difficult in the core engineering, engine level department. It’s a murder pit of the mind. I’ve written a few game engines. I don’t write game engines.
- deleted 6y ago
  [deleted]
- proc0 6y ago
  I think game engines are the most complex software category to be engineered. They aim to simulate reality for a given piece of hardware. It seems as hardware gets better game engines implement something closer to the physics equations that describe the universe. With some AI as shortcuts for biological formations, we'll soon have to reinvent the wheel less and take less algorithmic shortcuts whenever we need a basic simulation with gravity, lighting, materials etc.
  - hombre_fatal 6y ago
    Then there’s the other impossible half of the problem: making a game that’s fun. Such a hard nebulous problem that it’s easy to procrastinate by focusing on the technical stuff as a break.
    - gabereiser 6y ago
      This. More game companies need to focus on core fun and play than technical achievements and micro transactions.
      - wongarsu 6y ago
        In mobile games the motto seems to be "why make it fun when you can make it addicting instead".
        paulryanrogers 6y ago
        Isn't that intentional now? Interesting enough players want to continue yet uncomfortable enough they want to pay to bypass arbitrary gating.
    - AnIdiotOnTheNet 6y ago
      Indeed. It's one of the pitfalls I see a lot of first-time game developers fall into: they don't actually have a fun game, but rather than work on that they keep adding complex systems and digging in technically in the likely vein hope that fun will somehow emerge from that.
  - ReactiveJelly 6y ago
    How do you know that simulating reality is the most complex thing software can do? How do you measure "complex", anyway?
    - hutzlibu 6y ago
      Probably because reality is indefinite complex and that it would take a whole reality to correctly simulate a reality. So the hard part is to make aproximations that run in REAL TIME (unlike scientific simulations, that have some time to run). And how do you measure complex? Well, how about the number of changing variables involved? Simulating and solving chess is very easy in comparison. You have only very limited and fixed variables. Simulating a physic sandbox game with advanced chemistry and even biology that is supposed to run in real time - seems impossible at this moment on the other hand. Too many connected details.
    - proc0 6y ago
      Complexity as in software. It has many moving parts, and I think it includes most if not all of computer science areas, like networking for multiplayer, cutting edge rendering, client-server applications for managing updates and also the multiplayer, AI for NPCs but now AI for animations and environment generation, and the list is really long. The only exception I can think of is software for rockets/spaceships and rovers, which have to operate under very unique circumstances.
  - yurishimo 6y ago
    And that's assuming we want to play/make games with "realistic" simulations! I'm sure we've all played those sandbox physics games where a few values were tweaked just to far in one direction. Sometimes that can be really fun! I'd e interested to know if these more advanced tools take that into account, or will there be frequent cases of needing to roll your own physics engine? I'm trying to think of an example, but I believe I remember reading about a game that purposefully used a slightly lighter gravity for some models to help the player feel more powerful (?) than other characters. I wish I could remember, but the idea is the same. Physics that are dynamic could be a fun/useful gameplay mechanic and I hope we don't see those types of games go away because everyone decides it's too complicated/not worth it to use a different physics algorithm than the one shipped in some console SDK.
    - proc0 6y ago
      Yeah, movement speed is also perceptively slower in a game for some reason and realistic speeds are boring so most FPSs have the player running at car speeds. These tweaks will always be there to "fix" our limitations in reality. Teleporting is another example where nobody is really interested in traveling distances like IRL.
      - allo37 6y ago
        Hey if I could fast-travel IRL I probably would :p
- turing_complete 6y ago
  Isn't it what makes game programming interesting, though? I work at a FAANG company and do game and engine development as side projects and would like to do that full-time at some point. In game programming you actually solve difficult and interesting problems and really care about the performance your code. For the majority of other software engineering positions, managing dependencies seems to be the hardest challenge.
  - awild 6y ago
    With the size of game development teams it really depends on what you're working on (And I'd speculate, that less people work on engines than on game systems). Only the really big studios roll their own engines. Looking at something like Watchdogs, there are also people involved in writing Quest and Dialogue systems etc. for the game designers and these things are quite mundane.
    - ajuc 6y ago
      > Quest and Dialogue systems etc. for the game designers and these things are quite mundane. Depends what you want from them. I've written a small quest and dialog system for my indie game (never finished, as most of my side projects :( ), and it was quite involved. I used a graph programming library ( https://github.com/ajuc/pefjs https://github.com/ajuc/pefjs ) to create graphs of nodes where each edge is a blocking condition (is the $MONSTER alive? is player near $NPC?) and each node is next step in a quest (with some actions if needed). Then there was a in-memory database allowing to record arbitrary metadata that quests might require. Stuff like "have the player killed that monster", "does the player have the quest item equipped", "have the player talked with one of these 4 people about X", "has any NPC seen the player near the $PLACE during $EVENT". I briefly considered adding gossips - if $NPC1 seen player commit murder and $NPC2 meets $NPC1 - then $NPC2 remembers that too, and remember who told them. But then I calculated how much memory that would need and it's staggering. And players won't notice anyway most likely. But it would be so cool. Once you start recording data about data (does $NPC2 know about the fact that $NPC1 seen player commit murder?) it's hard to know where to stop. Anyway, even without these - there's lots of that stuff and it needs indexing and filtering. I considered using some in-memory SQL database or datalog, but eventually just rolled my own with basic hashmap indexing. Then there was dialog system and I wanted more variety than the same responses every time, so I created sets of possible greetings and goodbyes, confirmations, negations, reaction to repeatedly asking the same question, etc. Then the system chooses randomly from the list (each "kind" of NPC has separate list - one for working class characters, one for strangers, one for scholarly types, and special lists for important NPCs). You can also parametrize dialogs with data from quest database - so you can for example ask the player about his new $COLOR $BRAND vehicle when he comes to see the NPC the first time after the purchase. There's lots of interesting stuff to do there, even if it's not visual.
- 10x-dev 6y ago
  I've been trying to learn graphics programming for 10 years now, in my spare time. The amount of complexity that was slowly introduced for the sake of better graphics makes entering the field now a daunting task. I have about a dozen books, each thousands of pages long. People who have been writing games since the 90s added this knowledge incrementally. Learning it now is such a mess. I am partially disappointed with myself, partially with the state at which the field has arrived.
  - mrfox321 6y ago
    Look at the bright side, it will take less time to catch up than it took them to build the foundation!
  - gabereiser 6y ago
    How I started was I got interested in 3D modeling because I wanted to make a Star Wars fan film in 1996 with my 8mm. I bought a DVD set on how to do modeling and texturing in 3D Studio Max from the Gnomon School. It was amazing. Film level quality secrets and techniques. I got pretty good. This lead me down a rabbit hole of how do I take this and apply it towards games. 3D modeling and texturing for games. Then I started working on coding and realized that OpenGL/DirectX used similar concepts to what I learned in 3D studio max. There’s a scene, a camera, a mesh, a light, materials, textures, shaders (SM1.0), and all these things combined give you your lit scene using some advanced but cookie-cutter math.
- FartyMcFarter 6y ago
  > The rest is up to the engine and graphics programmers Don't forget technical artists, who implement and use tools to make art content run well on game engines. This could mean optimizing shaders, tweaking content streaming, tweaking occluders, etc.
  - gabereiser 6y ago
    I left out a lot of median positions between the extremes. Yes. Audio engineering is in there too. Technical artists sit between gfx programmers and the artists, also working with level designers to create the desired effects too.
bigcorp-slave 6y ago
A great write up, and don’t miss the publications section of his website. The paper on multiframe superresolution is a tour-de-force and crushes a ton of deep learning results despite not being learning based.
makepanic 6y ago
If one is interested in the rate limited images, try the archive.org mirror: https://web.archive.org/web/20201228071137/https://bartwronski.com/2020/12/27/why-are-video-games-graphics-still-a-challenge-productionizing-rendering-algorithms/ https://web.archive.org/web/20201228071137/https://bartwrons...
jasmcole 6y ago
What a great blog you have! I'm interested that your path to a research position occurred via game development. Is that common? (I've subscribed to your feed, the combination of topics you cover is uncannily similar to my own interests.)
- bartwr 6y ago
  My path to research is very uncommon - at least in the US. Being honest it was very disappointing when I tried to switch the industries. Getting a SWE position was very easy (and many ex-gamedevs do it and are very successful at FAANG; many of them consider new jobs boring, but not toxic like gamedev, no crunch, a few times larger salaries, stability...), but any inquiries about anything research related to recruiter were met with "you don't have a PhD? we will not even interview you for such a position". My "way in" was to just accept a generic offer and then later switch team to work inside Research (when you don't need to pass N layers of "recruiter abstraction" and can talk with colleagues and present your past work freely it becomes much easier). But still, among my collaborators maybe 5% don't have a PhD? All of this was a surprise to me and made me question my life path and life choices. I'm originally from Poland in Eastern Europe and the only reason to consider a CS PhD would be if you wanted to teach. And I even considered it, but never occurred to me to do it for any kind of R&D work - mostly because of quality of Polish academic "research" being very low, much worse than the industry. And I am still in lots of self-doubt and questioning if I belong in this environment. Questions like "who was your advisor / what was your grad group?" when introducing yourself to new people become almost micro-aggressions. Also my focus on R&D ("solving unsolved problems") is very different than academic incentives ("use PhD interns to do projects, then write papers and get citations").
Koiwai 6y ago
Nice write up, BTW this is a re-post: https://news.ycombinator.com/item?id=25557431 https://news.ycombinator.com/item?id=25557431 but you're the original author, hmm...
- bartwr 6y ago
  So I don't know how HN really works, but FWIW I posted it here 10mins after the post went live. Then it just went down into obscurity (almost always my submissions to HN do), so I just ignored it, later saw a repost and ignored it, "oh well, whatever". Then I was really surprised next dat that it went back to life and with "posted 2 hours ago", I thought it was some bug or some mod interaction, but most likely I just don't understand how HN as a platform works...
slx26 6y ago
Hey, bartwr, thank you very much for writing so clearly and in such an approachable way about the work you do, providing a more holistic (over)view of the field and the challenges involved. I assume that your other more technically detailed posts are also very valuable to the experts on the field, but it's great that you shared something like this that's more accessible to people with less knowledge on the topic. I can only hope that if more people understand the challenges involved in a field, maybe it will also become easier to get them on board and be more open minded and less reluctant to change when new solutions are explored and transitions are needed to continue making progress.
gfxgirl 6y ago
It's not that hard ... Or rather it's almost all solved problems. The proof is the number of 1-4 person indie teams making games that look like AAA games. Two examples. https://store.steampowered.com/app/417290/Ghost_of_a_Tale/ https://store.steampowered.com/app/417290/Ghost_of_a_Tale/ Made almost entirely by one person https://store.steampowered.com/app/242760/The_Forest/ https://store.steampowered.com/app/242760/The_Forest/ Small team There are tons of others. Sure, someone had to solve those issues but they mostly packaged the solutions into Unreal, Unity, etc. It's like concentrating on making cameras instead of making movies. Sure there are new techniques that might give your movie some special edge (like when they invented bullet time for The Matrix) but a good story (or a well designed game) doesn't really need the technical tricks. In fact some of the most famous games are designed around the limits of the tech (either the limits of the hardware or the limits of their particular engine). Note I've written several AAA game engines and have friends that work on AAA engines for others. They are all admitting / lamenting now that engines are basically a commodity and their skills are not needed. They're turning into scripters for the artist and designers and management is pushing to just license the engines.
- saberience 6y ago
  Teams of 1-4 people making indie games are either: 1) Using an existing product like Unity or Unreal Engine or GameMaker. 2) Using their own engine but with very basic graphics. "Ghost of a Tale" uses Unity. "The Forest" uses Unity. So yeah, it's not "that hard" if you use a product which has already had millions of developer hours put into it like Unity or UnrealEngine. Take away any prepackaged game engine and see if either of those two developer teams would have been able to make those games. :)
  - imadr 6y ago
    I would add that using an existing engine rendering pipeline with shaders is way easier that rolling your own. Making a small software renderer from scratch is a very good way to get a good grasp on how game engines work internally.
  - bilbo0s 6y ago
adwn 6y ago
Truly amazing and mind-boggling what modern hardware is capable of when software doesn't float atop a dozen layers of abstraction.
GistNoesis 6y ago
Now we have engines that ship for free all these techniques prepackaged. The whole ecosystem, hardware included, has been built to serve this way of rendering. With general programming GPU, like CUDA, we can have the flexibility of software and the performance of traditional hardware pipelines. Hopefully we will soon be able to add some neural networks in the rendering pipeline. Instead of computing expensive light effects, we could be faking them. For example the foliage could probably be handled better by just rendering a semantic mask indicating where the grass is, and let the neural network do some in-painting. Neural rendering seems like a great way to get some speed-ups. On a powerful machine you render with High quality (eventually cinematographic quality), and you render the same scene with very low quality, and you train a neural network to convert the low quality into the high quality. At game time, you render low-quality and predict with the neural network the high quality image. It's a trade-off of sacrificing accuracy to gain speed but because humans are bad at evaluating accuracy, it's poised to be worth it. The additional advantage of using neural rendering is that you don't need to spend as much creating the assets. You can get rid of one of the biggest mistake in 3D history, the invention of the triangle mesh. Representing objects as a list of textured triangle is just a bad representation. The main reason they are bad is that they are 2D surfaces in a 3D world which introduce some geometrical constraints that need to be solved explicitly when you deform the object, or you will get some artifacts Alternative representations like dense or sparse point-clouds either in feature-space or color-space, or implicit fields representation don't suffer from these. Their main issue was the need to be converted back to triangles for rendering. These representations are continuously deform-able, which means you can use machine learning to infer them.
- knolan 6y ago
  There are already denoising and upscaling neural nets being used to reduce the cost of ray tracing. Lighting in games is also benefiting from sparse voxel based global illumination. GPUs are very efficient at pushing triangles and the tooling and skill to build meshes is well established. Games like Teardown are showing alternative ways but when development time and budgets are so tight most games will use what’s there rather than risk new technologies.
mxmxnxor 6y ago
What about Dreams engine? It doesn't use triangles for rendering, how it fits into techniques described in the article?
- bartwr 6y ago
  I mention the Dreams in the passing - as it is such a special gem and special case that it would require much more care and analysis, probably material for multiple posts. What makes Dreams special is that it is a platform for creativity and content creation by its players. It's not a machine for making AAA products that cost 100s of millions of dollars and takes teams of hundreds people. (Also - I don't have any data nor insider knowledge despite working at Sony, but I am almost sure this project lost Sony money because of never-ending production... Which only speaks great of Sony for supporting it) Brilliant creators of Dreams didn't care if software like Maya or 3ds Max are not compatible with their approach - as it wasn't the target audience. They didn't care for scalability of the tools, pipelines, and the renderer in terms of business practices, parallelism, finishing on time, reusing existing technology, packing with certain features. If your core feature is something unique like this, many other constraints can just go away. Is it the future of rendering? I don't think so. Btw. Media Molecule's Alex Evans (amazing person, he used to create beautiful art for demoscene) has long history of innovative rendering tech - like LBP used lots of voxel and volumetric representations (look for "Voxels in LittleBigPlanet 2" Siggraph talk) on PS3 and it also didn't catch up. Different use-cases = different constraint. See also another example I mentioned in the post - Claybook, it uses SDFs for rendering representation (and physics!), but is also a very unique and special case.
  - mxmxnxor 6y ago
    But don't you think that Dreams engine is the sight of more superior rendering approach than current triangles pipeline and will dominate in the future? Can't we achieve more in terms of performance and flexibility if we skip triangles pipeline and do all rasterization and pixel color calculation in computed shaders? For example you don't need to carefully choose tesselation level (or spend memory for many different zoom level lods and then trying to avoid lods switching artifacts) for models and fases with many smooth curves if you can do resolution independent rasterization straight from sdf/nurbs/b-splines in realtime. And it seems like having much less overhead due to 2*2 pixel overdrawing of many triangles just for visual curve effect.
mxmxnxor 6y ago
Why don't use pixels caching solution for dense geometry problem described in the article? For example 2d/ui/gui engines don't render detalized vector shapes (like glyphs) from scratch every frame - glyphs rasterized to pixels, pixels stored to cache and used for subsequent rendering until zoom changes. And if you cache pixels for each 2x zoom level you just need to downsample next zoom level whan user changes zoom or resizes object (which is very fast comparing to rendering from scratch)
- rasz 6y ago
  Congratulations, you just invented https://en.wikipedia.org/wiki/Microsoft_Talisman https://en.wikipedia.org/wiki/Microsoft_Talisman This is also how Trespasser dealt with far geometry in 1998 https://en.wikipedia.org/wiki/Trespasser_(video_game)#Development https://en.wikipedia.org/wiki/Trespasser_(video_game)#Develo... Started top be used again with introduction of directx 9.0 under a name Impostors http://www.tommironty.fi/screenshots/Imposters/dynamic_2d_imposters_a_simple_.php.htm http://www.tommironty.fi/screenshots/Imposters/dynamic_2d_im... https://software.intel.com/content/www/us/en/develop/articles/impostors-made-easy.html https://software.intel.com/content/www/us/en/develop/article... https://developer.nvidia.com/gpugems/gpugems3/part-iv-image-effects/chapter-21-true-impostors https://developer.nvidia.com/gpugems/gpugems3/part-iv-image-... here at work in COD warzone: https://www.youtube.com/watch?t=691&v=mvdTtl27TpM https://www.youtube.com/watch?t=691&v=mvdTtl27TpM
- bartwr 6y ago
  In the article, I mention a few ways of dealing with it - from "old school" impostors/billboards (think of precomputed sprites), simplified representation, to finally temporal anti-aliasing (reusing pixels in screen space). All of those work well under some strong constraints and are used today, however all have problems, trade-offs, artifacts - whole point of the post. Temporal AA is the most general and mature solution, but at the same time, many gamers hate it due to introducing some blurriness through imperfect resampling (and makes some very small population motion sick) - just web-search for "temporal antialiasing reddit" and see the sentiment of many users.
  - deleted 6y ago
dirtyid 6y ago
I'm curious how far you can push real time rendering by regressing back to 480p or even VHS level resolution on a CRT monitor. There's a level of abstraction your brain does to fill in the details as long as sufficient layers of reality is being rendered even in excruciating low resolution.
phendrenad2 6y ago
This is about photorealistic game graphics, which are indeed hard, because there isn't enough computational power to really simulate the world, so it's 99% tricks that look "almost right". I'd like to add to this that another reason game graphics are challenging is schools don't teach it. If you get a compsci degree and focus on video game graphics, the most you'll likely do in the courses is make an incredibly unoptimized software raytracer render some shiny spheres, a task that could not be farther from actual game engine development.

309 points by bartwr 6y ago

Such a good write up. It’s mind boggling the amount of problems you have to solve with fakes, hacks, trickery to fool the human player’s eye. What’s worse is that all of these techniques across all of the mentioned areas are evolving and changing as hardware changes. It’s impossible for one person to keep up anymore. The art pipeline is pretty much cookie cutter from films with an added step of simplification of the mesh and baking normals. The rest is up to the engine and graphics programmers to get working as intended, the downstream of this is the design team. It’s gotten easier in the animation department. Easier in the texturing department. Easier due to painting and PBR, but it’s gotten exponentially more difficult in the core engineering, engine level department. It’s a murder pit of the mind. I’ve written a few game engines. I don’t write game engines.

deleted 6y ago
[deleted]
proc0 6y ago
I think game engines are the most complex software category to be engineered. They aim to simulate reality for a given piece of hardware. It seems as hardware gets better game engines implement something closer to the physics equations that describe the universe. With some AI as shortcuts for biological formations, we'll soon have to reinvent the wheel less and take less algorithmic shortcuts whenever we need a basic simulation with gravity, lighting, materials etc.
- hombre_fatal 6y ago
  Then there’s the other impossible half of the problem: making a game that’s fun. Such a hard nebulous problem that it’s easy to procrastinate by focusing on the technical stuff as a break.
  - gabereiser 6y ago
    This. More game companies need to focus on core fun and play than technical achievements and micro transactions.
    - wongarsu 6y ago
      In mobile games the motto seems to be "why make it fun when you can make it addicting instead".
      - paulryanrogers 6y ago
        Isn't that intentional now? Interesting enough players want to continue yet uncomfortable enough they want to pay to bypass arbitrary gating.
  - AnIdiotOnTheNet 6y ago
    Indeed. It's one of the pitfalls I see a lot of first-time game developers fall into: they don't actually have a fun game, but rather than work on that they keep adding complex systems and digging in technically in the likely vein hope that fun will somehow emerge from that.
- ReactiveJelly 6y ago
  How do you know that simulating reality is the most complex thing software can do? How do you measure "complex", anyway?
  - hutzlibu 6y ago
    Probably because reality is indefinite complex and that it would take a whole reality to correctly simulate a reality. So the hard part is to make aproximations that run in REAL TIME (unlike scientific simulations, that have some time to run). And how do you measure complex? Well, how about the number of changing variables involved? Simulating and solving chess is very easy in comparison. You have only very limited and fixed variables. Simulating a physic sandbox game with advanced chemistry and even biology that is supposed to run in real time - seems impossible at this moment on the other hand. Too many connected details.
  - proc0 6y ago
    Complexity as in software. It has many moving parts, and I think it includes most if not all of computer science areas, like networking for multiplayer, cutting edge rendering, client-server applications for managing updates and also the multiplayer, AI for NPCs but now AI for animations and environment generation, and the list is really long. The only exception I can think of is software for rockets/spaceships and rovers, which have to operate under very unique circumstances.
- yurishimo 6y ago
  And that's assuming we want to play/make games with "realistic" simulations! I'm sure we've all played those sandbox physics games where a few values were tweaked just to far in one direction. Sometimes that can be really fun! I'd e interested to know if these more advanced tools take that into account, or will there be frequent cases of needing to roll your own physics engine? I'm trying to think of an example, but I believe I remember reading about a game that purposefully used a slightly lighter gravity for some models to help the player feel more powerful (?) than other characters. I wish I could remember, but the idea is the same. Physics that are dynamic could be a fun/useful gameplay mechanic and I hope we don't see those types of games go away because everyone decides it's too complicated/not worth it to use a different physics algorithm than the one shipped in some console SDK.
  - proc0 6y ago
    Yeah, movement speed is also perceptively slower in a game for some reason and realistic speeds are boring so most FPSs have the player running at car speeds. These tweaks will always be there to "fix" our limitations in reality. Teleporting is another example where nobody is really interested in traveling distances like IRL.
    - allo37 6y ago
      Hey if I could fast-travel IRL I probably would :p
turing_complete 6y ago
Isn't it what makes game programming interesting, though? I work at a FAANG company and do game and engine development as side projects and would like to do that full-time at some point. In game programming you actually solve difficult and interesting problems and really care about the performance your code. For the majority of other software engineering positions, managing dependencies seems to be the hardest challenge.
- awild 6y ago
  With the size of game development teams it really depends on what you're working on (And I'd speculate, that less people work on engines than on game systems). Only the really big studios roll their own engines. Looking at something like Watchdogs, there are also people involved in writing Quest and Dialogue systems etc. for the game designers and these things are quite mundane.
  - ajuc 6y ago
    > Quest and Dialogue systems etc. for the game designers and these things are quite mundane. Depends what you want from them. I've written a small quest and dialog system for my indie game (never finished, as most of my side projects :( ), and it was quite involved. I used a graph programming library ( https://github.com/ajuc/pefjs https://github.com/ajuc/pefjs ) to create graphs of nodes where each edge is a blocking condition (is the $MONSTER alive? is player near $NPC?) and each node is next step in a quest (with some actions if needed). Then there was a in-memory database allowing to record arbitrary metadata that quests might require. Stuff like "have the player killed that monster", "does the player have the quest item equipped", "have the player talked with one of these 4 people about X", "has any NPC seen the player near the $PLACE during $EVENT". I briefly considered adding gossips - if $NPC1 seen player commit murder and $NPC2 meets $NPC1 - then $NPC2 remembers that too, and remember who told them. But then I calculated how much memory that would need and it's staggering. And players won't notice anyway most likely. But it would be so cool. Once you start recording data about data (does $NPC2 know about the fact that $NPC1 seen player commit murder?) it's hard to know where to stop. Anyway, even without these - there's lots of that stuff and it needs indexing and filtering. I considered using some in-memory SQL database or datalog, but eventually just rolled my own with basic hashmap indexing. Then there was dialog system and I wanted more variety than the same responses every time, so I created sets of possible greetings and goodbyes, confirmations, negations, reaction to repeatedly asking the same question, etc. Then the system chooses randomly from the list (each "kind" of NPC has separate list - one for working class characters, one for strangers, one for scholarly types, and special lists for important NPCs). You can also parametrize dialogs with data from quest database - so you can for example ask the player about his new $COLOR $BRAND vehicle when he comes to see the NPC the first time after the purchase. There's lots of interesting stuff to do there, even if it's not visual.
10x-dev 6y ago
I've been trying to learn graphics programming for 10 years now, in my spare time. The amount of complexity that was slowly introduced for the sake of better graphics makes entering the field now a daunting task. I have about a dozen books, each thousands of pages long. People who have been writing games since the 90s added this knowledge incrementally. Learning it now is such a mess. I am partially disappointed with myself, partially with the state at which the field has arrived.
- mrfox321 6y ago
  Look at the bright side, it will take less time to catch up than it took them to build the foundation!
- gabereiser 6y ago
  How I started was I got interested in 3D modeling because I wanted to make a Star Wars fan film in 1996 with my 8mm. I bought a DVD set on how to do modeling and texturing in 3D Studio Max from the Gnomon School. It was amazing. Film level quality secrets and techniques. I got pretty good. This lead me down a rabbit hole of how do I take this and apply it towards games. 3D modeling and texturing for games. Then I started working on coding and realized that OpenGL/DirectX used similar concepts to what I learned in 3D studio max. There’s a scene, a camera, a mesh, a light, materials, textures, shaders (SM1.0), and all these things combined give you your lit scene using some advanced but cookie-cutter math.
FartyMcFarter 6y ago
> The rest is up to the engine and graphics programmers Don't forget technical artists, who implement and use tools to make art content run well on game engines. This could mean optimizing shaders, tweaking content streaming, tweaking occluders, etc.
- gabereiser 6y ago
  I left out a lot of median positions between the extremes. Yes. Audio engineering is in there too. Technical artists sit between gfx programmers and the artists, also working with level designers to create the desired effects too.

bigcorp-slave 6y ago

A great write up, and don’t miss the publications section of his website. The paper on multiframe superresolution is a tour-de-force and crushes a ton of deep learning results despite not being learning based.

makepanic 6y ago

If one is interested in the rate limited images, try the archive.org mirror: https://web.archive.org/web/20201228071137/https://bartwronski.com/2020/12/27/why-are-video-games-graphics-still-a-challenge-productionizing-rendering-algorithms/ https://web.archive.org/web/20201228071137/https://bartwrons...

jasmcole 6y ago

What a great blog you have! I'm interested that your path to a research position occurred via game development. Is that common? (I've subscribed to your feed, the combination of topics you cover is uncannily similar to my own interests.)

bartwr 6y ago
My path to research is very uncommon - at least in the US. Being honest it was very disappointing when I tried to switch the industries. Getting a SWE position was very easy (and many ex-gamedevs do it and are very successful at FAANG; many of them consider new jobs boring, but not toxic like gamedev, no crunch, a few times larger salaries, stability...), but any inquiries about anything research related to recruiter were met with "you don't have a PhD? we will not even interview you for such a position". My "way in" was to just accept a generic offer and then later switch team to work inside Research (when you don't need to pass N layers of "recruiter abstraction" and can talk with colleagues and present your past work freely it becomes much easier). But still, among my collaborators maybe 5% don't have a PhD? All of this was a surprise to me and made me question my life path and life choices. I'm originally from Poland in Eastern Europe and the only reason to consider a CS PhD would be if you wanted to teach. And I even considered it, but never occurred to me to do it for any kind of R&D work - mostly because of quality of Polish academic "research" being very low, much worse than the industry. And I am still in lots of self-doubt and questioning if I belong in this environment. Questions like "who was your advisor / what was your grad group?" when introducing yourself to new people become almost micro-aggressions. Also my focus on R&D ("solving unsolved problems") is very different than academic incentives ("use PhD interns to do projects, then write papers and get citations").

Koiwai 6y ago

Nice write up, BTW this is a re-post: https://news.ycombinator.com/item?id=25557431 https://news.ycombinator.com/item?id=25557431 but you're the original author, hmm...

bartwr 6y ago
So I don't know how HN really works, but FWIW I posted it here 10mins after the post went live. Then it just went down into obscurity (almost always my submissions to HN do), so I just ignored it, later saw a repost and ignored it, "oh well, whatever". Then I was really surprised next dat that it went back to life and with "posted 2 hours ago", I thought it was some bug or some mod interaction, but most likely I just don't understand how HN as a platform works...

slx26 6y ago

Hey, bartwr, thank you very much for writing so clearly and in such an approachable way about the work you do, providing a more holistic (over)view of the field and the challenges involved. I assume that your other more technically detailed posts are also very valuable to the experts on the field, but it's great that you shared something like this that's more accessible to people with less knowledge on the topic. I can only hope that if more people understand the challenges involved in a field, maybe it will also become easier to get them on board and be more open minded and less reluctant to change when new solutions are explored and transitions are needed to continue making progress.

gfxgirl 6y ago

It's not that hard ... Or rather it's almost all solved problems. The proof is the number of 1-4 person indie teams making games that look like AAA games. Two examples. https://store.steampowered.com/app/417290/Ghost_of_a_Tale/ https://store.steampowered.com/app/417290/Ghost_of_a_Tale/ Made almost entirely by one person https://store.steampowered.com/app/242760/The_Forest/ https://store.steampowered.com/app/242760/The_Forest/ Small team There are tons of others. Sure, someone had to solve those issues but they mostly packaged the solutions into Unreal, Unity, etc. It's like concentrating on making cameras instead of making movies. Sure there are new techniques that might give your movie some special edge (like when they invented bullet time for The Matrix) but a good story (or a well designed game) doesn't really need the technical tricks. In fact some of the most famous games are designed around the limits of the tech (either the limits of the hardware or the limits of their particular engine). Note I've written several AAA game engines and have friends that work on AAA engines for others. They are all admitting / lamenting now that engines are basically a commodity and their skills are not needed. They're turning into scripters for the artist and designers and management is pushing to just license the engines.

saberience 6y ago
Teams of 1-4 people making indie games are either: 1) Using an existing product like Unity or Unreal Engine or GameMaker. 2) Using their own engine but with very basic graphics. "Ghost of a Tale" uses Unity. "The Forest" uses Unity. So yeah, it's not "that hard" if you use a product which has already had millions of developer hours put into it like Unity or UnrealEngine. Take away any prepackaged game engine and see if either of those two developer teams would have been able to make those games. :)
- imadr 6y ago
  I would add that using an existing engine rendering pipeline with shaders is way easier that rolling your own. Making a small software renderer from scratch is a very good way to get a good grasp on how game engines work internally.
- bilbo0s 6y ago

adwn 6y ago

Truly amazing and mind-boggling what modern hardware is capable of when software doesn't float atop a dozen layers of abstraction.

GistNoesis 6y ago

Now we have engines that ship for free all these techniques prepackaged. The whole ecosystem, hardware included, has been built to serve this way of rendering. With general programming GPU, like CUDA, we can have the flexibility of software and the performance of traditional hardware pipelines. Hopefully we will soon be able to add some neural networks in the rendering pipeline. Instead of computing expensive light effects, we could be faking them. For example the foliage could probably be handled better by just rendering a semantic mask indicating where the grass is, and let the neural network do some in-painting. Neural rendering seems like a great way to get some speed-ups. On a powerful machine you render with High quality (eventually cinematographic quality), and you render the same scene with very low quality, and you train a neural network to convert the low quality into the high quality. At game time, you render low-quality and predict with the neural network the high quality image. It's a trade-off of sacrificing accuracy to gain speed but because humans are bad at evaluating accuracy, it's poised to be worth it. The additional advantage of using neural rendering is that you don't need to spend as much creating the assets. You can get rid of one of the biggest mistake in 3D history, the invention of the triangle mesh. Representing objects as a list of textured triangle is just a bad representation. The main reason they are bad is that they are 2D surfaces in a 3D world which introduce some geometrical constraints that need to be solved explicitly when you deform the object, or you will get some artifacts Alternative representations like dense or sparse point-clouds either in feature-space or color-space, or implicit fields representation don't suffer from these. Their main issue was the need to be converted back to triangles for rendering. These representations are continuously deform-able, which means you can use machine learning to infer them.

knolan 6y ago
There are already denoising and upscaling neural nets being used to reduce the cost of ray tracing. Lighting in games is also benefiting from sparse voxel based global illumination. GPUs are very efficient at pushing triangles and the tooling and skill to build meshes is well established. Games like Teardown are showing alternative ways but when development time and budgets are so tight most games will use what’s there rather than risk new technologies.

mxmxnxor 6y ago

What about Dreams engine? It doesn't use triangles for rendering, how it fits into techniques described in the article?

bartwr 6y ago
I mention the Dreams in the passing - as it is such a special gem and special case that it would require much more care and analysis, probably material for multiple posts. What makes Dreams special is that it is a platform for creativity and content creation by its players. It's not a machine for making AAA products that cost 100s of millions of dollars and takes teams of hundreds people. (Also - I don't have any data nor insider knowledge despite working at Sony, but I am almost sure this project lost Sony money because of never-ending production... Which only speaks great of Sony for supporting it) Brilliant creators of Dreams didn't care if software like Maya or 3ds Max are not compatible with their approach - as it wasn't the target audience. They didn't care for scalability of the tools, pipelines, and the renderer in terms of business practices, parallelism, finishing on time, reusing existing technology, packing with certain features. If your core feature is something unique like this, many other constraints can just go away. Is it the future of rendering? I don't think so. Btw. Media Molecule's Alex Evans (amazing person, he used to create beautiful art for demoscene) has long history of innovative rendering tech - like LBP used lots of voxel and volumetric representations (look for "Voxels in LittleBigPlanet 2" Siggraph talk) on PS3 and it also didn't catch up. Different use-cases = different constraint. See also another example I mentioned in the post - Claybook, it uses SDFs for rendering representation (and physics!), but is also a very unique and special case.
- mxmxnxor 6y ago
  But don't you think that Dreams engine is the sight of more superior rendering approach than current triangles pipeline and will dominate in the future? Can't we achieve more in terms of performance and flexibility if we skip triangles pipeline and do all rasterization and pixel color calculation in computed shaders? For example you don't need to carefully choose tesselation level (or spend memory for many different zoom level lods and then trying to avoid lods switching artifacts) for models and fases with many smooth curves if you can do resolution independent rasterization straight from sdf/nurbs/b-splines in realtime. And it seems like having much less overhead due to 2*2 pixel overdrawing of many triangles just for visual curve effect.

mxmxnxor 6y ago

Why don't use pixels caching solution for dense geometry problem described in the article? For example 2d/ui/gui engines don't render detalized vector shapes (like glyphs) from scratch every frame - glyphs rasterized to pixels, pixels stored to cache and used for subsequent rendering until zoom changes. And if you cache pixels for each 2x zoom level you just need to downsample next zoom level whan user changes zoom or resizes object (which is very fast comparing to rendering from scratch)

rasz 6y ago
Congratulations, you just invented https://en.wikipedia.org/wiki/Microsoft_Talisman https://en.wikipedia.org/wiki/Microsoft_Talisman This is also how Trespasser dealt with far geometry in 1998 https://en.wikipedia.org/wiki/Trespasser_(video_game)#Development https://en.wikipedia.org/wiki/Trespasser_(video_game)#Develo... Started top be used again with introduction of directx 9.0 under a name Impostors http://www.tommironty.fi/screenshots/Imposters/dynamic_2d_imposters_a_simple_.php.htm http://www.tommironty.fi/screenshots/Imposters/dynamic_2d_im... https://software.intel.com/content/www/us/en/develop/articles/impostors-made-easy.html https://software.intel.com/content/www/us/en/develop/article... https://developer.nvidia.com/gpugems/gpugems3/part-iv-image-effects/chapter-21-true-impostors https://developer.nvidia.com/gpugems/gpugems3/part-iv-image-... here at work in COD warzone: https://www.youtube.com/watch?t=691&v=mvdTtl27TpM https://www.youtube.com/watch?t=691&v=mvdTtl27TpM
bartwr 6y ago
In the article, I mention a few ways of dealing with it - from "old school" impostors/billboards (think of precomputed sprites), simplified representation, to finally temporal anti-aliasing (reusing pixels in screen space). All of those work well under some strong constraints and are used today, however all have problems, trade-offs, artifacts - whole point of the post. Temporal AA is the most general and mature solution, but at the same time, many gamers hate it due to introducing some blurriness through imperfect resampling (and makes some very small population motion sick) - just web-search for "temporal antialiasing reddit" and see the sentiment of many users.
- deleted 6y ago

dirtyid 6y ago

I'm curious how far you can push real time rendering by regressing back to 480p or even VHS level resolution on a CRT monitor. There's a level of abstraction your brain does to fill in the details as long as sufficient layers of reality is being rendered even in excruciating low resolution.

phendrenad2 6y ago

This is about photorealistic game graphics, which are indeed hard, because there isn't enough computational power to really simulate the world, so it's 99% tricks that look "almost right". I'd like to add to this that another reason game graphics are challenging is schools don't teach it. If you get a compsci degree and focus on video game graphics, the most you'll likely do in the courses is make an incredibly unoptimized software raytracer render some shiny spheres, a task that could not be farther from actual game engine development.