7 ms·

Can somebody who knows about this tell me how much of the novelty in this is CS and how much physics? For example in graphics all the physics of light is well u

Can somebody who knows about this tell me how much of the novelty in this is CS and how much physics? For example in graphics all the physics of light is well understood - it's just infeasible to compute - so all advances are in CS and how to sample better. Is this the case here? Or is it adding new models?

sillysaurus3 10y ago
For example in graphics all the physics of light is well understood - it's just infeasible to compute - so all advances are in CS and how to sample better. Nitpick: The physics of light is understood. Human vision is also understood. But critically, human vision is completely overlooked in most graphical applications. It's not accurate to say that "the path toward making a computer-generated video indistinguishable from a camcorder recording is just a matter of sampling better." There will need to be a fundamental shift in the industry's thinking before realism will be achieved. It's a fascinating topic that occupied many years of my life, though this is a bit off-topic.
- Le_SDT 10y ago
  I'm curious. What do you mean by "Human vision is completely overlooked"? Are you talking about color perception? 3d rendering? Thanks :)
  - dharma1 10y ago
    Maybe foveation? dynamic range perception?
  - sillysaurus3 10y ago
    There's pretty much no way to avoid writing a large comment in response to this. So, with apologies... It was my focus for a long time to achieve perfectly photorealistic rendering. I come from a gamedev background, and I've been fascinated since around age ten about how to get a computer to paint pictures. By 17 I was writing game engines mostly equivalent to quake, and used this portfolio to get into the industry. The next three years were spent getting as close as possible to the bleeding edge of real-time graphics development. I remember having long debates with a colleague about what high-dynamic range lighting "meant." "If we spin around in our office chair, our brains do not suddenly change the overall brightness of this office. Why should we be programming games to do this? Why is everyone doing things that way?" My concerns were of a more fundamental nature as well. What is a diffuse texture? A diffuse texture is a well-understood concept in realtime graphics. Anyone with a basic knowledge of shaders should immediately recognize: color = lighting * diffuse + ambient Trouble is, it doesn't correspond to reality even slightly. It's not even roughly close. It happens to look good to humans, and that's why we use it. But the trouble was, the further I tried to probe the mystery of realism in computer graphics, the more I ran against this phenomenon of "We use X technique because it looks good." So I turned to research papers. Books. The medical field. Everywhere that was remotely related to possible breakthroughs in photorealistic rendering. Research papers are excellent for assembling techniques, but not results. The books in human vision and color science were more promising, yet most of the industry seemed (and still seems) to pay little attention to them. Compare a book about color perception to, say, http://www.pbrt.org/ http://www.pbrt.org/ and you'll see a stark difference. Flip through the table of contents and you get transformations, shapes, primitives, color and radiometery, sampling, reflection, materials, texture, scattering, light sources, monte carlo integration... And for what? We know that these techniques simply do not produce computer-generated videos that a human will identify as a real-life image. It's not for lack of processing power. There is a disconnect between the old rules and those that will ultimately result in real-time realism, and you won't find it in that table of contents. Now, the trouble with writing all of this is that if I knew how to do it, I'd have done it already. It's a life-long search, and it's not so easy to refute an entire industry without being (rightly) dismissed. But if you wish to know what I suspect are the ways forward, it's this: Get a camera. Take photos. Compare these photos to the results of the algorithms you write. Iterate on your algorithms until they are producing results that match something that already captures nature, not our beliefs about how we ought to be able to capture nature. "Just throw in physical models and presto!" has not thus far been true. You'll notice, for example, that computer graphics in videogames have plateaued. They get more impressive with each generation, but that impressiveness does not get them progressively closer to looking real. Nor should it. A computer game tells a story. The closer it looks to real life, the more restricted the artists are, along with the rest of the design of the game. So we turn to the movie industry for hope. But it's restricted in exactly the same way. The research papers are all along the lines of new techniques to try, or studies of existing pipelines and how to deal with their complexities. It's not fundamental research. As someone who has spent his life in pursuit of realism in computer-generated video, my recommendation is this: Read DaVinci's journal. Pay attention to what each page is saying. He had to discover from first principles what makes a painting look real, and why. You'll notice that he spends most of his time talking about human vision and our perceptions of color. If someone is going to make this development happen, it's not going to come from the game industry, and it won't come from the movie industry. That leaves you. Hopefully this will encourage some of you to pursue this. Once you accept that most of the computer graphics industry isn't actually focused on achieving realism, you'll start to develop your own techniques. My hope is that this will eventually lead to a breakthrough.
    - JabavuAdams 10y ago
      > And for what? We know that these techniques simply do not produce computer-generated videos that a human will identify as a real-life image. This is actually testable. Some archviz images are indistinguishable from reality for non-graphics experts. > Get a camera. Take photos. Compare these photos to the results of the algorithms you write. Iterate on your algorithms until they are producing results that match something that already captures nature, not our beliefs about how we ought to be able to capture nature. Yes, very important. Art teachers try to reinforce this by saying "seek reference". It's true for graphics as well, and it's possible to be more empirical.
      - sillysaurus3 10y ago
        Some archviz images are indistinguishable from reality for non-graphics experts. Photos, yes. Not video. It's an important distinction because the path to photorealistic video won't pass through impressive-looking still frames. Visual processing of movement is quite different.
    - adrianm 10y ago
      I thought you were going to go with a different angle on this. If all you care about is photorealism, well, we'll have to agree to disagree that the current state of the art is incapable of that (given enough time). What I thought your original point was is that 3D rendering ONLY cares about photorealism. Physically based renderers have been greatly influenced by photography (both still and film). Think light probes, camera lenses, etc. Much of the post-processing you see in a scene is also derived by what a camera would see, from focusing, to blur, to HDR now, and of course the infamous lens flare! So I think a physically based renderer which uses the human eye as its camera would be interesting to see more of.
    - formula1 10y ago
      I appreciate you probably put a lot of effort out but your post largely sums up to "They are wrong, I've done extra curricular research to prove it, I have no alternative" While noble, I cant help but wanting to understand what alternatives you were leading yourself done. As an example, light can arguably simplified to intersecting cylinders and spheres that bounce off surfaces to create new 3d shapes. Each shape also would have an origin 2d shape based upon whats reflecting it. an "eye" reads shape intersections with self and also can filter those intersections in respect to origin shape. After each bounce, the new shape takes form as the bouncing lights color multiplied by the color of the bounced object In low light situations, subtle luminosity differences can be enhanced. What I did was offer an example. Perhaps youll one day be successful but I got the impression you are some kind of renegade with a mission. While I can certainly relate to that, I view science and building the future quite far from renegade status. And in the mean time, you gave me a sob story with no algorithms/solutions except for "take real pictures and compare them". As a lazy programmer, walking outside and discoveringvthe world doesnt interest me too much.
- dekhn 10y ago
  There are scenes in Pixar's movies, rendered entirely by Renderman, that are effectively indistinguishable from a high resolution recording of a real instance of the subject. That really was all about algorithms and sampling.
privong 10y ago
> Can somebody who knows about this tell me how much of the novelty in this is CS and how much physics? It's a little bit of both. Over the past 5–8 years, some new numerical techniques ("moving mesh") have been developed (or at least first applied in astronomy) for hydrodynamic simulations[0,1]. These new techniques do a much better job of handling fluid instabilities, which can be important for properly simulating the gas in galaxies (which can have a huge range in densities and temperatures). Also, people have developed what they call "cosmological zoom" simulations. In these simulations, you run a low-resolution (sometimes dark-matter only) simulation of a fairly large volume of the universe. Then you identify a particular dark matter halo at the end, that you think matches the type of galaxy you're interested in (say, a Milky Way type galaxy). Then you go back to the initial conditions, re-initialize it at very high resolution and include "normal matter" (baryons) and re-simulate the whole thing to study the formation of that particular dark matter halo and its associated galaxy. Finally, the general increase in computing speed has allowed people to add in more physics to the simulations and run higher-resolution simulations. In this case, it is the addition of a physically-motivated model for energy injection into the gas in galaxies, from the ongoing star formation. This means energy from supernovae explosions and associated shocks, stellar winds, and momentum from scattering/absorption of photons by dust grains[2,3]. This more realistic set of models manages to reproduce the overall efficiency of star formation (what fraction of the gas gets turned into stars over some period of time) _and_ create a realistic temperature-density structure for the gas. Now that people seem confident that these models can reproduce the overall star formation properties in galaxies, they are starting to see what this type of star formation / feedback model implies for the properties of galaxies. Of course, there's still lots of work to be done in making detailed tests of these models. While they reproduce the overall efficiency of star formation, we aren't yet sure if they can reproduce the shapes and sizes of galaxies. But people are working on that and we should know before too long... [0] https://arxiv.org/abs/0901.4107 https://arxiv.org/abs/0901.4107 [1] https://arxiv.org/abs/1409.7395 https://arxiv.org/abs/1409.7395 [2] https://arxiv.org/abs/1311.2073 https://arxiv.org/abs/1311.2073 [3] https://arxiv.org/abs/1307.5639 https://arxiv.org/abs/1307.5639
Zardoz84 10y ago
And if these image comes from the simulation... Where is the stars bar around the core ??
adekok 10y ago
> Can somebody who knows about this tell me how much of the novelty in this is CS and how much physics? It's not so much a novelty as it is pushing the envelope. The main problem with these kind of simulations is that they're O(N^2). You can't get away with modelling each star, you have to model the interaction of each star with every other star. And then add in interstellar dust, which has non-neglible mass. And then run the simulation for 8 billion years. There are naturally some optimizations and short-cuts to be made. :)
- privong 10y ago
  > The main problem with these kind of simulations is that they're O(N^2). You can't get away with modelling each star, you have to model the interaction of each star with every other star. Summary of what's below: The type of simulations described in the article do not scale like O(N^2) but more like O(N log N). A small correction here.. The gravitational force calculation for N-body simulations are only O(N^2) if they're done using direct summation and force calculation. That's only necessary if you care about the motions of individual stars. The largest simulations I'm aware of that do that use a few 10s or 100s of millions of stars. They're typically used for studying individual star clusters, the regions around supermassive black holes, and perhaps dwarf galaxies. But the simulation in this article/press release has a few 10s of billions of particles, I believe. And those particles are all much more massive than individual stars. So while there are "star particles" in the simulations, it's more accurate to think of them as a set of particles which constitute a monte carlo sampling of the stellar mass distribution. One aspect of that is you now no longer care about the detailed trajectories of individual particles[0], instead caring about the overall representation of the mass distribution. So you don't need as high of accuracy in the force calculation. Going back to the 1980's, people have been using numerical tricks to reduce the scaling of the simulations with the particle number. This includes algorithms such as treecodes, particle-mesh codes, etc., which reduce the scaling to O(N log N) or even O(N). These are sufficient to run simulations of describing the overall gravitational field. The scaling for hydro codes is probably more complicated, particularly for simulations which use a moving mesh. There's a nice summary of numerical techniques for this type of work here: http://arxiv.org/abs/1412.5187 http://arxiv.org/abs/1412.5187 [0] One can determine if interactions between individual stars is important to a system's evolution by comparing the 2-body relaxation time to the age of the system. This is discussed on pages 8–9 of the preprint linked above.
yobaby1234 10y ago
Well, if you look at it, its physics all the way .. After all, CS is nothing but simple mathematics for dummies who can't but sit in front of a computer and blow themselves for being the masters of the world .!!

sillysaurus3 10y ago

For example in graphics all the physics of light is well understood - it's just infeasible to compute - so all advances are in CS and how to sample better. Nitpick: The physics of light is understood. Human vision is also understood. But critically, human vision is completely overlooked in most graphical applications. It's not accurate to say that "the path toward making a computer-generated video indistinguishable from a camcorder recording is just a matter of sampling better." There will need to be a fundamental shift in the industry's thinking before realism will be achieved. It's a fascinating topic that occupied many years of my life, though this is a bit off-topic.

Le_SDT 10y ago
I'm curious. What do you mean by "Human vision is completely overlooked"? Are you talking about color perception? 3d rendering? Thanks :)
- dharma1 10y ago
  Maybe foveation? dynamic range perception?
- sillysaurus3 10y ago
  There's pretty much no way to avoid writing a large comment in response to this. So, with apologies... It was my focus for a long time to achieve perfectly photorealistic rendering. I come from a gamedev background, and I've been fascinated since around age ten about how to get a computer to paint pictures. By 17 I was writing game engines mostly equivalent to quake, and used this portfolio to get into the industry. The next three years were spent getting as close as possible to the bleeding edge of real-time graphics development. I remember having long debates with a colleague about what high-dynamic range lighting "meant." "If we spin around in our office chair, our brains do not suddenly change the overall brightness of this office. Why should we be programming games to do this? Why is everyone doing things that way?" My concerns were of a more fundamental nature as well. What is a diffuse texture? A diffuse texture is a well-understood concept in realtime graphics. Anyone with a basic knowledge of shaders should immediately recognize: color = lighting * diffuse + ambient Trouble is, it doesn't correspond to reality even slightly. It's not even roughly close. It happens to look good to humans, and that's why we use it. But the trouble was, the further I tried to probe the mystery of realism in computer graphics, the more I ran against this phenomenon of "We use X technique because it looks good." So I turned to research papers. Books. The medical field. Everywhere that was remotely related to possible breakthroughs in photorealistic rendering. Research papers are excellent for assembling techniques, but not results. The books in human vision and color science were more promising, yet most of the industry seemed (and still seems) to pay little attention to them. Compare a book about color perception to, say, http://www.pbrt.org/ http://www.pbrt.org/ and you'll see a stark difference. Flip through the table of contents and you get transformations, shapes, primitives, color and radiometery, sampling, reflection, materials, texture, scattering, light sources, monte carlo integration... And for what? We know that these techniques simply do not produce computer-generated videos that a human will identify as a real-life image. It's not for lack of processing power. There is a disconnect between the old rules and those that will ultimately result in real-time realism, and you won't find it in that table of contents. Now, the trouble with writing all of this is that if I knew how to do it, I'd have done it already. It's a life-long search, and it's not so easy to refute an entire industry without being (rightly) dismissed. But if you wish to know what I suspect are the ways forward, it's this: Get a camera. Take photos. Compare these photos to the results of the algorithms you write. Iterate on your algorithms until they are producing results that match something that already captures nature, not our beliefs about how we ought to be able to capture nature. "Just throw in physical models and presto!" has not thus far been true. You'll notice, for example, that computer graphics in videogames have plateaued. They get more impressive with each generation, but that impressiveness does not get them progressively closer to looking real. Nor should it. A computer game tells a story. The closer it looks to real life, the more restricted the artists are, along with the rest of the design of the game. So we turn to the movie industry for hope. But it's restricted in exactly the same way. The research papers are all along the lines of new techniques to try, or studies of existing pipelines and how to deal with their complexities. It's not fundamental research. As someone who has spent his life in pursuit of realism in computer-generated video, my recommendation is this: Read DaVinci's journal. Pay attention to what each page is saying. He had to discover from first principles what makes a painting look real, and why. You'll notice that he spends most of his time talking about human vision and our perceptions of color. If someone is going to make this development happen, it's not going to come from the game industry, and it won't come from the movie industry. That leaves you. Hopefully this will encourage some of you to pursue this. Once you accept that most of the computer graphics industry isn't actually focused on achieving realism, you'll start to develop your own techniques. My hope is that this will eventually lead to a breakthrough.
  - JabavuAdams 10y ago
    > And for what? We know that these techniques simply do not produce computer-generated videos that a human will identify as a real-life image. This is actually testable. Some archviz images are indistinguishable from reality for non-graphics experts. > Get a camera. Take photos. Compare these photos to the results of the algorithms you write. Iterate on your algorithms until they are producing results that match something that already captures nature, not our beliefs about how we ought to be able to capture nature. Yes, very important. Art teachers try to reinforce this by saying "seek reference". It's true for graphics as well, and it's possible to be more empirical.
    - sillysaurus3 10y ago
      Some archviz images are indistinguishable from reality for non-graphics experts. Photos, yes. Not video. It's an important distinction because the path to photorealistic video won't pass through impressive-looking still frames. Visual processing of movement is quite different.
  - adrianm 10y ago
    I thought you were going to go with a different angle on this. If all you care about is photorealism, well, we'll have to agree to disagree that the current state of the art is incapable of that (given enough time). What I thought your original point was is that 3D rendering ONLY cares about photorealism. Physically based renderers have been greatly influenced by photography (both still and film). Think light probes, camera lenses, etc. Much of the post-processing you see in a scene is also derived by what a camera would see, from focusing, to blur, to HDR now, and of course the infamous lens flare! So I think a physically based renderer which uses the human eye as its camera would be interesting to see more of.
  - formula1 10y ago
    I appreciate you probably put a lot of effort out but your post largely sums up to "They are wrong, I've done extra curricular research to prove it, I have no alternative" While noble, I cant help but wanting to understand what alternatives you were leading yourself done. As an example, light can arguably simplified to intersecting cylinders and spheres that bounce off surfaces to create new 3d shapes. Each shape also would have an origin 2d shape based upon whats reflecting it. an "eye" reads shape intersections with self and also can filter those intersections in respect to origin shape. After each bounce, the new shape takes form as the bouncing lights color multiplied by the color of the bounced object In low light situations, subtle luminosity differences can be enhanced. What I did was offer an example. Perhaps youll one day be successful but I got the impression you are some kind of renegade with a mission. While I can certainly relate to that, I view science and building the future quite far from renegade status. And in the mean time, you gave me a sob story with no algorithms/solutions except for "take real pictures and compare them". As a lazy programmer, walking outside and discoveringvthe world doesnt interest me too much.
dekhn 10y ago
There are scenes in Pixar's movies, rendered entirely by Renderman, that are effectively indistinguishable from a high resolution recording of a real instance of the subject. That really was all about algorithms and sampling.

privong 10y ago

> Can somebody who knows about this tell me how much of the novelty in this is CS and how much physics? It's a little bit of both. Over the past 5–8 years, some new numerical techniques ("moving mesh") have been developed (or at least first applied in astronomy) for hydrodynamic simulations[0,1]. These new techniques do a much better job of handling fluid instabilities, which can be important for properly simulating the gas in galaxies (which can have a huge range in densities and temperatures). Also, people have developed what they call "cosmological zoom" simulations. In these simulations, you run a low-resolution (sometimes dark-matter only) simulation of a fairly large volume of the universe. Then you identify a particular dark matter halo at the end, that you think matches the type of galaxy you're interested in (say, a Milky Way type galaxy). Then you go back to the initial conditions, re-initialize it at very high resolution and include "normal matter" (baryons) and re-simulate the whole thing to study the formation of that particular dark matter halo and its associated galaxy. Finally, the general increase in computing speed has allowed people to add in more physics to the simulations and run higher-resolution simulations. In this case, it is the addition of a physically-motivated model for energy injection into the gas in galaxies, from the ongoing star formation. This means energy from supernovae explosions and associated shocks, stellar winds, and momentum from scattering/absorption of photons by dust grains[2,3]. This more realistic set of models manages to reproduce the overall efficiency of star formation (what fraction of the gas gets turned into stars over some period of time) _and_ create a realistic temperature-density structure for the gas. Now that people seem confident that these models can reproduce the overall star formation properties in galaxies, they are starting to see what this type of star formation / feedback model implies for the properties of galaxies. Of course, there's still lots of work to be done in making detailed tests of these models. While they reproduce the overall efficiency of star formation, we aren't yet sure if they can reproduce the shapes and sizes of galaxies. But people are working on that and we should know before too long... [0] https://arxiv.org/abs/0901.4107 https://arxiv.org/abs/0901.4107 [1] https://arxiv.org/abs/1409.7395 https://arxiv.org/abs/1409.7395 [2] https://arxiv.org/abs/1311.2073 https://arxiv.org/abs/1311.2073 [3] https://arxiv.org/abs/1307.5639 https://arxiv.org/abs/1307.5639

Zardoz84 10y ago

And if these image comes from the simulation... Where is the stars bar around the core ??

adekok 10y ago

> Can somebody who knows about this tell me how much of the novelty in this is CS and how much physics? It's not so much a novelty as it is pushing the envelope. The main problem with these kind of simulations is that they're O(N^2). You can't get away with modelling each star, you have to model the interaction of each star with every other star. And then add in interstellar dust, which has non-neglible mass. And then run the simulation for 8 billion years. There are naturally some optimizations and short-cuts to be made. :)

privong 10y ago
> The main problem with these kind of simulations is that they're O(N^2). You can't get away with modelling each star, you have to model the interaction of each star with every other star. Summary of what's below: The type of simulations described in the article do not scale like O(N^2) but more like O(N log N). A small correction here.. The gravitational force calculation for N-body simulations are only O(N^2) if they're done using direct summation and force calculation. That's only necessary if you care about the motions of individual stars. The largest simulations I'm aware of that do that use a few 10s or 100s of millions of stars. They're typically used for studying individual star clusters, the regions around supermassive black holes, and perhaps dwarf galaxies. But the simulation in this article/press release has a few 10s of billions of particles, I believe. And those particles are all much more massive than individual stars. So while there are "star particles" in the simulations, it's more accurate to think of them as a set of particles which constitute a monte carlo sampling of the stellar mass distribution. One aspect of that is you now no longer care about the detailed trajectories of individual particles[0], instead caring about the overall representation of the mass distribution. So you don't need as high of accuracy in the force calculation. Going back to the 1980's, people have been using numerical tricks to reduce the scaling of the simulations with the particle number. This includes algorithms such as treecodes, particle-mesh codes, etc., which reduce the scaling to O(N log N) or even O(N). These are sufficient to run simulations of describing the overall gravitational field. The scaling for hydro codes is probably more complicated, particularly for simulations which use a moving mesh. There's a nice summary of numerical techniques for this type of work here: http://arxiv.org/abs/1412.5187 http://arxiv.org/abs/1412.5187 [0] One can determine if interactions between individual stars is important to a system's evolution by comparing the 2-body relaxation time to the age of the system. This is discussed on pages 8–9 of the preprint linked above.

yobaby1234 10y ago

Well, if you look at it, its physics all the way .. After all, CS is nothing but simple mathematics for dummies who can't but sit in front of a computer and blow themselves for being the masters of the world .!!