4 ms·

> This is using gradient descent and “differentiable rendering” and all the other things that are way over my head. then.. > And finally, they have resisted t

> This is using gradient descent and “differentiable rendering” and all the other things that are way over my head. then.. > And finally, they have resisted the temptation to do “neural” anything ;) So, they are doing something similar to NeRF, in a sense, but using a different basis function, and a slightly different target. The whole "neural" or not "neural" is just about what you are optimizing, but it's not that different, conceptually -- they are optimizing a data-driven approximator of a 3d scene-related variable based on images. The big difference of course, based on reading this, is that NeRF models the whole light transmission function ("radiance field") whereas this seems to model only the boundary conditions (what the light "hits"). So given the different modelling target, then yes, a different, and perhaps simpler, representation basis definitely seems warranted, and is shown here to give good results. But the comment, "they avoided neural anything" feels a bit smug, as if "avoiding the hype" is a laudable goal on its own merits. As if people are using neural networks for no reason but because they're cool, and not because they're actually an appropriate solution that are shown to work really well in practice. They surely are cool, don't get me wrong, but the hype is so often justified, because neural networks are really good function approximators. They are also not just one thing -- the choice and breadth of architectures that we happen to call "neural networks" is huge, so explicitly avoiding this huge potential solution space for some reason without good reason feels pretty specious; heck you could even consider having a Gaussian "output layer" to a neural network, which is called an RBF, being something like this "splat" approach. (This is in response to the blog post, not the paper -- I'm sure the paper has plenty of justification for modeling choices.) Having said that, of course it's interesting to also explore simpler, perhaps more efficient approaches, but I don't see "resisting using neural anything" as a good thing, just because they are popular. It ignores that they are popular for a reason. (nb: This assessment is just based on the blog post and giving my impression from the tone of the introduction, I haven't read the paper yet, so might not be accurate with respect to the paper contents. I just found this off-hand comment in the post worth reflecting on.)

graffix 3y ago
> I don't see "resisting using neural anything" as a good thing, just because they are popular I read Aras' quip as more narrowly technical. OFC it's ambiguous and you'd have to ask the man himself. The gist of NeRF is to obtain an NN representation of a 5D light field (3D position + 2D direction) from samples (photographs) of the real-world light field. Alarm bells ring already- 5 dimensions isn't that many! Considering NeRF has always used a low-rank spherical harmonic representation of the directional domain, it's even more like 3D-and-change. To reconstruct a function of such low dimensionality, why choose an NN? Then at inference time, for each pixel, you have sample the NN repeatedly over the view ray. This part is exceedingly silly, as compact representations of light fields are a solved bread-and-butter problem in graphics. Later on Plenoxels explicitly took the "Ne" out of "NeRF", giving far higher training and inference performance (also mentioned ITT). To be fair, and later still, Nvidia somewhat redeemed NNs here with Instant NeRF: https://nvlabs.github.io/instant-ngp/assets/mueller2022instant.pdf https://nvlabs.github.io/instant-ngp/assets/mueller2022insta... ...where the twist was to interpolate fancy input emeddings, which are run through a tiny NN. That tininess is important, as the need to fetch NN weights from VRAM would kick NNs right off the Pareto frontier. Zooming out, NNs have only seen wide adoption in graphics engineering for reconstruction from sparse data (inc. denoising). Makes sense, as that's a high-dimensional problem. Still, beware that the NN solutions rarely blow handmade algorithms out of the water. I also think using tiny NNs for compression- closely related to reconstruction- has a future too. Beyond that, if NNs were to set the graphics world ablaze, it would've happened by now. Lots of graphics engineering is just approximating functions, so it's natural NNs have some place here. However, our functions tend to be more understandable, tractable, malleable. It's not an application domain where it's virtually impossible to write an algorithmic solution by hand (let alone one that performs well), like natural language understanding.
sorenjan 3y ago
One big difference between NeRFs and this technique is that a NeRF is a jumbled mess of interconnected nodes, while this is a collection of values that lives in 3D space. It's much easier to reason about and could probably be integrated fairly easily in other 3D software like editors and renderers, and probably even animated. I agree that neural nets are cool and useful, but the whole black box function approximator that represents a whole scene at once thing doesn't feel scalable or elegant.
- nbardy 3y ago
  Why doesn’t it feel scalable or elegant. It’s just a function call and you can always transform the NeRF into a new representation if you want to use traditional 3d software.
  - sorenjan 3y ago
    How do you edit a NeRF? Let's say I want to change material of an object in the scene, or rotate part of it, or compose several into a new scene, etc. Or animate it. It's not really a good representation of a 3D scene IMO, unless all you care about is novel view synthesis, which is what it was designed to do. Transforming it to a mesh or SDF is cheating, then it's no longer a NeRF and of course the same cons doesn't apply anymore.
  - Vt71fcAqt7 3y ago
    This is still an active area of research. Seems like marching cubes alone are not enough IIRC. Do you have any links I can read for NeRf to USD for example? Saw something pretty good from nvidea a while ago but still not great — still a lot of loss.
- geon 3y ago
  I don't think Nerf as it is now needs to be scalable or elegant. It is research into how far you can neural networks. It has merit in the research itself.
AbrahamParangi 3y ago
Thank you for saying this. I think once things enter the culture war, normal and sensible people will get mugged by stupid, unexamined ideas so long as they’re on the right side of the culture war. So many conversations boil down to AI=BAD or CRYPTO=BAD, which is a shame not because the opposition is wrong - that’s your opinion who am I to disagree - but because once people have their conclusion they just stop thinking and once you stop thinking you’ve got no way of noticing if your thoughts on a matter are totally stupid.
- pierat 3y ago
  One thing I will say, is that AI = means you don't understand how to quantify the problem AI is a great hack to quickly and unknowingly solve classification problems and other similar things. But you trade massive datasets for explainability. And that's a pretty bitter pill to swallow.
  - AbrahamParangi 3y ago
    That it works without you understanding it is a failure of your understanding not of the approach.
    - pierat 3y ago
      Oh please. Dumb word games isn't going to make the "is this a bird" AI approach explainable. https://xkcd.com/1425/ https://xkcd.com/1425/ Even the "simple" AI's with only a few TBs of data for training still have low/no understanding what's going on to identify the features and how they relate. Sure, we can tell this neuron identifies this squiggle, but what does it mean? No fucking clue. It also shows us collectively that most of these "hard" problems we have no clue how to understand them. And although AI allows us to "do it", it's a bad hack for actually understanding, optimizing, and doing. > That it works without you understanding it is a failure of your understanding not of the approach. Cool. Then point me towards the tools to explain how weights in a moderate-ly sized NN are created with training data, and associated with each other and how they find the features in question. Oh yeah. Those tools don't exist.
oslem 3y ago
I agree that ostracizing researchers using neural networks is not a good path for anyone to go down. However, even though NNs are great at modeling problems…we usually have no idea how those learned models actually work. So NNs are great at solving problems in the short term, but aren’t terribly useful for developing long term solutions due to data requirements and regular retraining. In this instance, I’m going to choose to take an optimistic stance and believe the authors were talking a friendly elbow jab rather than fueling a mini culture war. The researchers leveraging DNNs are doing great work, but perhaps could be wrangled in just a bit :)
- shwaj 3y ago
  It is incorrect that we don’t understand how NeRFs work. Unlike language models etc., they are not using deep neural networks; they are using shallow multi-layer perceptrons as function optimizers. The behavior of these is well-understood.
nobbis 3y ago
> The big difference of course, based on reading this, is that NeRF models the whole light transmission function ("radiance field") whereas this seems to model only the boundary conditions (what the light "hits"). The title of the paper is "3D Gaussian Splatting for Real-Time Radiance Field Rendering". Each rendered pixel weights the contribution of unbounded view-dependent Gaussians. So, no, that's not a difference.
- radarsat1 3y ago
  Ah gotcha, thanks for pointing that out.

graffix 3y ago

> I don't see "resisting using neural anything" as a good thing, just because they are popular I read Aras' quip as more narrowly technical. OFC it's ambiguous and you'd have to ask the man himself. The gist of NeRF is to obtain an NN representation of a 5D light field (3D position + 2D direction) from samples (photographs) of the real-world light field. Alarm bells ring already- 5 dimensions isn't that many! Considering NeRF has always used a low-rank spherical harmonic representation of the directional domain, it's even more like 3D-and-change. To reconstruct a function of such low dimensionality, why choose an NN? Then at inference time, for each pixel, you have sample the NN repeatedly over the view ray. This part is exceedingly silly, as compact representations of light fields are a solved bread-and-butter problem in graphics. Later on Plenoxels explicitly took the "Ne" out of "NeRF", giving far higher training and inference performance (also mentioned ITT). To be fair, and later still, Nvidia somewhat redeemed NNs here with Instant NeRF: https://nvlabs.github.io/instant-ngp/assets/mueller2022instant.pdf https://nvlabs.github.io/instant-ngp/assets/mueller2022insta... ...where the twist was to interpolate fancy input emeddings, which are run through a tiny NN. That tininess is important, as the need to fetch NN weights from VRAM would kick NNs right off the Pareto frontier. Zooming out, NNs have only seen wide adoption in graphics engineering for reconstruction from sparse data (inc. denoising). Makes sense, as that's a high-dimensional problem. Still, beware that the NN solutions rarely blow handmade algorithms out of the water. I also think using tiny NNs for compression- closely related to reconstruction- has a future too. Beyond that, if NNs were to set the graphics world ablaze, it would've happened by now. Lots of graphics engineering is just approximating functions, so it's natural NNs have some place here. However, our functions tend to be more understandable, tractable, malleable. It's not an application domain where it's virtually impossible to write an algorithmic solution by hand (let alone one that performs well), like natural language understanding.

sorenjan 3y ago

One big difference between NeRFs and this technique is that a NeRF is a jumbled mess of interconnected nodes, while this is a collection of values that lives in 3D space. It's much easier to reason about and could probably be integrated fairly easily in other 3D software like editors and renderers, and probably even animated. I agree that neural nets are cool and useful, but the whole black box function approximator that represents a whole scene at once thing doesn't feel scalable or elegant.

nbardy 3y ago
Why doesn’t it feel scalable or elegant. It’s just a function call and you can always transform the NeRF into a new representation if you want to use traditional 3d software.
- sorenjan 3y ago
  How do you edit a NeRF? Let's say I want to change material of an object in the scene, or rotate part of it, or compose several into a new scene, etc. Or animate it. It's not really a good representation of a 3D scene IMO, unless all you care about is novel view synthesis, which is what it was designed to do. Transforming it to a mesh or SDF is cheating, then it's no longer a NeRF and of course the same cons doesn't apply anymore.
- Vt71fcAqt7 3y ago
  This is still an active area of research. Seems like marching cubes alone are not enough IIRC. Do you have any links I can read for NeRf to USD for example? Saw something pretty good from nvidea a while ago but still not great — still a lot of loss.
geon 3y ago
I don't think Nerf as it is now needs to be scalable or elegant. It is research into how far you can neural networks. It has merit in the research itself.

AbrahamParangi 3y ago

Thank you for saying this. I think once things enter the culture war, normal and sensible people will get mugged by stupid, unexamined ideas so long as they’re on the right side of the culture war. So many conversations boil down to AI=BAD or CRYPTO=BAD, which is a shame not because the opposition is wrong - that’s your opinion who am I to disagree - but because once people have their conclusion they just stop thinking and once you stop thinking you’ve got no way of noticing if your thoughts on a matter are totally stupid.

pierat 3y ago
One thing I will say, is that AI = means you don't understand how to quantify the problem AI is a great hack to quickly and unknowingly solve classification problems and other similar things. But you trade massive datasets for explainability. And that's a pretty bitter pill to swallow.
- AbrahamParangi 3y ago
  That it works without you understanding it is a failure of your understanding not of the approach.
  - pierat 3y ago
    Oh please. Dumb word games isn't going to make the "is this a bird" AI approach explainable. https://xkcd.com/1425/ https://xkcd.com/1425/ Even the "simple" AI's with only a few TBs of data for training still have low/no understanding what's going on to identify the features and how they relate. Sure, we can tell this neuron identifies this squiggle, but what does it mean? No fucking clue. It also shows us collectively that most of these "hard" problems we have no clue how to understand them. And although AI allows us to "do it", it's a bad hack for actually understanding, optimizing, and doing. > That it works without you understanding it is a failure of your understanding not of the approach. Cool. Then point me towards the tools to explain how weights in a moderate-ly sized NN are created with training data, and associated with each other and how they find the features in question. Oh yeah. Those tools don't exist.

oslem 3y ago

I agree that ostracizing researchers using neural networks is not a good path for anyone to go down. However, even though NNs are great at modeling problems…we usually have no idea how those learned models actually work. So NNs are great at solving problems in the short term, but aren’t terribly useful for developing long term solutions due to data requirements and regular retraining. In this instance, I’m going to choose to take an optimistic stance and believe the authors were talking a friendly elbow jab rather than fueling a mini culture war. The researchers leveraging DNNs are doing great work, but perhaps could be wrangled in just a bit :)

shwaj 3y ago
It is incorrect that we don’t understand how NeRFs work. Unlike language models etc., they are not using deep neural networks; they are using shallow multi-layer perceptrons as function optimizers. The behavior of these is well-understood.

nobbis 3y ago

> The big difference of course, based on reading this, is that NeRF models the whole light transmission function ("radiance field") whereas this seems to model only the boundary conditions (what the light "hits"). The title of the paper is "3D Gaussian Splatting for Real-Time Radiance Field Rendering". Each rendered pixel weights the contribution of unbounded view-dependent Gaussians. So, no, that's not a difference.

radarsat1 3y ago
Ah gotcha, thanks for pointing that out.