4 ms·

NeRF's were inveted in 2020 and started an avalanche of papers (637 citations so far). > Original one: "Nerf, Representing scenes as neural radiance fields for

NeRF's were inveted in 2020 and started an avalanche of papers (637 citations so far). > Original one: "Nerf, Representing scenes as neural radiance fields for view synthesis" > https://scholar.google.com/scholar?cites=9378169911033868166&as_sdt=2005&sciodt=0,5&hl=en https://scholar.google.com/scholar?cites=9378169911033868166... What I find most exciting about it is that a NeRF represents images as neural nets, one neural net for each image (in the OP paper generalised to image + deformations). By evaluating the net at various pixel coordinates it gives the color. Up until now learning to replicate the input exactly was called overfitting and considered a bug, not a feature, but they showed a completely new way to wield neural nets. An interesting detail is that they depend on Fourier encoding for the input coordinates. A variant called SIREN uses `sin` as activation function throughout the net. Maybe neural nets will become the data compressors of tomorrow? Shoot a picture, send a neural net around. Game assets could be NeRFs.

sbierwagen 5y ago
>Maybe neural nets will become the data compressors of tomorrow? Long a topic of research, with many interesting ramifications: https://ai.googleblog.com/2016/09/image-compression-with-neural-networks.html https://ai.googleblog.com/2016/09/image-compression-with-neu... In analogy to standard compression techniques, you can think of the neural net as a very large "dictionary". For text compression, there is of course the famous Hutter Prize, launched in 2006: https://en.wikipedia.org/wiki/Hutter_Prize https://en.wikipedia.org/wiki/Hutter_Prize ("Prediction is the golden key that opens all locks". Compressing each byte of wikipedia text is equivalent to predicting it-- to compactly represent its knowledge is to understand it.)
- vintermann 5y ago
  1. If the first byte is 0, insert the text of Wikipedia, 2. If it isn't, ignore it, and all further bytes are interpreted literally. To avoid these sort of "joke" decompressors, they evaluate compressors on (size of compressed data) + (size of decompressor) in the compression competitions last time I checked. That means we won't get a winner based on GPT3 anytime soon. 350+ GB of weights is a lot to overcome :) Though of course, given enough data to compress, it might well be that full-on neural language models are still worth it.
  - deleted 5y ago
    [deleted]
  - visarga 5y ago
    > That means we won't get a winner based on GPT3 anytime soon. 350+ GB of weights is a lot to overcome :) You're right, and in-there lays the crucial difference between compression and language modelling. Models are concerned with having a good representation for both past and future distributions, while compressors only care about the past. Models support many tasks while compressors are just for input replication.
    - vintermann 5y ago
      No, I don't quite agree. All compressors have a model, at least the general purpose ones. If you're writing self-extracting compression for your 8-bit demo, maybe you only care about the data you have right in front of you... But for anything general purpose, from DEFLATE to paq, there's an implicit, fixed model of future data in there (even if we sometimes prefer to see it as an "adaptive" model). A good illustration of this is that you can quite easily use a general purpose compression algorithm for classification, and get surprisingly good performance.
  - injidup 5y ago
    I don't think it is unreasonable that both sender and receiver have the 350GB NN at each end. If for example we are talking video conferencing and compression of video data then only a small amount of data need to be transmitted real time and at each end a high fidelity image can be reconstructed.
    - vintermann 5y ago
      Maybe one day that will be practical. But right now, you can't even fit that into a top-tier GPU you know...
vintermann 5y ago
> Up until now learning to replicate the input exactly was called overfitting and considered a bug, not a feature, but they showed a completely new way to wield neural nets. Well, for that particular ting there was a predecessor of sort in Deep Image Priors from 2017. https://arxiv.org/abs/1711.10925 https://arxiv.org/abs/1711.10925 That was all about overfitting a neural net on a single image, which they used to get impressive inpainting, noise removal and superresolution results without any training at all (though of course it did not beat state of the art training-based approaches, even then.) I had a lot of fun playing around with it when it came. The idea is dead simple, within reach to implement yourself with no complex mathematical understanding.
orlp 5y ago
> What I find most exciting about it is that a NeRF represents images as neural nets, one neural net for each image (in the OP paper generalised to image + deformations). By evaluating the net at various pixel coordinates it gives the color. I actually had a similar idea and used it to write a lossy image compressor for my bachelor thesis in 2018: https://theses.liacs.nl/pdf/2018-2019-PetersO.pdf https://theses.liacs.nl/pdf/2018-2019-PetersO.pdf

sbierwagen 5y ago

>Maybe neural nets will become the data compressors of tomorrow? Long a topic of research, with many interesting ramifications: https://ai.googleblog.com/2016/09/image-compression-with-neural-networks.html https://ai.googleblog.com/2016/09/image-compression-with-neu... In analogy to standard compression techniques, you can think of the neural net as a very large "dictionary". For text compression, there is of course the famous Hutter Prize, launched in 2006: https://en.wikipedia.org/wiki/Hutter_Prize https://en.wikipedia.org/wiki/Hutter_Prize ("Prediction is the golden key that opens all locks". Compressing each byte of wikipedia text is equivalent to predicting it-- to compactly represent its knowledge is to understand it.)

vintermann 5y ago
1. If the first byte is 0, insert the text of Wikipedia, 2. If it isn't, ignore it, and all further bytes are interpreted literally. To avoid these sort of "joke" decompressors, they evaluate compressors on (size of compressed data) + (size of decompressor) in the compression competitions last time I checked. That means we won't get a winner based on GPT3 anytime soon. 350+ GB of weights is a lot to overcome :) Though of course, given enough data to compress, it might well be that full-on neural language models are still worth it.
- deleted 5y ago
  [deleted]
- visarga 5y ago
  > That means we won't get a winner based on GPT3 anytime soon. 350+ GB of weights is a lot to overcome :) You're right, and in-there lays the crucial difference between compression and language modelling. Models are concerned with having a good representation for both past and future distributions, while compressors only care about the past. Models support many tasks while compressors are just for input replication.
  - vintermann 5y ago
    No, I don't quite agree. All compressors have a model, at least the general purpose ones. If you're writing self-extracting compression for your 8-bit demo, maybe you only care about the data you have right in front of you... But for anything general purpose, from DEFLATE to paq, there's an implicit, fixed model of future data in there (even if we sometimes prefer to see it as an "adaptive" model). A good illustration of this is that you can quite easily use a general purpose compression algorithm for classification, and get surprisingly good performance.
- injidup 5y ago
  I don't think it is unreasonable that both sender and receiver have the 350GB NN at each end. If for example we are talking video conferencing and compression of video data then only a small amount of data need to be transmitted real time and at each end a high fidelity image can be reconstructed.
  - vintermann 5y ago
    Maybe one day that will be practical. But right now, you can't even fit that into a top-tier GPU you know...

vintermann 5y ago

> Up until now learning to replicate the input exactly was called overfitting and considered a bug, not a feature, but they showed a completely new way to wield neural nets. Well, for that particular ting there was a predecessor of sort in Deep Image Priors from 2017. https://arxiv.org/abs/1711.10925 https://arxiv.org/abs/1711.10925 That was all about overfitting a neural net on a single image, which they used to get impressive inpainting, noise removal and superresolution results without any training at all (though of course it did not beat state of the art training-based approaches, even then.) I had a lot of fun playing around with it when it came. The idea is dead simple, within reach to implement yourself with no complex mathematical understanding.

orlp 5y ago

> What I find most exciting about it is that a NeRF represents images as neural nets, one neural net for each image (in the OP paper generalised to image + deformations). By evaluating the net at various pixel coordinates it gives the color. I actually had a similar idea and used it to write a lossy image compressor for my bachelor thesis in 2018: https://theses.liacs.nl/pdf/2018-2019-PetersO.pdf https://theses.liacs.nl/pdf/2018-2019-PetersO.pdf