3 ms·

I think it's fascinating, and at least to me completely unexpected, how good these image generation models are. What happens when we have a 42gb model, or a 4.2

I think it's fascinating, and at least to me completely unexpected, how good these image generation models are. What happens when we have a 42gb model, or a 4.2tb model? For comparison, the human brain is estimated to have a (equivalent to a computer) capacity of about 2.5 petabytes [0]. I think I read in the past that the human mind holds memories in a picture like way, I wander if that's why these image based models are so incredible when compared to the text models. Maybe we are in a new "Moore's Law" like period where the complexity and size of these models is going to double something like every 18 months. It's going to be fascinating what's possible in a few short years time, I fully expect to be continually surprised. I'm looking forward to seeing a video model trained on 10 second clips, someone somewhere is working on it. 0: https://www.scientificamerican.com/article/what-is-the-memory-capacity/ https://www.scientificamerican.com/article/what-is-the-memor...

dougmwne 4y ago
There is evidence that the performance of the models scale linearly with size, so moores law scaling is likely to get us some “free” improvement even if no one ever invents a better ML technique.
- happycube 4y ago
  I forget where I saw it, but with a 200B parameter model generated text actually makes sense.
  - barrkel 4y ago
    Google Parti, kangaroo holding a sign. 20B not 200B. https://parti.research.google/ https://parti.research.google/
    - dougmwne 4y ago
      Wow, somehow I missed this one with all the whirlwind of image models recently. It’s very illuminating how the capability keeps scaling in their examples.
- salawat 4y ago
  You realize Moore's law is about kaput right? We're running up against fundamental physical limits at this point.
  - dougmwne 4y ago
    I realize we are not done with it yet. There are new process node launches planned for the next few years and each processor generation continues to improve density, power consumption and price per transistor. I’ll hold off declaring it dead till it is well and truly dead. And even then we could expect cost improvements as the great wheel of investment into the next node would no longer need to turn and the last node would become a final standard. As to physical limits, there are plenty of weird quantum particle effects to explore so that seems overstated. We are still just flipping on and off electromagnetic charge. Haven’t even gotten to the quarks yet!
    - trention 4y ago
      >I’ll hold off declaring it dead till it is well and truly dead. The classical Moore's law formulation has been dead for 15 years already. What we have now is whataboutism about why it still holds.
      - dougmwne 4y ago
        You can draw a straight line right through this log scale plot that goes to 2020. Not sure what definition of Moores Law you are using, but it doesn’t seem to match the one on Wikipedia. https://en.m.wikipedia.org/wiki/Moore's_law https://en.m.wikipedia.org/wiki/Moore's_law
        donkarma 4y ago
        well that is transistor count, not transitor per square centimeter which can't be measured because it is variable. the top chips are simply bigger
        4y ago
- igorkraw 4y ago
  IIRC it scales logarithmically, which is the wrong side of the logarithm to be on. I might have missed some new compute-data ratio breakthrough though
- XCSme 4y ago
  "Performance" is hard to define in cases like this, I think, what does an image that's 10x better mean?
  - dougmwne 4y ago
    If you read through the research papers, a big section always deals with benchmarks. That’s because the output needs to be quantified in some way in order to improve the models. Several benchmarks have been proposed for text to image models.
    - XCSme 4y ago
      That makes sense, but that would imply that there's a limit, right? Once the image is pixel-perfect and outputs the optimal image, what does increasing the model size do? Who and how can decide: "yes, this is more Picasso looking than that one", or "this one indeed looks more energetic", or "this image does make me sadder than this one". How do you benchmark this?
      - XCSme 4y ago
        Also, after a point the differences will be more given by the specific individual that views the image, not by what the AI can generate, so the AI would have to optimize it's output per individual and would need to have a deep understanding of them.
      - dougmwne 4y ago
        Yes you are on the right track. Once you get really close to a perfect score on your benchmark you can no longer improve so you need to develop a better benchmark with more headroom. And you have the right idea of how you go about benchmarking subjective quality. A bunch of humans produce output-scoring pairings and the model is judged against that. To train an AI you need a very measurable goal and in this case the measure is “humans like it.” If you are noticing that this seems to fundamentally limit model performance on certain tasks to aggregate human capability, you are noticing correctly. To give you some idea of what these benchmarks look like, here’s the prompt list from DrawBench which Google created as part of training their Imagen model. https://docs.google.com/spreadsheets/u/0/d/1y7nAbmR4FREi6npB1u-Bo3GFdwdOPYJc617rBOxIRHY/htmlview#gid=0 https://docs.google.com/spreadsheets/u/0/d/1y7nAbmR4FREi6npB...
visarga 4y ago
The size of neural nets grew 3000x in 10 years - from 60M params AlexNet to 175B params GPT-3. Thats about 2.23x per year. Moore's law was a doubling every two years, that comes up at 2^(10/2)=32x. Model size scaled 100x faster than compute over a decade. We are paying for this difference by using more energy and hardware, but it's already too expensive to train except for a few labs, and deployment is restricted. Can't even load GPT-3 on most computers. Stable Diffusion is an exception, they did a good job and were lucky the model can be so small.
- hydrolox 4y ago
JackFr 4y ago
> I think it's fascinating, and at least to me completely unexpected, how good these image generation models are. What happens when we have a 42gb model, or a 4.2tb model? First 'how good' is an ill-defined metric -- that it seems in this case is a measurement of how much surprise and wonder they generate in the audience. Second, it might just be that the real wonder of the models is the their compression -- that is there is a space of mappings of line drawings and simple descriptions into art and this technique was able to lossy compress that space down to 4.2G. If you only compress it down to 42G, you'll be looking at the JPEG that's 90% compressed instead of 99%. Yeah it will be better but incrementally, not necessarily "Wow!" better. Honestly it's not obvious that it will be better at all. > For comparison, the human brain is estimated to have a (equivalent to a computer) capacity of about 2.5 petabytes [0]. That's a terrible and basically non-sensical comparison to make.

dougmwne 4y ago

There is evidence that the performance of the models scale linearly with size, so moores law scaling is likely to get us some “free” improvement even if no one ever invents a better ML technique.

happycube 4y ago
I forget where I saw it, but with a 200B parameter model generated text actually makes sense.
- barrkel 4y ago
  Google Parti, kangaroo holding a sign. 20B not 200B. https://parti.research.google/ https://parti.research.google/
  - dougmwne 4y ago
    Wow, somehow I missed this one with all the whirlwind of image models recently. It’s very illuminating how the capability keeps scaling in their examples.
salawat 4y ago
You realize Moore's law is about kaput right? We're running up against fundamental physical limits at this point.
- dougmwne 4y ago
  I realize we are not done with it yet. There are new process node launches planned for the next few years and each processor generation continues to improve density, power consumption and price per transistor. I’ll hold off declaring it dead till it is well and truly dead. And even then we could expect cost improvements as the great wheel of investment into the next node would no longer need to turn and the last node would become a final standard. As to physical limits, there are plenty of weird quantum particle effects to explore so that seems overstated. We are still just flipping on and off electromagnetic charge. Haven’t even gotten to the quarks yet!
  - trention 4y ago
    >I’ll hold off declaring it dead till it is well and truly dead. The classical Moore's law formulation has been dead for 15 years already. What we have now is whataboutism about why it still holds.
    - dougmwne 4y ago
      You can draw a straight line right through this log scale plot that goes to 2020. Not sure what definition of Moores Law you are using, but it doesn’t seem to match the one on Wikipedia. https://en.m.wikipedia.org/wiki/Moore's_law https://en.m.wikipedia.org/wiki/Moore's_law
      - donkarma 4y ago
        well that is transistor count, not transitor per square centimeter which can't be measured because it is variable. the top chips are simply bigger
        4y ago
igorkraw 4y ago
IIRC it scales logarithmically, which is the wrong side of the logarithm to be on. I might have missed some new compute-data ratio breakthrough though
XCSme 4y ago
"Performance" is hard to define in cases like this, I think, what does an image that's 10x better mean?
- dougmwne 4y ago
  If you read through the research papers, a big section always deals with benchmarks. That’s because the output needs to be quantified in some way in order to improve the models. Several benchmarks have been proposed for text to image models.
  - XCSme 4y ago
    That makes sense, but that would imply that there's a limit, right? Once the image is pixel-perfect and outputs the optimal image, what does increasing the model size do? Who and how can decide: "yes, this is more Picasso looking than that one", or "this one indeed looks more energetic", or "this image does make me sadder than this one". How do you benchmark this?
    - XCSme 4y ago
      Also, after a point the differences will be more given by the specific individual that views the image, not by what the AI can generate, so the AI would have to optimize it's output per individual and would need to have a deep understanding of them.
    - dougmwne 4y ago
      Yes you are on the right track. Once you get really close to a perfect score on your benchmark you can no longer improve so you need to develop a better benchmark with more headroom. And you have the right idea of how you go about benchmarking subjective quality. A bunch of humans produce output-scoring pairings and the model is judged against that. To train an AI you need a very measurable goal and in this case the measure is “humans like it.” If you are noticing that this seems to fundamentally limit model performance on certain tasks to aggregate human capability, you are noticing correctly. To give you some idea of what these benchmarks look like, here’s the prompt list from DrawBench which Google created as part of training their Imagen model. https://docs.google.com/spreadsheets/u/0/d/1y7nAbmR4FREi6npB1u-Bo3GFdwdOPYJc617rBOxIRHY/htmlview#gid=0 https://docs.google.com/spreadsheets/u/0/d/1y7nAbmR4FREi6npB...

visarga 4y ago

The size of neural nets grew 3000x in 10 years - from 60M params AlexNet to 175B params GPT-3. Thats about 2.23x per year. Moore's law was a doubling every two years, that comes up at 2^(10/2)=32x. Model size scaled 100x faster than compute over a decade. We are paying for this difference by using more energy and hardware, but it's already too expensive to train except for a few labs, and deployment is restricted. Can't even load GPT-3 on most computers. Stable Diffusion is an exception, they did a good job and were lucky the model can be so small.

hydrolox 4y ago

JackFr 4y ago

> I think it's fascinating, and at least to me completely unexpected, how good these image generation models are. What happens when we have a 42gb model, or a 4.2tb model? First 'how good' is an ill-defined metric -- that it seems in this case is a measurement of how much surprise and wonder they generate in the audience. Second, it might just be that the real wonder of the models is the their compression -- that is there is a space of mappings of line drawings and simple descriptions into art and this technique was able to lossy compress that space down to 4.2G. If you only compress it down to 42G, you'll be looking at the JPEG that's 90% compressed instead of 99%. Yeah it will be better but incrementally, not necessarily "Wow!" better. Honestly it's not obvious that it will be better at all. > For comparison, the human brain is estimated to have a (equivalent to a computer) capacity of about 2.5 petabytes [0]. That's a terrible and basically non-sensical comparison to make.