4 ms·

There is evidence that the performance of the models scale linearly with size, so moores law scaling is likely to get us some “free” improvement even if no one

There is evidence that the performance of the models scale linearly with size, so moores law scaling is likely to get us some “free” improvement even if no one ever invents a better ML technique.

happycube 4y ago
I forget where I saw it, but with a 200B parameter model generated text actually makes sense.
- barrkel 4y ago
  Google Parti, kangaroo holding a sign. 20B not 200B. https://parti.research.google/ https://parti.research.google/
  - dougmwne 4y ago
    Wow, somehow I missed this one with all the whirlwind of image models recently. It’s very illuminating how the capability keeps scaling in their examples.
salawat 4y ago
You realize Moore's law is about kaput right? We're running up against fundamental physical limits at this point.
- dougmwne 4y ago
  I realize we are not done with it yet. There are new process node launches planned for the next few years and each processor generation continues to improve density, power consumption and price per transistor. I’ll hold off declaring it dead till it is well and truly dead. And even then we could expect cost improvements as the great wheel of investment into the next node would no longer need to turn and the last node would become a final standard. As to physical limits, there are plenty of weird quantum particle effects to explore so that seems overstated. We are still just flipping on and off electromagnetic charge. Haven’t even gotten to the quarks yet!
  - trention 4y ago
    >I’ll hold off declaring it dead till it is well and truly dead. The classical Moore's law formulation has been dead for 15 years already. What we have now is whataboutism about why it still holds.
    - dougmwne 4y ago
      You can draw a straight line right through this log scale plot that goes to 2020. Not sure what definition of Moores Law you are using, but it doesn’t seem to match the one on Wikipedia. https://en.m.wikipedia.org/wiki/Moore's_law https://en.m.wikipedia.org/wiki/Moore's_law
      - donkarma 4y ago
        well that is transistor count, not transitor per square centimeter which can't be measured because it is variable. the top chips are simply bigger
        dougmwne 4y ago
        Here’s a chart for density. Still going strong with maybe a bit of drop off. https://www.researchgate.net/figure/Density-of-logic-transistors-solid-line-has-advanced-on-average-by-2-per-generation_fig2_338517514 https://www.researchgate.net/figure/Density-of-logic-transis...
igorkraw 4y ago
IIRC it scales logarithmically, which is the wrong side of the logarithm to be on. I might have missed some new compute-data ratio breakthrough though
XCSme 4y ago
"Performance" is hard to define in cases like this, I think, what does an image that's 10x better mean?
- dougmwne 4y ago
  If you read through the research papers, a big section always deals with benchmarks. That’s because the output needs to be quantified in some way in order to improve the models. Several benchmarks have been proposed for text to image models.
  - XCSme 4y ago
    That makes sense, but that would imply that there's a limit, right? Once the image is pixel-perfect and outputs the optimal image, what does increasing the model size do? Who and how can decide: "yes, this is more Picasso looking than that one", or "this one indeed looks more energetic", or "this image does make me sadder than this one". How do you benchmark this?
    - XCSme 4y ago
      Also, after a point the differences will be more given by the specific individual that views the image, not by what the AI can generate, so the AI would have to optimize it's output per individual and would need to have a deep understanding of them.
    - dougmwne 4y ago
      Yes you are on the right track. Once you get really close to a perfect score on your benchmark you can no longer improve so you need to develop a better benchmark with more headroom. And you have the right idea of how you go about benchmarking subjective quality. A bunch of humans produce output-scoring pairings and the model is judged against that. To train an AI you need a very measurable goal and in this case the measure is “humans like it.” If you are noticing that this seems to fundamentally limit model performance on certain tasks to aggregate human capability, you are noticing correctly. To give you some idea of what these benchmarks look like, here’s the prompt list from DrawBench which Google created as part of training their Imagen model. https://docs.google.com/spreadsheets/u/0/d/1y7nAbmR4FREi6npB1u-Bo3GFdwdOPYJc617rBOxIRHY/htmlview#gid=0 https://docs.google.com/spreadsheets/u/0/d/1y7nAbmR4FREi6npB...

There is evidence that the performance of the models scale linearly with size, so moores law scaling is likely to get us some “free” improvement even if no one ever invents a better ML technique.

happycube 4y ago

I forget where I saw it, but with a 200B parameter model generated text actually makes sense.

barrkel 4y ago
Google Parti, kangaroo holding a sign. 20B not 200B. https://parti.research.google/ https://parti.research.google/
- dougmwne 4y ago
  Wow, somehow I missed this one with all the whirlwind of image models recently. It’s very illuminating how the capability keeps scaling in their examples.

salawat 4y ago

You realize Moore's law is about kaput right? We're running up against fundamental physical limits at this point.

dougmwne 4y ago
I realize we are not done with it yet. There are new process node launches planned for the next few years and each processor generation continues to improve density, power consumption and price per transistor. I’ll hold off declaring it dead till it is well and truly dead. And even then we could expect cost improvements as the great wheel of investment into the next node would no longer need to turn and the last node would become a final standard. As to physical limits, there are plenty of weird quantum particle effects to explore so that seems overstated. We are still just flipping on and off electromagnetic charge. Haven’t even gotten to the quarks yet!
- trention 4y ago
  >I’ll hold off declaring it dead till it is well and truly dead. The classical Moore's law formulation has been dead for 15 years already. What we have now is whataboutism about why it still holds.
  - dougmwne 4y ago
    You can draw a straight line right through this log scale plot that goes to 2020. Not sure what definition of Moores Law you are using, but it doesn’t seem to match the one on Wikipedia. https://en.m.wikipedia.org/wiki/Moore's_law https://en.m.wikipedia.org/wiki/Moore's_law
    - donkarma 4y ago
      well that is transistor count, not transitor per square centimeter which can't be measured because it is variable. the top chips are simply bigger
      - dougmwne 4y ago
        Here’s a chart for density. Still going strong with maybe a bit of drop off. https://www.researchgate.net/figure/Density-of-logic-transistors-solid-line-has-advanced-on-average-by-2-per-generation_fig2_338517514 https://www.researchgate.net/figure/Density-of-logic-transis...

igorkraw 4y ago

IIRC it scales logarithmically, which is the wrong side of the logarithm to be on. I might have missed some new compute-data ratio breakthrough though

XCSme 4y ago

"Performance" is hard to define in cases like this, I think, what does an image that's 10x better mean?

dougmwne 4y ago
If you read through the research papers, a big section always deals with benchmarks. That’s because the output needs to be quantified in some way in order to improve the models. Several benchmarks have been proposed for text to image models.
- XCSme 4y ago
  That makes sense, but that would imply that there's a limit, right? Once the image is pixel-perfect and outputs the optimal image, what does increasing the model size do? Who and how can decide: "yes, this is more Picasso looking than that one", or "this one indeed looks more energetic", or "this image does make me sadder than this one". How do you benchmark this?
  - XCSme 4y ago
    Also, after a point the differences will be more given by the specific individual that views the image, not by what the AI can generate, so the AI would have to optimize it's output per individual and would need to have a deep understanding of them.
  - dougmwne 4y ago
    Yes you are on the right track. Once you get really close to a perfect score on your benchmark you can no longer improve so you need to develop a better benchmark with more headroom. And you have the right idea of how you go about benchmarking subjective quality. A bunch of humans produce output-scoring pairings and the model is judged against that. To train an AI you need a very measurable goal and in this case the measure is “humans like it.” If you are noticing that this seems to fundamentally limit model performance on certain tasks to aggregate human capability, you are noticing correctly. To give you some idea of what these benchmarks look like, here’s the prompt list from DrawBench which Google created as part of training their Imagen model. https://docs.google.com/spreadsheets/u/0/d/1y7nAbmR4FREi6npB1u-Bo3GFdwdOPYJc617rBOxIRHY/htmlview#gid=0 https://docs.google.com/spreadsheets/u/0/d/1y7nAbmR4FREi6npB...