5 ms·

I'm curious why you decide to declare victory after one successful attempt, but try many times for unsuccessful models. Are you trying to measure whether a mode

I'm curious why you decide to declare victory after one successful attempt, but try many times for unsuccessful models. Are you trying to measure whether a model _can_ get it right, or whether it frequently _does_ get it right? I feel like success rate is a better metric here, or at least a fixed number of trials with some success rate threshold to determine model success.

ipnon 1y ago
It is indicative of marginal improvements instead of new breakthroughs. iPhone 1 was a paradigm shift. iPhone 10 was essentially iPhone 9 with tweaks. As an AI optimist I would be disappointed to find we are already seeing diminishing returns on R&D.
- sethaurus 1y ago
  At the risk of unbearable pedantry, there's never been an iPhone 9. (There was never a 2 either; there was kind of a 3, although it was really called the 3G.)
  - deleted 1y ago
    [deleted]
- Uehreka 1y ago
  Not only did the iPhone 9 never exist, the iPhone X was a huge paradigm shift in design and capabilities. That was the phone that introduced edge-to-edge OLED screens to the iPhone line, as well as the IR camera that enabled FaceID and the first generation of Portrait mode. I know it well since it also introduced the ability for developers to build facial motion capture apps that would’ve previously required expensive pro hardware and allowed people like me to build live facial motion capture effects for theatre. Sorry to dunk so hard, but your example of technology stagnating is actually an example of breakthrough technological innovation deep into a product’s lifecycle: the very thing you were trying to say doesn’t happen.
  - vincnetas 1y ago
    technological advancements yes, but did it drastically changed how users (majority) use the iphone. I'd say marginally. Fancy selfie filters, ok i'll give it that. But edge to edge screens, meh, give me back my home button :D
  - ofrzeta 1y ago
    Arguably OLED screens and IR cameras are no paradigm shift. At least nothing comparable to "no smartphone" to iPhone 1.
    - jere 1y ago
      There were smartphones before the iPhone. One could also describe the difference as "just a touchscreen".
      - ofrzeta 1y ago
        The iPhone 1 featured "touch screen, GPS, camera, iPod, and internet access. Its software capabilities were a turning point for the smartphone industry" (random source: https://www.textline.com/blog/smartphone-history https://www.textline.com/blog/smartphone-history). If you want to doubt that it was in fact a not a turning point you'd need to provide very strong arguments.
        hollerith 1y ago
        Also, the touchscreen was the type that unlike all previous touchscreens (except the ones made by a startup that Apple had bought) could detect touches at more than one screen location simultaneously.
        spogbiper 1y ago
        as someone who owned a number of smartphones and PDAs prior to the first iPhone coming out, the real advance was a usable mobile browser. i'd had all the same capabilities with devices for quite some time before the iphone came out, but their browsers were painful to use. the touch interface was also a big advance over previous touch interfaces. in other areas the first iphone was lacking compared to other smartphones.. copy and paste and 3rd party apps were missing for example.
- TeMPOraL 1y ago
  Just moments ago, I managed to turn a photo of a person into a short clip of them dancing, in half-decent quality, fully locally, on a mid-range gaming GPU (RTX 4070 Ti, 12GB VRAM). I almost run out of RAM (32GB), but it worked, worked well, and took only couple of minutes. Half a year ago, that was sort of possible for some genius really bent on making it happen. A year ago, that was unthinkable. Today, it's a matter of drag&dropping a workflow to a fresh ComfyUI install and downloading a couple dozen GB of img2vid models. The returns on R&D are not diminishing, the progress is just not happening everywhere evenly and at the same time.
vunderba 1y ago
It's hard to nail down a good objective metric on something that is always going to be marginally qualitative in nature but it's a good call out - I should probably add a FAQ to the site. To clarify this test is purely a PASS/FAIL - unsuccessful means that the model NEVER managed to generate an image adhering to the prompt. So as an example, Midjourney 7 did not manage to generate the correct vertical stack of translucent cubes ordered by color in 64 gen attempts. It's a little beyond the scope of my site but I do like the idea of maintaining a more granular metric for the models that were successful to see how often they were successful.
- bigmadshoe 1y ago
  Makes sense. It just set off some statistical alarm bells in my head to see a model marked as passing with 1 trial, and some models marked as failing with 5. What if the probability of success is 5% for both models? How confident are we that our grading of the models is correct? It's an interesting problem. Cool site btw! Thanks for sharing.
  - Taek 1y ago
    Its made a little bit better by the fact that there's something like a dozen different prompts. Across all of the prompts each model had a fair number of opportunities to show off.
  - npinsker 1y ago
    The current metric is actually quite strong -- it mirrors the real-world use case of people trying a few times and being satisfied if any of them's what they're looking for. It rewards diversity of responses. Actually, search engines do this this too: Google something with many possible meanings -- like "egg" -- on Google, and you'll get a set of intentionally diversified results. I get Wikipedia; then a restaurant; then YouTube cooking videos; Big Green Egg's homepage; news stories about egg shortages. Each individual link is very unlike the others to maximize the chance that one of them's the one you want.

ipnon 1y ago

It is indicative of marginal improvements instead of new breakthroughs. iPhone 1 was a paradigm shift. iPhone 10 was essentially iPhone 9 with tweaks. As an AI optimist I would be disappointed to find we are already seeing diminishing returns on R&D.

sethaurus 1y ago
At the risk of unbearable pedantry, there's never been an iPhone 9. (There was never a 2 either; there was kind of a 3, although it was really called the 3G.)
- deleted 1y ago
  [deleted]
Uehreka 1y ago
Not only did the iPhone 9 never exist, the iPhone X was a huge paradigm shift in design and capabilities. That was the phone that introduced edge-to-edge OLED screens to the iPhone line, as well as the IR camera that enabled FaceID and the first generation of Portrait mode. I know it well since it also introduced the ability for developers to build facial motion capture apps that would’ve previously required expensive pro hardware and allowed people like me to build live facial motion capture effects for theatre. Sorry to dunk so hard, but your example of technology stagnating is actually an example of breakthrough technological innovation deep into a product’s lifecycle: the very thing you were trying to say doesn’t happen.
- vincnetas 1y ago
  technological advancements yes, but did it drastically changed how users (majority) use the iphone. I'd say marginally. Fancy selfie filters, ok i'll give it that. But edge to edge screens, meh, give me back my home button :D
- ofrzeta 1y ago
  Arguably OLED screens and IR cameras are no paradigm shift. At least nothing comparable to "no smartphone" to iPhone 1.
  - jere 1y ago
    There were smartphones before the iPhone. One could also describe the difference as "just a touchscreen".
    - ofrzeta 1y ago
      The iPhone 1 featured "touch screen, GPS, camera, iPod, and internet access. Its software capabilities were a turning point for the smartphone industry" (random source: https://www.textline.com/blog/smartphone-history https://www.textline.com/blog/smartphone-history). If you want to doubt that it was in fact a not a turning point you'd need to provide very strong arguments.
      - hollerith 1y ago
        Also, the touchscreen was the type that unlike all previous touchscreens (except the ones made by a startup that Apple had bought) could detect touches at more than one screen location simultaneously.
      - spogbiper 1y ago
        as someone who owned a number of smartphones and PDAs prior to the first iPhone coming out, the real advance was a usable mobile browser. i'd had all the same capabilities with devices for quite some time before the iphone came out, but their browsers were painful to use. the touch interface was also a big advance over previous touch interfaces. in other areas the first iphone was lacking compared to other smartphones.. copy and paste and 3rd party apps were missing for example.
TeMPOraL 1y ago
Just moments ago, I managed to turn a photo of a person into a short clip of them dancing, in half-decent quality, fully locally, on a mid-range gaming GPU (RTX 4070 Ti, 12GB VRAM). I almost run out of RAM (32GB), but it worked, worked well, and took only couple of minutes. Half a year ago, that was sort of possible for some genius really bent on making it happen. A year ago, that was unthinkable. Today, it's a matter of drag&dropping a workflow to a fresh ComfyUI install and downloading a couple dozen GB of img2vid models. The returns on R&D are not diminishing, the progress is just not happening everywhere evenly and at the same time.

vunderba 1y ago

It's hard to nail down a good objective metric on something that is always going to be marginally qualitative in nature but it's a good call out - I should probably add a FAQ to the site. To clarify this test is purely a PASS/FAIL - unsuccessful means that the model NEVER managed to generate an image adhering to the prompt. So as an example, Midjourney 7 did not manage to generate the correct vertical stack of translucent cubes ordered by color in 64 gen attempts. It's a little beyond the scope of my site but I do like the idea of maintaining a more granular metric for the models that were successful to see how often they were successful.

bigmadshoe 1y ago
Makes sense. It just set off some statistical alarm bells in my head to see a model marked as passing with 1 trial, and some models marked as failing with 5. What if the probability of success is 5% for both models? How confident are we that our grading of the models is correct? It's an interesting problem. Cool site btw! Thanks for sharing.
- Taek 1y ago
  Its made a little bit better by the fact that there's something like a dozen different prompts. Across all of the prompts each model had a fair number of opportunities to show off.
- npinsker 1y ago
  The current metric is actually quite strong -- it mirrors the real-world use case of people trying a few times and being satisfied if any of them's what they're looking for. It rewards diversity of responses. Actually, search engines do this this too: Google something with many possible meanings -- like "egg" -- on Google, and you'll get a set of intentionally diversified results. I get Wikipedia; then a restaurant; then YouTube cooking videos; Big Green Egg's homepage; news stories about egg shortages. Each individual link is very unlike the others to maximize the chance that one of them's the one you want.