3 ms·

> Just a few years ago it would have been regarded as unbelievable science fiction that a model with such capabilities would soon be available. No, it wouldn't

> Just a few years ago it would have been regarded as unbelievable science fiction that a model with such capabilities would soon be available. No, it wouldn't have – not to people in the know. We just didn't have powerful enough computers back in the 90s. Sure, the techniques we've got now are better, but 90s algorithms (with modern supercomputers) can get you most of the way. Transformers are awesome, but they're not that much of a stretch from 90s technology. GANs are… ridiculously obvious, in hindsight, and people have been doing similar things since the dawn of AI; I imagine the people who came up with the idea were pretty confident of its capabilities even before they tested them. Both these kinds of system – and neural-net-based systems in general – are based around mimicry. Their inability to draw limbs, or to tell the truth, or count, are fundamental to how they function, and iterative improvement isn't going to fix them. Iterative improvement would be going faster, if researchers (outside of OpenAI and similar corporations) thought it was worthwhile to focus on improving these systems specifically. ChatGPT is not where transformers shine. StyleGAN3 is not where GANs shine. Midjourney is not where diffusion models shine. They're really useful lenses for visualising the way the architectures work, so they are useful test-beds for iterative algorithmic improvements…¹ but they aren't all that they're made out to be. ¹: See the 3 in StyleGAN3. Unlike the 4 in GPT-4, it actually means something more than "we made it bigger and changed the training data a bit".

cubefox 4y ago
> No, it wouldn't have – not to people in the know. We just didn't have powerful enough computers back in the 90s. I'm not talking about the 90s. I'm talking about April 29, 2020.
- wizzwizz4 4y ago
  What's special about that day? That's after the algorithms were developed, models and drivers were built, and most of these behaviours were discovered. I've got fairly-photorealistic "AI-generated" photos on my laptop timestamped September 2019, and that was before I started learning how it all worked. If you're talking about popular awareness of GPT-style autocomplete, then I agree. If you're talking about academic awareness of what these things can and can't do, we've had that for a while.
  - cubefox 4y ago
    What photorealistic AI generated image? In September 2019 this must have been a GAN face. I admit those are impressive, but incredibly limited compared to todays text to image models. If you look at an iPhone from 2019, or a car, or a videogame ... they all still look about the same today. Three years ago there was nothing remotely as impressive as modern GPT style or text to image models. Basically nobody predicted what was about to happen. The only exception I know is Scott Alexander [1]. I don't know about any similar predictions from the experts, but I'm happy to be proven wrong. [1] https://slatestarcodex.com/2019/02/19/gpt-2-as-step-toward-general-intelligence/ https://slatestarcodex.com/2019/02/19/gpt-2-as-step-toward-g...
    - wizzwizz4 4y ago
      > In September 2019 this must have been a GAN face. Well, yes,¹ but actually no. StyleGAN1's public release was February 2019, and it's capable of far more than just faces. > Three years ago there was nothing remotely as impressive as modern GPT style I predicted that! Albeit, not publicly, so I can't proveit. My predictions claimed it would have certain limitations, which GPT-3 (and, later, GPT-4) exhibit. I can show that those still exist, but few people on Hacker News seem to understand when I try to communicate that.² > or text to image models. Artbreeder (then called Ganbreeder) existed in early 2020, and it didn't take me by surprise when it came out. It parameterises the output of the model by mapping sliders to regions of the latent space; quite an obvious thing to do, if you want to try getting fine-grained control over the output. (A 2015 paper built on this technique: https://arxiv.org/abs/1508.06576 https://arxiv.org/abs/1508.06576) I was using spaCy back in 2017–2018-time. It represents sentences as vectors, that you can do stuff like cosine similarity on. If I'd been more interested in the field back then, I could have put two and two together, and realised you could train a net on labelled images (with supervised learning) to map a spaCy model's space to StyleGAN's, which would be a text-to-image model. It was very much imaginable back before April of 2020; a wealthy non-researcher hobbyist could've made one, using off-the-shelf tools. If I were better at literature searches, I could probably find you an example of someone who'd done that, or something like it! --- ¹: That file was, because that's what I was playing with in September. I do have some earlier ones, of landscapes; they're in a different folder. ²: See e.g. here: they tell me GPT-4 can translate more than just explicitly-specified meaning, and the "evidence" doesn't even manage that. https://news.ycombinator.com/item?id=35530316 https://news.ycombinator.com/item?id=35530316 (They also think translating the title of a game is the same as translating the game, for some reason; that confusion was probably my fault.)

cubefox 4y ago

> No, it wouldn't have – not to people in the know. We just didn't have powerful enough computers back in the 90s. I'm not talking about the 90s. I'm talking about April 29, 2020.

wizzwizz4 4y ago
What's special about that day? That's after the algorithms were developed, models and drivers were built, and most of these behaviours were discovered. I've got fairly-photorealistic "AI-generated" photos on my laptop timestamped September 2019, and that was before I started learning how it all worked. If you're talking about popular awareness of GPT-style autocomplete, then I agree. If you're talking about academic awareness of what these things can and can't do, we've had that for a while.
- cubefox 4y ago
  What photorealistic AI generated image? In September 2019 this must have been a GAN face. I admit those are impressive, but incredibly limited compared to todays text to image models. If you look at an iPhone from 2019, or a car, or a videogame ... they all still look about the same today. Three years ago there was nothing remotely as impressive as modern GPT style or text to image models. Basically nobody predicted what was about to happen. The only exception I know is Scott Alexander [1]. I don't know about any similar predictions from the experts, but I'm happy to be proven wrong. [1] https://slatestarcodex.com/2019/02/19/gpt-2-as-step-toward-general-intelligence/ https://slatestarcodex.com/2019/02/19/gpt-2-as-step-toward-g...
  - wizzwizz4 4y ago
    > In September 2019 this must have been a GAN face. Well, yes,¹ but actually no. StyleGAN1's public release was February 2019, and it's capable of far more than just faces. > Three years ago there was nothing remotely as impressive as modern GPT style I predicted that! Albeit, not publicly, so I can't proveit. My predictions claimed it would have certain limitations, which GPT-3 (and, later, GPT-4) exhibit. I can show that those still exist, but few people on Hacker News seem to understand when I try to communicate that.² > or text to image models. Artbreeder (then called Ganbreeder) existed in early 2020, and it didn't take me by surprise when it came out. It parameterises the output of the model by mapping sliders to regions of the latent space; quite an obvious thing to do, if you want to try getting fine-grained control over the output. (A 2015 paper built on this technique: https://arxiv.org/abs/1508.06576 https://arxiv.org/abs/1508.06576) I was using spaCy back in 2017–2018-time. It represents sentences as vectors, that you can do stuff like cosine similarity on. If I'd been more interested in the field back then, I could have put two and two together, and realised you could train a net on labelled images (with supervised learning) to map a spaCy model's space to StyleGAN's, which would be a text-to-image model. It was very much imaginable back before April of 2020; a wealthy non-researcher hobbyist could've made one, using off-the-shelf tools. If I were better at literature searches, I could probably find you an example of someone who'd done that, or something like it! --- ¹: That file was, because that's what I was playing with in September. I do have some earlier ones, of landscapes; they're in a different folder. ²: See e.g. here: they tell me GPT-4 can translate more than just explicitly-specified meaning, and the "evidence" doesn't even manage that. https://news.ycombinator.com/item?id=35530316 https://news.ycombinator.com/item?id=35530316 (They also think translating the title of a game is the same as translating the game, for some reason; that confusion was probably my fault.)