9 ms·

As someone who's very skeptical of AI, for a looooong time. I am skeptical about self driving, skeptical about no code, skeptical about ML. But the stable diffu

As someone who's very skeptical of AI, for a looooong time. I am skeptical about self driving, skeptical about no code, skeptical about ML. But the stable diffusion and chat GPT are truly the only AI tech that made my mind blown. I really want to know how it works. It is like magic. This is the first time I've seen AI create truly new things. To me it doesn't matter that what it creates is wrong or right, but the fact that it creates new things, is truly truly, up there with wizardry, nature, magic, etc.

Ancalagon 4y ago
Yup this is incredibly cool. I asked it to build an, albeit simple, redstone building in Minecraft of all things. It didn't get everything quite right (namely, torches can't be used in redstone circuits), but most everything else was correct! > How would you design a simple Minecraft building, which had 4 rooms? Each room would have a single light which could be turned on or off via a switch near the entrance of that room. The redstone used to connect the switches to the lights would not be visible, except for the switches and lights themselves. > To design a simple Minecraft building with 4 rooms and a light switch for each room, I would start by creating a basic structure with 4 walls, a ceiling, and a floor. Each wall would have a door that leads to one of the rooms. Next, I would add a light source in each of the 4 rooms, such as a torch or a lamp. These lights would be placed in the center of the room for maximum visibility. To connect the light switches to the lights, I would use redstone wire and repeaters to create a hidden circuit that connects each switch to the light in its corresponding room. The redstone wire would be hidden behind the walls or under the floor, and the repeaters would be hidden behind blocks or furniture to keep the circuit hidden. Finally, I would add the light switches near the entrance of each room, using a lever or button that players can interact with to turn the lights on or off. This would allow players to easily control the lighting in each room without having to see the redstone wiring. Overall, this simple Minecraft building would have 4 rooms, each with its own light source that can be controlled by a hidden redstone circuit and a visible switch near the entrance of the room. This design would provide a functional and visually appealing lighting system for the building.
simonw 4y ago
I like thinking about GPT-3 in terms of the iPhone predictive text keyboard. It's effectively the same thing: given some words it predicts which word should come next. But unlike the iPhone keyboard it's been trained for months on multiple TBs of text, and has the ability to use ~4,000 previous words as the input to its prediction.
- gptadmirer 4y ago
  I saw inputs like "please write C code that writes lisp code that writes pascal code" and stuffs like "please generate some songs in style of spongebob or KJV" and it made it. Is the power of predictive text that crazy?
  - czzr 4y ago
    Turns out, yes. The hard thing is internalising the sheer scale of data it was trained on.
    - visarga 4y ago
      infinite recursion on language, that's what it internalises, it's not a simple storage
  - simonw 4y ago
    It turns out it is! That's what I find so interesting about large language models: they consistently demonstrate abilities that were not predicted when they were first invented. I don't think anyone working on them initially expected them to be able to write Lisp and Pascal, certainly not as well as they can already.
    - visarga 4y ago
      >> Is the power of predictive text that crazy? > It turns out it is! Intelligence was in the data, we were just its vehicles.
  - pfortuny 4y ago
margalabargala 4y ago
The "fastai" course is free, and does a really nice job walking you through building simple neural nets from the ground up: https://github.com/fastai/fastai https://github.com/fastai/fastai What's going on here is the exact same thing, just much, much larger.
- gptadmirer 4y ago
  Thank you kind sir.
qsort 4y ago
How it works: a probability distribution over sequences of consecutive tokens. Why it works: these absolute madmen downloaded the internet.
- bogdanoff_2 4y ago
  You're forgetting to mention that you need to have a model that is capable of learning that probability distribution.
- gptadmirer 4y ago
  But how does probability distribution over sequences of consecutive tokens can create new things? Like, I saw the other day it creates a C code that creates a Lisp code that creates a Pascal code. Is this based on an entirely previous creation?
  - qsort 4y ago
    It doesn't create anything new. It creates things that look new. The code examples are perfect case studies, they don't actually work. They aren't just slightly wrong, they're completely nonsensical. Another example is "is <number> prime?", it can't answer things like that, and it will make up something that may or may not be accurate. The model has no concept of what is true or false, it's essentially trying to predict what is the most likely token to come next. It seems to know stuff because the knowledge comes from the dataset, hence techniques like zero-shot, few-shot and prompt-based learning.
    - gptadmirer 4y ago
      Interesting, but I wonder how does it have the ability to combine those. i.e, creating a song in a KJV/spongebob style, or creating a code that writes a code that writes a code.
      - jimbokun 4y ago
        Nobody really knows, because the model is too large and complex to really analyze.
idiotsecant 4y ago
Same. I routinely pose the following question to chatbots to see how well they are able to parse strange sentence structure and understand abstract properties. >Please describe the similarities and differences of the following two items: A beach ball and a howitzer cannon. What follows is the response from ChatGPT. For just about the first time I legitimately feel like this beats the turing test. >A beach ball and a howitzer cannon are two very different objects, and they have very few similarities. A beach ball is a small, inflatable ball that is typically used for recreational purposes, such as playing on the beach or in a pool. A howitzer cannon, on the other hand, is a large, heavy artillery weapon that is used to fire shells over long distances. >One of the main differences between a beach ball and a howitzer cannon is their size and weight. A beach ball is small and lightweight, while a howitzer cannon is large and extremely heavy. Another difference is their intended use. A beach ball is designed for play and entertainment, while a howitzer cannon is designed for warfare and destruction. >One possible similarity between the two objects is that they both can be inflated, although this is done for very different reasons. A beach ball is inflated with air to make it bouncy and fun to play with, while a howitzer cannon is typically loaded with gunpowder and other explosives, which are ignited to launch a shell. However, even this similarity is quite tenuous, as the methods of inflation and the materials used are vastly different for the two objects.
- throwaway2037 4y ago
  Incredible. As an educated adult, I cannot distinguish that answer from one that a native speaker would write. The style feels very much like a English comprehension exam you take before university. The written structure is so clean. How long until people are using models like this to cheat on university entrance exams for written comprehension? Very soon.
- gptadmirer 4y ago
  Crazy, truly crazy.
- dTal 4y ago
  This strikes me as a very intriguing glimpse into its "mind". No human would describe loading a howitzer with gunpowder as "inflating" - the howitzer does not increase in volume. However it's clearly grasped that inflating involves putting something into something else. I wonder how it would respond if you asked it to define the word?
anentropic 4y ago
I've been pretty underwhelmed by stable diffusion so far (admittedly even this much would have seemed like magic to me 10 years ago). First thing I asked it for was a picture of a dragon. I've subsequently a few different models and all sorts of prompt engineering (but perhaps I still haven't found the right one?)... I cannot get it to draw something anatomically coherent. Are there some tricks I am missing? Do I need to run it through a pipeline of further steps to refine the mangled creature into something that makes sense?
- Nemrod67 4y ago
  go try Midjourney on Discord, I'm sure it can "draw" you a dragon just fine ;)
  - anentropic 4y ago
    I have done exactly that... the results were basically the same as I get from DiffusionBee app for stable diffusion i.e. regions of the image are locally impressive, it has understood the prompt well, but the overall picture is incoherent... the head or one or more legs may be missing, or legs or wings sprout from odd places like, it gets the 'texture' spot on but the 'structure' is off
    - anentropic 4y ago
      So I was wondering if it needs a more complicated procedure? Lateral thinking? Should I be asking it for a picture of a cat in the style and setting I want and then use image-to-image to replace the cat with a dragon?
blihp 4y ago
It can only interpolate, not extrapolate. So the 'new' things you're seeing are just rearrangements of the (millions/billions of) things that the DNN was trained on. It has no understanding of what it has 'learned' (or more accurately: lossy memorization a.k.a. compression) and makes all kinds of mistakes (some due to training losses, some due to garbage/conflicting data fed in.) This is probably why the creative applications (i.e. Stable Diffusion etc) seem more impressive than the functional applications (i.e. Galactica) as even 'wrong' output can be creatively interesting. For example, if a new comic/movie/video game character came out tomorrow that had a very unique physical look, Stable Diffusion would have difficulty even approximating it (i.e. its training data wouldn't have what was needed to reproduce the appearance.) But it can produce a decent Darth Vader because it's probably been fed at least thousands of drawings/photos of this very well known fictional character.
- Filligree 4y ago
  You need about ten pictures for Dreambooth to give Stable Diffusion a good idea of what a new character looks like.
- krackers 4y ago
  That's like saying the novels people write are just rearrangements of words we learned as a kid. I don't see how you can't possibly consider something like this [1] as extrapolation and genuine creation. [1] https://twitter.com/pic/orig/media%2FFi4HMw9WQAA3j-m.jpg https://twitter.com/pic/orig/media%2FFi4HMw9WQAA3j-m.jpg
  - renewiltord 4y ago
    link broken
    - krackers 4y ago
      Sorry, try https://pbs.twimg.com/media/Fi4HMw9WQAA3j-m.jpg https://pbs.twimg.com/media/Fi4HMw9WQAA3j-m.jpg Source: https://twitter.com/typedfemale/status/1598222844902936578 https://twitter.com/typedfemale/status/1598222844902936578
spaceman_2020 4y ago
Same. This chat assistant blew my mind. I've ignored most of the bots that were released before this because they would trip up over trivial issues. But this...it's an actual assistant. I wanted to know how to figure out device type based on screen width. Google sent me down half a dozen articles. I asked this AI and it spat out a function. I can see myself using this very regularly and even paying a fee for it.
ajuc 4y ago
> I really want to know how it works. It's a big hash table, turns out human intelligence can be cashed and interpolated.
- visarga 4y ago
  Oh no, that sounds like the Chinese Room.
visarga 4y ago
> I really want to know how it works. You may be disappointed to know that the exact inner workings of the model are still largely unknown. We understand the basics of how it works, such as how changes in the model size and data size will affect its performance, or how to combine various supervised datasets to train it to solve tasks and what the model probabilities are supposed to mean, but the complexity of the model is such that it cannot be fully simulated or imagined. It is similar to the workings of a cell, the brain, or even a protein - we know the basics, but the full complexity of it is beyond our current understanding. The true complexity is in the weights, and in the dynamic activation patterns. This excellent visualisation article will show you the state of the art in interpreting transformers. They develop a new way to peek into the network. https://www.lesswrong.com/posts/mkbGjzxD8d8XqKHzA/the-singular-value-decompositions-of-transformer-weight https://www.lesswrong.com/posts/mkbGjzxD8d8XqKHzA/the-singul... https://www.youtube.com/watch?v=pC4zRb_5noQ https://www.youtube.com/watch?v=pC4zRb_5noQ

Ancalagon 4y ago

Yup this is incredibly cool. I asked it to build an, albeit simple, redstone building in Minecraft of all things. It didn't get everything quite right (namely, torches can't be used in redstone circuits), but most everything else was correct! > How would you design a simple Minecraft building, which had 4 rooms? Each room would have a single light which could be turned on or off via a switch near the entrance of that room. The redstone used to connect the switches to the lights would not be visible, except for the switches and lights themselves. > To design a simple Minecraft building with 4 rooms and a light switch for each room, I would start by creating a basic structure with 4 walls, a ceiling, and a floor. Each wall would have a door that leads to one of the rooms. Next, I would add a light source in each of the 4 rooms, such as a torch or a lamp. These lights would be placed in the center of the room for maximum visibility. To connect the light switches to the lights, I would use redstone wire and repeaters to create a hidden circuit that connects each switch to the light in its corresponding room. The redstone wire would be hidden behind the walls or under the floor, and the repeaters would be hidden behind blocks or furniture to keep the circuit hidden. Finally, I would add the light switches near the entrance of each room, using a lever or button that players can interact with to turn the lights on or off. This would allow players to easily control the lighting in each room without having to see the redstone wiring. Overall, this simple Minecraft building would have 4 rooms, each with its own light source that can be controlled by a hidden redstone circuit and a visible switch near the entrance of the room. This design would provide a functional and visually appealing lighting system for the building.

simonw 4y ago

I like thinking about GPT-3 in terms of the iPhone predictive text keyboard. It's effectively the same thing: given some words it predicts which word should come next. But unlike the iPhone keyboard it's been trained for months on multiple TBs of text, and has the ability to use ~4,000 previous words as the input to its prediction.

gptadmirer 4y ago
I saw inputs like "please write C code that writes lisp code that writes pascal code" and stuffs like "please generate some songs in style of spongebob or KJV" and it made it. Is the power of predictive text that crazy?
- czzr 4y ago
  Turns out, yes. The hard thing is internalising the sheer scale of data it was trained on.
  - visarga 4y ago
    infinite recursion on language, that's what it internalises, it's not a simple storage
- simonw 4y ago
  It turns out it is! That's what I find so interesting about large language models: they consistently demonstrate abilities that were not predicted when they were first invented. I don't think anyone working on them initially expected them to be able to write Lisp and Pascal, certainly not as well as they can already.
  - visarga 4y ago
    >> Is the power of predictive text that crazy? > It turns out it is! Intelligence was in the data, we were just its vehicles.
- pfortuny 4y ago

margalabargala 4y ago

The "fastai" course is free, and does a really nice job walking you through building simple neural nets from the ground up: https://github.com/fastai/fastai https://github.com/fastai/fastai What's going on here is the exact same thing, just much, much larger.

gptadmirer 4y ago
Thank you kind sir.

qsort 4y ago

How it works: a probability distribution over sequences of consecutive tokens. Why it works: these absolute madmen downloaded the internet.

bogdanoff_2 4y ago
You're forgetting to mention that you need to have a model that is capable of learning that probability distribution.
gptadmirer 4y ago
But how does probability distribution over sequences of consecutive tokens can create new things? Like, I saw the other day it creates a C code that creates a Lisp code that creates a Pascal code. Is this based on an entirely previous creation?
- qsort 4y ago
  It doesn't create anything new. It creates things that look new. The code examples are perfect case studies, they don't actually work. They aren't just slightly wrong, they're completely nonsensical. Another example is "is <number> prime?", it can't answer things like that, and it will make up something that may or may not be accurate. The model has no concept of what is true or false, it's essentially trying to predict what is the most likely token to come next. It seems to know stuff because the knowledge comes from the dataset, hence techniques like zero-shot, few-shot and prompt-based learning.
  - gptadmirer 4y ago
    Interesting, but I wonder how does it have the ability to combine those. i.e, creating a song in a KJV/spongebob style, or creating a code that writes a code that writes a code.
    - jimbokun 4y ago
      Nobody really knows, because the model is too large and complex to really analyze.

idiotsecant 4y ago

Same. I routinely pose the following question to chatbots to see how well they are able to parse strange sentence structure and understand abstract properties. >Please describe the similarities and differences of the following two items: A beach ball and a howitzer cannon. What follows is the response from ChatGPT. For just about the first time I legitimately feel like this beats the turing test. >A beach ball and a howitzer cannon are two very different objects, and they have very few similarities. A beach ball is a small, inflatable ball that is typically used for recreational purposes, such as playing on the beach or in a pool. A howitzer cannon, on the other hand, is a large, heavy artillery weapon that is used to fire shells over long distances. >One of the main differences between a beach ball and a howitzer cannon is their size and weight. A beach ball is small and lightweight, while a howitzer cannon is large and extremely heavy. Another difference is their intended use. A beach ball is designed for play and entertainment, while a howitzer cannon is designed for warfare and destruction. >One possible similarity between the two objects is that they both can be inflated, although this is done for very different reasons. A beach ball is inflated with air to make it bouncy and fun to play with, while a howitzer cannon is typically loaded with gunpowder and other explosives, which are ignited to launch a shell. However, even this similarity is quite tenuous, as the methods of inflation and the materials used are vastly different for the two objects.

throwaway2037 4y ago
Incredible. As an educated adult, I cannot distinguish that answer from one that a native speaker would write. The style feels very much like a English comprehension exam you take before university. The written structure is so clean. How long until people are using models like this to cheat on university entrance exams for written comprehension? Very soon.
gptadmirer 4y ago
Crazy, truly crazy.
dTal 4y ago
This strikes me as a very intriguing glimpse into its "mind". No human would describe loading a howitzer with gunpowder as "inflating" - the howitzer does not increase in volume. However it's clearly grasped that inflating involves putting something into something else. I wonder how it would respond if you asked it to define the word?

anentropic 4y ago

I've been pretty underwhelmed by stable diffusion so far (admittedly even this much would have seemed like magic to me 10 years ago). First thing I asked it for was a picture of a dragon. I've subsequently a few different models and all sorts of prompt engineering (but perhaps I still haven't found the right one?)... I cannot get it to draw something anatomically coherent. Are there some tricks I am missing? Do I need to run it through a pipeline of further steps to refine the mangled creature into something that makes sense?

Nemrod67 4y ago
go try Midjourney on Discord, I'm sure it can "draw" you a dragon just fine ;)
- anentropic 4y ago
  I have done exactly that... the results were basically the same as I get from DiffusionBee app for stable diffusion i.e. regions of the image are locally impressive, it has understood the prompt well, but the overall picture is incoherent... the head or one or more legs may be missing, or legs or wings sprout from odd places like, it gets the 'texture' spot on but the 'structure' is off
  - anentropic 4y ago
    So I was wondering if it needs a more complicated procedure? Lateral thinking? Should I be asking it for a picture of a cat in the style and setting I want and then use image-to-image to replace the cat with a dragon?

blihp 4y ago

It can only interpolate, not extrapolate. So the 'new' things you're seeing are just rearrangements of the (millions/billions of) things that the DNN was trained on. It has no understanding of what it has 'learned' (or more accurately: lossy memorization a.k.a. compression) and makes all kinds of mistakes (some due to training losses, some due to garbage/conflicting data fed in.) This is probably why the creative applications (i.e. Stable Diffusion etc) seem more impressive than the functional applications (i.e. Galactica) as even 'wrong' output can be creatively interesting. For example, if a new comic/movie/video game character came out tomorrow that had a very unique physical look, Stable Diffusion would have difficulty even approximating it (i.e. its training data wouldn't have what was needed to reproduce the appearance.) But it can produce a decent Darth Vader because it's probably been fed at least thousands of drawings/photos of this very well known fictional character.

Filligree 4y ago
You need about ten pictures for Dreambooth to give Stable Diffusion a good idea of what a new character looks like.
krackers 4y ago
That's like saying the novels people write are just rearrangements of words we learned as a kid. I don't see how you can't possibly consider something like this [1] as extrapolation and genuine creation. [1] https://twitter.com/pic/orig/media%2FFi4HMw9WQAA3j-m.jpg https://twitter.com/pic/orig/media%2FFi4HMw9WQAA3j-m.jpg
- renewiltord 4y ago
  link broken
  - krackers 4y ago
    Sorry, try https://pbs.twimg.com/media/Fi4HMw9WQAA3j-m.jpg https://pbs.twimg.com/media/Fi4HMw9WQAA3j-m.jpg Source: https://twitter.com/typedfemale/status/1598222844902936578 https://twitter.com/typedfemale/status/1598222844902936578

spaceman_2020 4y ago

Same. This chat assistant blew my mind. I've ignored most of the bots that were released before this because they would trip up over trivial issues. But this...it's an actual assistant. I wanted to know how to figure out device type based on screen width. Google sent me down half a dozen articles. I asked this AI and it spat out a function. I can see myself using this very regularly and even paying a fee for it.

ajuc 4y ago

> I really want to know how it works. It's a big hash table, turns out human intelligence can be cashed and interpolated.

visarga 4y ago
Oh no, that sounds like the Chinese Room.

visarga 4y ago

> I really want to know how it works. You may be disappointed to know that the exact inner workings of the model are still largely unknown. We understand the basics of how it works, such as how changes in the model size and data size will affect its performance, or how to combine various supervised datasets to train it to solve tasks and what the model probabilities are supposed to mean, but the complexity of the model is such that it cannot be fully simulated or imagined. It is similar to the workings of a cell, the brain, or even a protein - we know the basics, but the full complexity of it is beyond our current understanding. The true complexity is in the weights, and in the dynamic activation patterns. This excellent visualisation article will show you the state of the art in interpreting transformers. They develop a new way to peek into the network. https://www.lesswrong.com/posts/mkbGjzxD8d8XqKHzA/the-singular-value-decompositions-of-transformer-weight https://www.lesswrong.com/posts/mkbGjzxD8d8XqKHzA/the-singul... https://www.youtube.com/watch?v=pC4zRb_5noQ https://www.youtube.com/watch?v=pC4zRb_5noQ