3 ms·

ChatGPT is probably the first software product that I have no idea how I'd go about implementing. I watched a number of YT videos about it including Andrew Car

ChatGPT is probably the first software product that I have no idea how I'd go about implementing. I watched a number of YT videos about it including Andrew Carpathy's 2 hour coding session building a mini GPT. I understand the process abstractly, but I am unable to grok the details about how it's able to take my vague interpretation of what I want and then write code and actually give me exactly what I wanted.

minimaxir 4y ago
I would recommend playing with the new ChatGPT API a bit. Trial and error in terms of tweaking the system prompt is surprisingly instructive.
base698 4y ago
The ChatGPT video by Karpathy is the last in a 7 video series. The one that really hit it home for me was the first video on MicroGrad [1]. The second video builds on MicroGrad and is also great for understanding how a basic NN works. [1] https://youtu.be/VMj-3S1tku0 https://youtu.be/VMj-3S1tku0
- antegamisou 4y ago
  Yeah, unless you're already familiar with the advanced math behind all this don't expect an in-depth understanding of the implementation details the parent comment is concerned about.
hgsgm 4y ago
Do you know how you'd go about implementing a video codec? A Haskell compiler? Linux kernel? Efficient relational database? wifi 802.11n radio? An efficient classical AI chess engine?
- starik36 4y ago
  I don't exactly how, but I have a good idea where to start. I've built a compiler before (in college). I've bootstrapped from hardware before, so I probably could built a very basic kernel. I've worked with radios at the low level in the 90s. But with ChatGPT, it just seems like magic.
  - int_19h 4y ago
    None of the actual moving bits are magic, though, and it's way more accessible than an OS kernel. The problem is that 1) you need a lot of raw GPU compute power to train and run a model that has GPT-3 capabilities, and 2) because the end result is a black box, nobody knows how it "actually works". In that sense, it is indeed technomagic.
Jensson 4y ago
It parses natural text into a format representing logic, that is the biggest thing to get. Then once you have such a parser you can now make a logical Markov chain, it predicts the continuation of your text based on continuations of things that looks similar in the logic space rather than the text space, and that is what you get back from the model. So ChatGPT does what we could have easily done if we had if natural language was more logical. Then we could make a simple model based on all the logic encoded in all writing of humanity that just looks up human done logic similar to what you asked for and then return an average of those logics. Now since you want human language output it now has to translate that back to human text from logic space, and it could be done in English, Spanish, it could speak like a pirate etc.

minimaxir 4y ago

I would recommend playing with the new ChatGPT API a bit. Trial and error in terms of tweaking the system prompt is surprisingly instructive.

base698 4y ago

The ChatGPT video by Karpathy is the last in a 7 video series. The one that really hit it home for me was the first video on MicroGrad [1]. The second video builds on MicroGrad and is also great for understanding how a basic NN works. [1] https://youtu.be/VMj-3S1tku0 https://youtu.be/VMj-3S1tku0

antegamisou 4y ago
Yeah, unless you're already familiar with the advanced math behind all this don't expect an in-depth understanding of the implementation details the parent comment is concerned about.

hgsgm 4y ago

Do you know how you'd go about implementing a video codec? A Haskell compiler? Linux kernel? Efficient relational database? wifi 802.11n radio? An efficient classical AI chess engine?

starik36 4y ago
I don't exactly how, but I have a good idea where to start. I've built a compiler before (in college). I've bootstrapped from hardware before, so I probably could built a very basic kernel. I've worked with radios at the low level in the 90s. But with ChatGPT, it just seems like magic.
- int_19h 4y ago
  None of the actual moving bits are magic, though, and it's way more accessible than an OS kernel. The problem is that 1) you need a lot of raw GPU compute power to train and run a model that has GPT-3 capabilities, and 2) because the end result is a black box, nobody knows how it "actually works". In that sense, it is indeed technomagic.

Jensson 4y ago

It parses natural text into a format representing logic, that is the biggest thing to get. Then once you have such a parser you can now make a logical Markov chain, it predicts the continuation of your text based on continuations of things that looks similar in the logic space rather than the text space, and that is what you get back from the model. So ChatGPT does what we could have easily done if we had if natural language was more logical. Then we could make a simple model based on all the logic encoded in all writing of humanity that just looks up human done logic similar to what you asked for and then return an average of those logics. Now since you want human language output it now has to translate that back to human text from logic space, and it could be done in English, Spanish, it could speak like a pirate etc.