3 ms·
1. The trained model has 7B parameters or weights for each neuron. 2. It can handle upto 8k tokens. Tokens are usually some representation for a word. If your
by going_ham 3y ago
1. The trained model has 7B parameters or weights for each neuron.
2. It can handle upto 8k tokens. Tokens are usually some representation for a word. If your tokens are characters then, "h", "e", "y" represent 3 tokens for hey. Most of the algos use byte pair encoding. For example "hand-le" has two tokens "hand" and "le". This is a very crud example which is enough to give the gist but is not accurate. You can look into byte pair encoding for more details.
3. The token size 1.5T token means they have huge variations for input and output. Simply put, it was trained on large data corpus.
I hope this simplifies it. You can research further if you are interested! Hope it helps!
- thomasahle 3y agoDon't just post ChatGPT answers as comments on hackernews. This one doesn't even make any sense. Of course it doesn't have 7B parameters _per_ neuron.
- johanvts 3y agoDon’t be too critical dr. Ahle. Maybe it’s a new single-neuron architecture.
- Sunhold 3y agoDoesn't look like ChatGPT. Grammatical errors like "on large data corpus," the poor comma usage, misspelling crude, etc. are more of a human thing.
- thomasahle 3y agoMaybe you are right. I was confused by sentences like "I hope this simplifies it. You can research further if you are interested! Hope it helps!" which seemed to be responding to a prompt other than just the previous comment.
- going_ham 3y agoSorry doc, I wrote that comment in a smartphone without putting any thought. What I wanted to say was: > there are 7B parameters. A parameter is a weight assigned to single neuron. I hope this clarifies the answer now. Now that is done I am quite curious on how you came up with the idea it was written by ChatGPT? I just wanted to simplify as best as I could. It’s funny you thought it that way. What could I have done so that it didn’t sound like response from ChatGPT? I am asking it to prevent future misunderstandings. I thought my grammatical errors would be enough to show it wasn’t a ChatGPT response. Looking forward to your reply!
- raincole 3y agoThis is a good example on why StackOverflow banned ChatGPT-generated answers.