3 ms·

ChatGPT can't really introspect though. It has no idea how it works, so it'll just blurt out something that sounds feasible, biased by your prompt. The slow pr

• tkanarsky 4y ago
  What specifically causes the output layer to trickle-print the response like that? I thought it was a skeumorphic effect to simulate a human typing out an answer slowly.
  • redox99 4y ago
    Its input is a sequence of tokens (so text), and its output a list of probabilities for the single next token. You pick the highest probability token, append it to input and execute the neural network again. If it outputs an "<end>" token, it means it's done you stop this "while loop". In other words, it outputs one token at a time.
    • EricLeer 4y ago
      True, but this entire loop happens within the model, if you would return an output at every intermediate step the model would be extremely slow. My take on why they have build the output layer like it is, is that next to feeling more human, it also forces you to be a bit more thoughtfull with your requests, and thus spam the system less. In the end it is still really expensive to run these models..
      • redox99 4y ago
        > True, but this entire loop happens within the model No, it's literally a for loop in python that runs the whole thing from scratch[1] after appending each new token. No artificial slowdowns, what you're seeing is literally what it's spitting out in real time Here's an example https://github.com/karpathy/minGPT/blob/master/mingpt/model.py#L289 https://github.com/karpathy/minGPT/blob/master/mingpt/model.... [1] Some things can be cached such as kv, but still