3 ms·
One thing I find fascinating about GPT-4 (and I'm curious about your take) is that it can not only generate novel, non-trivial code, but it can (upon request) o
by resters 3y ago
One thing I find fascinating about GPT-4 (and I'm curious about your take) is that it can not only generate novel, non-trivial code, but it can (upon request) output that code as a base64 encoded string... seemingly all from the model itself.
- mike_hearn 3y agoI don't have a great explanation for that, other than the obvious trivial one that it must have seen a lot of base64 encoded text alongside the decoded text in its training set, and that was sufficient for a small part of the network to learn how to decode it. If you look at visualizations of smaller RNNs trained on code then you can identify neurons that activate for things like "inside a quoted string", "inside the expression of an if statement" and so on.