3 ms·
I don't believe it does. To me, it implies that over K passes the statement must be executed P times, with P being close to N/100 * K. Assuming N - the specifi
by TeMPOraL 3y ago
I don't believe it does. To me, it implies that over K passes the statement must be executed P times, with P being close to N/100 * K.
Assuming N - the specified percent chance - can only be an integer constant, naive implementation would hardcode random-looking pattern for all 100 values of N.
- layer8 3y agoI interpret it more literally, as “each time it is encountered in the course of execution”, “the statement” must “have only the specified percent chance of being executed”. The “each time” for me implies that you can’t just consider the global statistics over all execution, but instead the probability must hold for each single execution of the statement. If it is 100% predictable, that would mean that for any given single execution of the statement the probability would be either 0 or 100%, so in general not the specified probability.
- TeMPOraL 3y agoI agree this is more reasonable, but at least in programming languages with a language standard and multiple compiler/runtime implementations, you have to read it like a language lawyer - if the spec allows multiple reads, compiler/runtime authors are free to choose whichever read they like, and conversely, users must not assume anything but what's explicitly stated. With that in mind, my (however weak) defense of GPT-4 here is that it's taking a language lawyer approach to the problem and refuses to make assumptions narrower than justified by the reference documentation it knows.
- db48x 3y agoNo, GPT-4 isn’t reasoning about the language standard (as a language lawyer or otherwise), because GPT-4 does not reason about anything. All it does is pick the most likely next token, given the ones that have come before. Very few people talk about INTERCAL on the internet, so it has very little training data to go on. It keeps regurgitating parts of the documentation and the wiki page because that’s all it has. For example, the line “… the `%50` operation (which in INTERCAL means to mingle the bits of two 16-bit numbers) …” is nonsense because the mingle operation is denoted by a “¢” character, not a “%”. All the language model knows is that the tokens “to mingle the bits of two 16-bit numbers” are likely to follow “which in INTERCAL means”.
- TeMPOraL 3y agoAt GPT-4 level, saying "All it does is pick the most likely next token, given the ones that have come before" is like saying all people do is actions that maximize their wealth, based on what they saw worked best before. Technically close enough, but also missing the point. Yes, GPT-4 is predicting next tokens best associatied with ones that came before. It does so based not on one score, but a hundred thousand ones. The extreme amount of dimensions of the latent space are enough to capture pretty much any kind of semantic or structural association you can come up with. I'm not convinced that this isn't sufficient in principle to cover most of what we'd call reasoning.
- db48x 3y agoNo matter how many parameters you give it, it’s still just predicting the next token is most likely, given the training data. The number of parameters is what gives it the ability to pattern match a wide variety of inputs and generate acceptable outputs, but it doesn’t give it any reasoning ability. It can indeed capture textual associations, like the fact that the sequence of tokens “mingle the bits of two 16-bit numbers” is commonly associated with INTERCAL. It just has no reasoning ability, so it cannot recognize that it has incorrectly associated it with the % operator. It just didn’t have anything with a higher probability.
- layer8 3y agoI don’t think GPT is really reasoning on that level. In addition, that would mean that you can’t write a password generator in most languages, because most languages don’t give you any strict randomness guarantees. Yet I bet GPT would be happy to write a password generator in C or Python.