5 ms·

A Logic for Expressing Transformers

2 points by lambdaviking 3y ago

lambdaviking 3y ago
Our recent paper proves that transformers can be translated to sentences in first-order logic with majority-vote quantifiers (FOM). FOM is a symbolic language that can capture computation inside transformers!
- ftxbro 3y ago
  If some method becomes popular (like transformers in this case) there are so many waves of papers where people 'interpret' the new thing by saying how it's really [their own research project / grant / area of study]. It's like when 'deep learning' came there were also waves of papers like this. Is yours different from these kinds of papers?
  - lambdaviking 3y ago
    Good question! A key use for this type of analysis is that it acts as an upper bound on transformers, rather just demonstrating "transformers are like logic". That is, it allows us to fairly easily identify problems that transformers cannot solve. Some examples are graph connectivity, linear programming, various forms of state tracking, etc. Definitely finding negative results like this is tough by just doing experiments, since it could be that the model might have succeeded if you had just trained it for more time/made it bigger/etc. Our prior paper discusses how this works more in depth: https://arxiv.org/abs/2207.00729 https://arxiv.org/abs/2207.00729
    - ftxbro 3y ago
      > "Definitely finding negative results like this is tough by just doing experiments, since it could be that the model might have succeeded if you had just trained it for more time/made it bigger/etc." This is interesting to me because I've been impressed by how we have unlocked so many cognitive abilities just by scaling token predictors. It seems like every time the perplexity is significantly reduced, a whole new set of capabilities are unlocked. How do the results of your work inform these predictions that stack-more-layers token-prediction scaling will keep unlocking new abilities? For example, how would it affect a prediction of the nature and capabilities of a hypothetical 'GPT-5'-like model that reduces perplexity by a comparable about to the reduction that happened between 'GPT-3' and 'GPT-4' (maybe comparable on some nonlinear scale)?
      - lambdaviking 3y ago
        > How do the results of your work inform these predictions that stack-more-layers token-prediction scaling will keep unlocking new abilities? The short answer is that scaling up alone probably won't fix these shortcomings. Better decoding strategies like CoT may help, but the issue we're identifying is a limitation of any transformer (no matter the size) arising from its parallelism (the property that enables scaling to large model size/data volume). As some evidence of this, we were able to use these results to find simple problems that ChatGPT and GPT4 get chance performance on. Moreover, the questions seem to lead these models to hallucinate supporting evidence for the incorrect answers: https://arxiv.org/abs/2305.13534 https://arxiv.org/abs/2305.13534
        ftxbro 3y ago
        Thanks that one is very interesting. I especially like how GPT4 performs worse (zero shot greedy decoding) than ChatGPT on those questions. I wonder have you thought about how it relates to 'inverse scaling' (https://github.com/inverse-scaling/prize https://github.com/inverse-scaling/prize)? As I was reading it I was thinking how that kind of 'hallucination' is exactly the problem that beam search (as opposed to greedy decoding) is designed to solve, and I see this was addressed later in the paper - unfortunately you guys didn't have access to it on the API and also no access to other things like log likelihoods. A related question I had, is to decompose the blame for the problems these LLMs were having into two parts: (1) the weaknesses of the model size and data size and architecture caused for example by parallelism tradeoffs versus (2) the inherent problems of greedy token prediction. For example I could imagine that even an 'oracular' (kolmogorov or solomonoff or whoever absolute minimum theoretical perplexity) single-token-at-a-time-predictor could have related errors purely as a result of greedy decoding, which would be in no way attributable to any kind of size issue or architectural property or parallelism property of the model. As I guess you already know, the reason is just that 'the most likely next token of the response' isn't necessarily the same as 'the next token of the most likely response'.

2 points by lambdaviking 3y ago

Our recent paper proves that transformers can be translated to sentences in first-order logic with majority-vote quantifiers (FOM). FOM is a symbolic language that can capture computation inside transformers!

ftxbro 3y ago
If some method becomes popular (like transformers in this case) there are so many waves of papers where people 'interpret' the new thing by saying how it's really [their own research project / grant / area of study]. It's like when 'deep learning' came there were also waves of papers like this. Is yours different from these kinds of papers?
- lambdaviking 3y ago
  Good question! A key use for this type of analysis is that it acts as an upper bound on transformers, rather just demonstrating "transformers are like logic". That is, it allows us to fairly easily identify problems that transformers cannot solve. Some examples are graph connectivity, linear programming, various forms of state tracking, etc. Definitely finding negative results like this is tough by just doing experiments, since it could be that the model might have succeeded if you had just trained it for more time/made it bigger/etc. Our prior paper discusses how this works more in depth: https://arxiv.org/abs/2207.00729 https://arxiv.org/abs/2207.00729
  - ftxbro 3y ago
    > "Definitely finding negative results like this is tough by just doing experiments, since it could be that the model might have succeeded if you had just trained it for more time/made it bigger/etc." This is interesting to me because I've been impressed by how we have unlocked so many cognitive abilities just by scaling token predictors. It seems like every time the perplexity is significantly reduced, a whole new set of capabilities are unlocked. How do the results of your work inform these predictions that stack-more-layers token-prediction scaling will keep unlocking new abilities? For example, how would it affect a prediction of the nature and capabilities of a hypothetical 'GPT-5'-like model that reduces perplexity by a comparable about to the reduction that happened between 'GPT-3' and 'GPT-4' (maybe comparable on some nonlinear scale)?
    - lambdaviking 3y ago
      > How do the results of your work inform these predictions that stack-more-layers token-prediction scaling will keep unlocking new abilities? The short answer is that scaling up alone probably won't fix these shortcomings. Better decoding strategies like CoT may help, but the issue we're identifying is a limitation of any transformer (no matter the size) arising from its parallelism (the property that enables scaling to large model size/data volume). As some evidence of this, we were able to use these results to find simple problems that ChatGPT and GPT4 get chance performance on. Moreover, the questions seem to lead these models to hallucinate supporting evidence for the incorrect answers: https://arxiv.org/abs/2305.13534 https://arxiv.org/abs/2305.13534
      - ftxbro 3y ago
        Thanks that one is very interesting. I especially like how GPT4 performs worse (zero shot greedy decoding) than ChatGPT on those questions. I wonder have you thought about how it relates to 'inverse scaling' (https://github.com/inverse-scaling/prize https://github.com/inverse-scaling/prize)? As I was reading it I was thinking how that kind of 'hallucination' is exactly the problem that beam search (as opposed to greedy decoding) is designed to solve, and I see this was addressed later in the paper - unfortunately you guys didn't have access to it on the API and also no access to other things like log likelihoods. A related question I had, is to decompose the blame for the problems these LLMs were having into two parts: (1) the weaknesses of the model size and data size and architecture caused for example by parallelism tradeoffs versus (2) the inherent problems of greedy token prediction. For example I could imagine that even an 'oracular' (kolmogorov or solomonoff or whoever absolute minimum theoretical perplexity) single-token-at-a-time-predictor could have related errors purely as a result of greedy decoding, which would be in no way attributable to any kind of size issue or architectural property or parallelism property of the model. As I guess you already know, the reason is just that 'the most likely next token of the response' isn't necessarily the same as 'the next token of the most likely response'.