Tarq0n: Also a great point. The question then is whether the attention mechanism is being triggered on the proper word or character sequences. Lawyers employ a

Tarq0n: Also a great point. The question then is whether the attention mechanism is being triggered on the proper word or character sequences. Lawyers employ a form of attention mechanism when “issue spotting.” For example, in a non-compete I might scan for the duration (how many years until the client can join a similar venture?) and breadth (what is the definition of a similar venture?). For an attention mechanism to work well for legal summaries, it seems to me that it must trigger on many relevant context triggers at different levels. As an extreme example, if I put 50 contracts involving multiple different parties into one big document and the attention mechanism was triggered on the first document title, would the subsequent document titles sufficiently demarcate a new contract to reset the context? Or would the attention stay “on” the first high accuracy context match and mix up the terms and parties? In context of the paper, GPT-3 missed a ton of issues, indicating to me that the attention mechanism is not being properly triggered. Again I may be wrong that tokenizing and predicting based off of clauses, paragraphs and sections would help improve the output, but that was one way I thought would capture the different levels of context.