4 ms·

As the lawyer behind the one-click patent once personally explained to me, it's the claims that matter most. And the first claim matters more than all of the ot

by ttul 3y ago

As the lawyer behind the one-click patent once personally explained to me, it's the claims that matter most. And the first claim matters more than all of the other sub-claims. This patent specifically covers ONLY transformers in which there is an encoder and a decoder. Claim 1 of the patent contains the following: "...the sequence transduction neural network comprising: an encoder neural network configured to receive the input sequence and generate a respective encoded representation of each of the network inputs in the input sequence... and a decoder neural network configured to receive the encoded representations and generate the output sequence." Claims 29 and 30 (the only other independent claims) also specify an encoder and a decoder. So long as your transformer network does not make use of an encoder in combination with a decoder, this patent does not apply to you.

ttul 3y ago
Of course, in patent US11556786B2, Google finally mopped up the decoder-only idea. https://patents.google.com/patent/US11556786B2/en https://patents.google.com/patent/US11556786B2/en "1. A method of generating an output sequence comprising a plurality of output tokens from an input sequence comprising a plurality of input tokens, the method comprising, at each of a plurality of generation time steps: generating a combined sequence for the generation time step that includes the input sequence followed by the output tokens that have already been generated as of the generation time step; processing the combined sequence using a self-attention decoder neural network, wherein the self-attention decoder neural network comprises a plurality of neural network layers that include a plurality of masked self-attention neural network layers, and wherein the self-attention decoder neural network is configured to process the combined sequence through the plurality of neural network layers to generate a time step output that defines a score distribution over a set of possible output tokens; and selecting, using the time step output, an output token from the set of possible output tokens as the next output token in the output sequence."

by ttul 3y ago

ttul 3y ago

Of course, in patent US11556786B2, Google finally mopped up the decoder-only idea. https://patents.google.com/patent/US11556786B2/en https://patents.google.com/patent/US11556786B2/en "1. A method of generating an output sequence comprising a plurality of output tokens from an input sequence comprising a plurality of input tokens, the method comprising, at each of a plurality of generation time steps: generating a combined sequence for the generation time step that includes the input sequence followed by the output tokens that have already been generated as of the generation time step; processing the combined sequence using a self-attention decoder neural network, wherein the self-attention decoder neural network comprises a plurality of neural network layers that include a plurality of masked self-attention neural network layers, and wherein the self-attention decoder neural network is configured to process the combined sequence through the plurality of neural network layers to generate a time step output that defines a score distribution over a set of possible output tokens; and selecting, using the time step output, an output token from the set of possible output tokens as the next output token in the output sequence."