8 ms·
Patent for attention-based sequence transduction neural networks (2019)
- netdur 3y agohere your moat google!
- arjvik 3y agoWhat are transduction neural networks, and how are they different from existing attention-based transformer models?
- mcyc 3y agoThe mean transduction in the sense of sequence-to-sequence models (transducing one sequence to another). Attention based models don't necessarily need to be sequence to sequence. They can be classifiers, decoder only, etc. Attention is just one tool in the ML architecture toolkit.
- ttul 3y agoPatent lingo is necessarily more generic than the lingo used within the machine learning space. For example, within patents, they will talk about using computers to do things, which is obvious outside of the patent world. But when writing a patent, you have to actually mention that you're going to use a computer to implement a neural network.
- Jackson__ 3y ago>2018-06-28 - Application filed by Google LLC 2019-10-22 - Application granted Should probably put a (2019) in the title. Furthermore considering how fast the ML space moves, the fact that google hasn't used this to create a model significantly better than competitors seems to show that the patented architecture did not perform better than others.
- dragonwriter 3y agoOr, it shows that Google can’t deliver products in this space to (literally) save its life, no matter how good its technology is (which they’ve also shown lots of other ways.)
- onlyrealcuzzo 3y agoYes, the company with 8 billion user products can't do anything.
- brookst 3y agoDo you think Google has 8 billion products in the AI space? Because it seems pretty clear that the criticism is that Google is an advertising company that uses search as leadgen, and is struggling to ship AI products.
- cma 3y agoGoogle translate is widely used. Search also uses transformers for generating snippit summary boxes.
- brookst 3y agoSo, would you say Google has been as successful with AI products as anyone else?
- onlyrealcuzzo 3y agoWho is making a ton of money on AI? What AI product has more users than Google Translate?
- gerash 3y agoThe patent was granted in 2019. The patented architecture is the Transformer architecture that everyone and their mom is now using for LLMs. Patents in your portfolio help defend against trolls and other hostile entities.
- tough 3y agoLet's hope Google doesn't become the Troll here
- lettergram 3y agoIs it really trolling if you’re utilizing the tech and someone is infringing?
- mabbo 3y agoConsider Amazon One-click purchase. That patent was one of very few (out of tens of thousands) that they defended heavily. Every other company on the planet had to have at least two steps to purchase an item or Amazon would come knocking with lawyers. They did this until the day that patent finally expired. One has to ask: was that to the benefit of society? Google may make good use of this tech. But it would be better for all of us if everyone did and didn't have to pay them a fee for the privilege.
- amelius 3y ago> Google may make good use of this tech. But it would be better for all of us if everyone did and didn't have to pay them a fee for the privilege. This is reverse logic because then Google would not have published the tech and would have kept it a trade secret.
- lettergram 3y ago100% it’s probably worth it in the end. You quite literally may not have had Amazon without it. In this case, google might not have funded the development of transformers which could dramatically reduce costs of everything for humanity. Same goes for drug development. That said, I think there’s a question around software patents and how long they should last. Perhaps it should just be to recoup costs, plus some multiple. I’m not sure.
- theGnuMe 3y agoI think this patent is invalid based on the disclosure date. The attention is all you need paper was on Arxiv on 12 Jun 2017 and you have to patent within one year. In this case June 28th 2018 is too late..
- avsteele 3y agoI doubt it. They probably filed a provisional earlier.
- theGnuMe 3y agoAh yes, looks like they did.
- kurthr 3y agoApparently, they thought of that, Line 3 of the patent application US10452978B2: Provisional Application No. 62/510,256, filed on May 23, 2017 There are however other patents and applications: Neural machine translation with latent tree attention US20180300317A1 James BRADBURY I'm not an expert so I can't read the spec and claims as to relevance with authority. However, it still wouldn't count directly as prior art as it was published in Oct 2018. Patents are now first to file and not first to invent (to match the rest of the world). edit: They do also site non-patent prior are regarding attention. Whether this covers self attention I'm not clear and of course their claims have been reviewed by the examiner in light of the art so they're presumed valid until re-exam. Luong et al. "Effective approaches to attention based neural machine translation," arXiv 1508.04025v2, Sep. 20, 2015, 11 pages.Luong et al. "Effective approaches to attention based neural machine translation," arXiv 1508.04025v2, Sep. 20, 2015, 11 pages.
- cma 3y ago> However, it still wouldn't count directly as prior art as it was published in Oct 2018. Patents are now first to file and not first to invent (to match the rest of the world). There is an investor's grace period, as long as your own public prior art disclosure is the earliest, you get 1 year to file even under first to file. First to file refers to people filing for undisclosed inventions, disclosure acts as prior art against anyone else filing, along with the grace period for delaying your filing. Often "wow the company detailing internals at this conference is so generous!" is actually them getting an extra year on the of patent expiration date by taking advantage of the grace period.
- CaptainNegative 3y agoSilly question, how did they get their patent granted in 16 months when I have some patents pending for more than 3x as long?
- netdur 3y agothey have better lawyer
- ilrwbwrkhv 3y agoGoogle might go down as one of the greatest business failures in history.
- iamgopal 3y agoIf one would have opportunity, who should be the CEO of the Google ?
- nerdo 3y agoMarissa Meyer
- WanderPanda 3y agoElon, releasing 150k people from their golden handcuffs
- speedgoose 3y agoI suggest myself. I wouldn't mind a few hours of the CEO salary and the line on my CV.
- phkahler 3y agoOh, can we get in line for this? I might even try to make an improvement in my hours.
- deleted 3y ago[deleted]
- AnimalMuppet 3y agoA lot of us wish we had businesses that were half the failure that Google is.
- qeternity 3y agoI’d settle for 0.1%
- 3y ago
- happytiger 3y agoCan’t wait to see the patent system break under AI. It’s time for patents to go away or at least lower the timeframes of protection so we don’t stifle the innovation waves that are inevitably coming. Whole areas will be off limits and dominated by a few companies — the patent system is a system for the olden times. The patent system unfairly enforced incumbent advantage and needs to be reformed.
- thewataccount 3y agoI'm very curious to see if patents/copyright/IP just become "washable" with AI making them near useless. Given the recent determinations, it almost sounds like since it won't be human made, it might not be protectable? Not to mention the exceptions generally made for modifications to the IP to improve it.
- qeternity 3y agoPeople are going to work around this. Midjourney generates an image, but then I change the opacity 0.01% and so it becomes copyrightable (or whatever the smallest amount of human modification is required).
- thewataccount 3y ago> but then I change the opacity 0.01% and so it becomes copyrightable AFAIK Courts generally have determined it must be "transformative" for fair use, and for patents - but like a substantive change. You can't just add a pixel in a corner and call it good. https://www.justia.com/intellectual-property/patents/types-of-patents/improvement-and-new-use-patents/ https://www.justia.com/intellectual-property/patents/types-o... Slightly different (was for getting a patent for ai work) - https://cdn.arstechnica.net/wp-content/uploads/2023/02/AI-COPYRIGHT-decision.pdf https://cdn.arstechnica.net/wp-content/uploads/2023/02/AI-CO...
- orhmeh09 3y agoDo you mean like, "Here is an algorithm for attention-based sequence transduction neural networks. Come up with an algorithm that is as similar as possible and produces the same outputs, but is unencumbered by patents"?
- deleted 3y ago[deleted]
- lordvivek 3y ago[flagged]
- refulgentis 3y agoMeaningless spam? Why?
- lordvivek 3y agoits an actual explanation of the patent. but as if you were an alchemist. I thought it was funny.
- refulgentis 3y agoIt’s cool, I got -2’d and youre still up, I misread the room. Just hope it’s not the future here, no reason why other people shouldn’t post ones as a wizard, orc, crypto ceo…
- andrewchambers 3y agoIts getting down voted because if people want that sort of thing they can generate it themselves.
- refulgentis 3y agoThere's a sort of circular logic there: the alchemist one is fine because people wanted it, the other ones aren't fine because people don't want it. I prefer an HN that ends up skipping "explain [ARTICLE] as $X" altogether unless $X is on-topic
- boppo1 3y agoHonestly this kind of stuff is already boring. It's an extremely simplified explanation, and overall a pretty low effort post. Don't get me wrong, ChatGPT is amazing, and I had it write me some recipes in Plato's voice. But now the novelty has worn off and this is essentially throwaway content. It's not contributing anything to the discussion.
- tejus_vignesh 3y agoCan someone explain how this will affect any open-source or transformers-based model?
- mirekrusin 3y agoThe party is over.
- karmasimida 3y agoIt means if Google sued (not in all cases), they can block your service from using Transformer model.
- sebzim4500 3y agoLooking over the patent, I think they would really struggle to enforce this against someone using a decoder only transformer. Which is basically everyone at this point.
- ttul 3y agoYes, but patent US11556786B2 would perhaps give them that ability.
- j-pb 3y agoffs, that patent never should have made it in the european patent office. It's obviously math. This whole "computer implemented invention" workaround is a complete sham. To think that the EU wastes billions annually on this broken institution, while completely failing to properly fund startups is simply infuriating.
- eikaramba 3y agoEU? what are you talking about? The patent is registered in USA. the only country missing from the list as far as i see is actually the EU. Or did you mean USA instead of EU?
- j-pb 3y agoThe patent is applied (and granted) with multiple patent offices. The European (EU) patent office is abbreviated as EP. See: https://patents.google.com/patent/EP3542316B1/en https://patents.google.com/patent/EP3542316B1/en
- deleted 3y ago[deleted]
- karmasimida 3y agoI don't find this too be very controversial, Google sponsored the invention of Transformer, they deserve to enjoy the benefits for a period of time.
- amelius 3y agoBut first they got everybody addicted to it ... including OpenAI.
- dudeinjapan 3y agoAttention is all you need. And a gigantic army of lawyers. But mostly attention.
- jahewson 3y agoAttorneys are all you need.
- ianbutler 3y agoThis does not mean much for non encoder-decoder versions. GPT and the like are decoder only and wouldn't fall under this patent. You can look at their supplied diagrams and general summary to confirm.
- mosfets 3y agoIs Google actually suing or plan to sue OpenAI or Microsoft for patent infringement?
- ttul 3y agoAs the lawyer behind the one-click patent once personally explained to me, it's the claims that matter most. And the first claim matters more than all of the other sub-claims. This patent specifically covers ONLY transformers in which there is an encoder and a decoder. Claim 1 of the patent contains the following: "...the sequence transduction neural network comprising: an encoder neural network configured to receive the input sequence and generate a respective encoded representation of each of the network inputs in the input sequence... and a decoder neural network configured to receive the encoded representations and generate the output sequence." Claims 29 and 30 (the only other independent claims) also specify an encoder and a decoder. So long as your transformer network does not make use of an encoder in combination with a decoder, this patent does not apply to you.
- ttul 3y agoOf course, in patent US11556786B2, Google finally mopped up the decoder-only idea. https://patents.google.com/patent/US11556786B2/en https://patents.google.com/patent/US11556786B2/en "1. A method of generating an output sequence comprising a plurality of output tokens from an input sequence comprising a plurality of input tokens, the method comprising, at each of a plurality of generation time steps: generating a combined sequence for the generation time step that includes the input sequence followed by the output tokens that have already been generated as of the generation time step; processing the combined sequence using a self-attention decoder neural network, wherein the self-attention decoder neural network comprises a plurality of neural network layers that include a plurality of masked self-attention neural network layers, and wherein the self-attention decoder neural network is configured to process the combined sequence through the plurality of neural network layers to generate a time step output that defines a score distribution over a set of possible output tokens; and selecting, using the time step output, an output token from the set of possible output tokens as the next output token in the output sequence."
- amelius 3y agoIf you prune your transformer based NN after training, does that still violate the patent?
- ai_ja_nai 3y agoI don't get this: the Transformer dates back to 2017, the patent is from 2018. This should be invalid as prior art by default.