5 ms·
I still find it amazing how LLM translation capabilities are an almost accidental feature. Yet they still managed to leapfrog decades of research and billions o
by spacebanana7 3y ago
I still find it amazing how LLM translation capabilities are an almost accidental feature. Yet they still managed to leapfrog decades of research and billions of investment dollars in traditional machine translation.
- dartos 3y agoTranslation is the reason transformer models exist. Every other use case is basically an accidental feature
- jampekka 3y agoThe original Transformer, indeed for translation, was an encoder-decoder architecture. Most current LLMs, like Claude 3, are "decoder only". So in a sense the decoder-only translation is kinda accidental feature too.
- _giorgio_ 3y agoSo anything except encoder decoder translation is accidental? Crazy way to define research.
- jampekka 3y agoResearch, especially in academia, isn't usually that interested in the single concrete task that is under study. Translation is just one case where you need to find some good functions to map sequences to other sequences. The "Attention Is All You Need" paper frames the problem and contribution as: "The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely." Translation is "just an experiment" for the general architecture and they study other tasks in the paper too. In a sense, all applications of basic research are "accidental".
- herculity275 3y agoDidn't "Attention Is All you Need" bill transformers primarily as a translation model?
- Alifatisk 3y agoYes, it was primarly for translation. I don’t know how OP come to the conclusion that it was accidental
- magoghm 3y agoI guess that what OP meant is that LLMs are not specifically trained to do translation, they learn to do it as a side effect.
- hackmaxim 3y agoYes, but the translation model in the “Attention is All You Need” was trained on parallel sentences (input: English sentence, output: French). LLMs such as Claude are not trained on any labeled data. Yet, they still surpass the special-case models.
- londons_explore 3y agoThey surpass because there is faaaaaaar more language data generally than there is data that's paired english-french.
- authorfly 3y agoAnd the key takeaway of the T5 paper was also how "instructions" could vary massively. But the main instructions were two types - summarise x, and translate y to z. Translation in many ways was the root of seeing that multiple tasks/instructions could fit a model and not just in the context of multiple training loss methods, like NSP vs Gap filling(a modus of training difference, not actual task itself). And the special tokens for BERT etc trace their origins back to enabling the task above and positional embeddings(and encodings). Which largely trace their origin to translation work.
- 3y ago
- endisneigh 3y agoOrders of magnitude more energy necessary though.
- frabcus 3y agoThe GPT 3.5 API is cheaper than Google Translate. And in our testing (over a year ago) was better for translation. So I assume in that case the energy use is less?
- jsheard 3y agoIsn't OpenAI still operating at a loss? You can't infer much from an API bring cheaper if it's being subsidised.
- CuriouslyC 3y agoGPT4 on the web is at a loss, $20/mo doesn't cover it unless someone's a rare user. GPT4 through API is not at a loss, and I doubt GPT3.5 is at a loss either.
- endisneigh 3y agoThese assertions are based off of what?
- hnfong 3y agoGiven that you can buy a at least a couple hours worth of GPU compute at $20, it's hard to believe the average user spends that much time (specifically waiting for the response) on GPT-4 on the web every month... Not to mention that they probably have optimizations that batch together queries at the same time and make things more efficient at scale...
- dopp0 3y agothe assertions were based on a prompt sent to chatgpt: "are you more efficient than google translate, cost-wise?"
- 3y ago
- Al-Khwarizmi 3y agoAnd still, there are people saying that LLMs are useless. I find that even more amazing.
- SirMaster 3y agoThey probably mean useless to themselves, not useless to everyone.
- EasyMark 3y agoI'm still not sold on LLM basically becoming the next doctors and scientists like some claim, but they can be useful. Since I'm not an LLM PhD or Wallstreet "hacker" I'm not going to pretend I can predict where it's going to go. It's a useful tool under my belt currently. Hopefully I don't get replaced
- racional 3y agoThe charge generally made against LLMs is not that they're useless. Rather that they are in many settings overhyped and overprescribed.
- _giorgio_ 3y agoTransformers have been built for translation since the very first iteration. 2017 paper.
- EasyMark 3y agoBut don't LLM also have decades of research and billions of dollars of investment to get where they are if you consider the history of AI and machine learning?
- raxxorraxor 3y agoThe compute power needed for translations is vastly different though.