18 ms·
Apple’s new Transformer-powered predictive text model
- flakiness 3y agoFor text input support, being boring feels like a feature not a bug. For writing on mobile, you'll be writing a lot of boring short messages. That's said, it'd be cool if it can precondition with the app name - You'll probably write something more interesting on Pages vs on iMessage.
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- callalex 3y agoI was surprised to see the direct references to GPT2 by name.
- ShamelessC 3y agoIt's not an official Apple piece or anything.
- Angostura 3y agoI think they mean surprising to see gpt directly referenced in file names.
- furyofantares 3y agoI don't see that anywhere.
- rrsp 3y ago'Most of the layers within each decoder block have names like gpt2_transformer_layer_3d'
- ilaksh 3y ago[flagged]
- TheRoque 3y agoI know it's infuriating but that's how marketing works, and since they are a successful company, idk why they would stop using it. Just like some people say "RTX" instead of RayTracing now, it's quite a success for the company if it managed to replace a technical term in the mind of most consumers
- deprecative 3y ago[flagged]
- dchest 3y agoWhat? Transformer model is a standard industry term for a particular machine learning model architecture: https://en.wikipedia.org/wiki/Transformer_(machine_learning_model) https://en.wikipedia.org/wiki/Transformer_(machine_learning_... At 34M parameters, it's not very large. A LLM can be a transformer model, but also it can have a different architecture (e.g. RNN, such as RWKV). Apple has been using RNNs for many things, I believe it's the first time they're shipping an on-device transformer model.
- bobbylarrybobby 3y agoThis is the one case where the technical term and Apple’s marketing actually do happen to overlap! These are transformer models. Apple didn't invent them, but their description of them is technically accurate. “LLM” is the dumbed-down term for the masses, describing only the “what” but not the “how”.
- codeflo 3y agoAnd interesting test. I didn’t see a mention of the temperature setting used. Temperature controls the probability to pick a token that isn’t the top prediction, which leads to more creative/less robotic results. For actual input prediction, you probably want the temperature to be zero. But even a model as good as GPT-3 becomes very boring and repetitive with those settings.
- jackcook 3y agoI used greedy sampling (temperature 0) for all of them. Since I didn't have access to logits/probabilities for Apple's model, I wasn't able to do anything else in a way that would be fair.
- smpanaro 3y agoThere is an input for temperature in the CPU model. If you can find and hook the call to predict you can probably see what is being passed. Interestingly the Neural Engine version of the model does not take a temperature input, but it does output the raw embeddings in addition to the logits.
- tpowell 3y agoIs it possible for it to improve based on experience/iMessage history or is it locked in? I’d love to opt-in to training it on my previous convos…
- ComputerGuru 3y agoIt already does and I hate it. If a person you’re speaking with misspells a word, it’ll happy suggest^w forcibly autocorrect a similar word to that misspelling if it was recent enough.
- mcintyre1994 3y agoDo they do that across conversations, eg if I misspell something while texting you does it recommend my misspelling when you’re talking to other people?
- pjerem 3y agoYes
- petesergeant 3y agoYes. Ever since I went to the Maldives it wants to capitalize "male" and I have to work quite hard to get it to not correct "Yes" to "Yea", and "Thailand" to "Thailnd". Hate it.
- layer8 3y agoIn Settings there is Reset Keyboard Dictionary to revert to the default dictionary.
- petesergeant 3y agoThank you!!
- 3y ago
- blindriver 3y agoI don't want better predictive text. I need better autocorrect. Something happened about 6 years ago where the quality of the autocorrect fell off the roof, and it's been absolutely terrible since then. I spend too much of my time fighting with either mispelling, flipping to the wrong word even though I spelled the word properly, etc. It has made typing on my iPhone an unpleasant experience and I need it to change. That's just about the only thing better between the iPhone and Android for me.
- tasogare 3y ago[dead]
- dchest 3y agoIt's powered by the same thing. They even mention "Even more accurate autocorrect" in iOS 17 on the product page.
- pentae 3y agoSame with voice to text translation. It's absolutely, utterly, worthless. I end up having to go and edit half the message. Apple can't even get basic things right
- kaba0 3y agoWith ios 17 it also greatly improved (at least the input, mechanism itself).
- jamiedumont 3y agoI broke my right (dominant) collarbone on Wednesday so I’ve entered the realm of the temporarily disabled user. So far I prefer iOS because the predictive text above the keyboard (in iOS 16 too) and in particular the speech to text feature has been so good! The only egregious error that I have encountered has been confusion around the phrase “into” when I mean “in two”. An understandable mistake.
- messe 3y ago
- chaxor 3y agoUnilmCtrl seems to imply some dependence on Sochers CTRL models, but I wouldn't be surprised if it's just another name collision. Would be interesting if it were, as that would likely mean they have been working on releasing this for a few more years than the "chat-gpt craze". I am a bit biased, but I always have more respect if someone has worked in the NLP field for at least ~8 years, rather than jumping on after Sparrow and RLHF.
- Toutouxc 3y agoMeanwhile my language and dozens of others with hundreds of millions of speakers still don’t get the predictions at all, don’t support multi-language typing and don’t support swipe typing. Typing on a $1000 iPhone in one of these languages is still a 2007 experience.
- post-it 3y agoMy biggest multilingual gripe on iOS is that highlighting text in, say, Czech and pressing Translate on the popup just tells you to fuck off. Like, sure, don't support the language, but at least give me a Google Translate link then.
- PartiallyTyped 3y agoSame for Danish :(
- chongli 3y agoDid you enable the Czech-English dictionary under Settings -> General -> Dictionary? You can also add Czech as a language under Settings -> General -> Language & Region -> Add Language. This will let you switch your keyboard to Czech when typing and enable the language on websites and stuff. You can also switch the whole phone to Czech if you like, just by moving Czech to the top of the list of preferred languages.
- ComputerGuru 3y agoJust give me the option of not having autocorrect change the previous word when I am typing a new one. I don’t move on to the next word unless the current one was correct or I have already corrected it. Why does it assume I’m stupid? It even does this when I’ve picked the completion for the prior word or swiped to type and then made manual corrections. It’s painfully obvious that it shouldn’t be touched again.
- jeffbee 3y agoAnd then deleting BOTH words with a single backspace. This is its most irritating behavior.
- LeoPanthera 3y agoSettings > General > Keyboard > Delete Slide-to-Type by Word = Off
- willis936 3y agoThat changes more behavior than what GP wants.
- deleted 3y ago[deleted]
- kristopolous 3y agoApple has assumed its users are hapless morons for many years. Most people seem to prefer it this way.
- 01851159997 3y ago[flagged]
- 01851159997 3y ago[flagged]
- dkga 3y agoBeyond fixing iPhone’s autocorrect as widely mentioned below, I wonder if in future versions Apple will end up switching to phi-1.5 or other models that are much smaller but trained on higher quality data. Would also be cool if they trained their own copilot for Xcode, given their obviously enturmes code base in Swift and Obj-C.
- dchest 3y agoI'd guess they already trained it on high quality (and highly censored) data. It's GPT-2-like architecture, but not GPT-2 weights. phi-1.5 is 1.3 BILLION, this one is a lot smaller at 34 million.
- fragmede 3y agoWhat's "high quality" refer to here? The amount of txt spk I use on my iphone keyboard, vs words that are in the dictionary, is heavily skewed in the direction of short words that a corpus consisting of English text is likely to omit.
- basisword 3y agoYou use text speak on an iPhone? How? I find it impossible due to autocorrect. Either it will correct with the full word or it will correct with a wrong word making it more efficient to just type in full.
- toxik 3y agoThe example output reads exactly like the existing output, I have had it get caught in exactly that cycle.
- dchest 3y agoEven large language models with billions of parameters get caught in the cycle. You don't usually see it exposed to users because there are sampling tricks applied, such as repetition/frequency penalty.
- cubefox 3y agoSuch penalties would only exist for fine-tuned models. Base models have only the temperature setting. As the example at the end shows, even GPT-2 seems "smarter" than the Apple model, probably because of the number of parameters.
- dchest 3y agoThere's no such thing as "base models have only the temperature setting". Models do not have sampling settings (temperature, repetition penalty, etc), the sampling code does, which obviously you can use on any model. For example, here's a function from llama.cpp that applies repetition penalty: https://github.com/ggerganov/llama.cpp/blob/master/llama.cpp#L4893-L4951 https://github.com/ggerganov/llama.cpp/blob/master/llama.cpp... Here's the one from transformers: https://github.com/huggingface/transformers/blob/0a55d9f7376f72ad3ff296d4249840021b03bcc4/src/transformers/generation/logits_process.py#L270 https://github.com/huggingface/transformers/blob/0a55d9f7376... To summarize how they work: you keep some number of previously generated tokens, and once you get logits that you want to sample a new token from, you find the logits for existing tokens and multiply them by a penalty, thus lowering the probability of the corresponding tokens.
- cubefox 3y agoI don't think such penalties were applied to GPT-2 or even GPT-3, yet they weren't repetitive like that.
- danjc 3y agoInline predictive text is an anti-feature that disrupts your flow of thought. I've tried but cannot understand how anyone could actually want this. Edit: I was referring to writing prose where you're making creative decisions rather than code which is closer to a technical document. In code things are different. Traditional intellisense is usually just filtering a small set of possibilities. For example, auto completion of a reference to a variable. This is different to code prediction where for example, a variable name will be suggested. I can see value in code prediciton ala Copilot but personally don't use it.
- dchest 3y agoI loved it in OpenOffice Writer in early 2000s: type a couple of letters and press Enter to complete. Same with autocomplete when programming.
- deleted 3y ago[deleted]
- Roark66 3y agoMy favourite input app on android does real time autocorrect by displaying 3 most likely predictions on a bar above the keyboard. So it is not really inline. With programming IDEs when I can I configure them to display suggestions below the text I type, but if there is no such setting I don't find inline autocomplete bothersome at all. (as long as it displays it's prediction in dark shade of grey or another color sufficiently different from the text I typed, also there has to be a special key to accept the suggestion, like tab, no "enter accept")
- IndySun 3y agoApple's current autocorrect is skewed towards verbal, not typo, errors. Apple wants your voice. Implementing this deliberately worse change also gave Apple a path to 'improve autocorrect'. Comments are correct, T9 was based on the keyboard and common mistyped keys, simple, and effective.
- deleted 3y ago[deleted]
- IndySun 3y agoDon't take my word for it, as you all blindly have not. Take the word 'lie', often mistyped as 'loe'. This is because the o is next to the i. After accidentally typing 'loe', look at what iOS offers to replace it. low, lot, Joe. Soundalikes. This is because iOS autocorrect is skewered towards spoken vocabulary and not commonly mistyped words in english on a typical 1874 Sholes & Glidden qwerty layout. T9 worked because of this distinction. And iOS does not work as consistently because of this distinction. My 'feeling' is that Apple were very much aware of this from the iOS outset, and not wanting to use T9, offered this poorer auto-correct, and are now riding over the hill shouting 'we've solved it'. This may not be so for other languages.
- Zinu 3y agoThe example at the end made me wonder if Apple's model is actually better than GPT2 for text prediction. It generated garbage, but all that garbage made somewhat sense in the context of only the word "Today". Whereas GPT2 hallucinated random stuff about the US government. A text prediction model should predict what the user wanted to type, so if you evaluate the models based on that, GPT2 actually performed horribly, since the user showed zero intent in talking about the US.
- stavros 3y agoIt seems obvious to me that it's not, because if you asked a human to guess what comes after "today" in a text, they'd never say "probably some gibberish about a day a day".
- ricardobeat 3y agoGarbage in, garbage out? The preceding text is gibberish, so the prediction will be worse. Presumably they also only show completions with a much higher confidence threshold.
- coldtea 3y agoMaybe: "Today was fine. Since I've retired, I'm taking my life a day a day". Or maybe I wanted to express myself in the timeless words of the poets: "A day, a day of glory! A day that ends our woe! A day that tells of triumph. Against our vanquished foe!" "Rose is a rose is a rose is a rose. Loveliness extreme. Extra gaiters. Loveliness extreme." "A-well now, everybody's heard about the bird, everybody's heard about the bird, About the bird, the bird, bird bird bird, Haven't you heard about the bird? Don't you know that the bird's the word?"
- cubefox 3y agoThe example at the end sounds just like the predictions you get from normal phone keyboards in the last couple of years, which presumably don't use a modern GPT-style language model. A bit disappointing.
- 3y ago
- demarq 3y agoI've been arguing this is the way AI should be deployed. Rather than trying to sell ai as an end to end solution, just let it do the small part that it can reliably do. It's cost effective for the host, and valuable for the user. win win engineering!
- deleted 3y ago[deleted]
- apozem 3y agoI mean, I get why all these startups are trying to sell AI as a panacea. It’s an exciting technology and someone has to figure out its limits. That said, as a user, it is nice to see LLMs used in a small, discrete, undeniably useful way like this. No flashy promises, nothing new for me to learn- it’s just autocorrect, but better.
- tempaccount420 3y agoIt makes a lot more economical sense to cloud-host big models like GPT-4 and optimize them for the hardware they run on. But for small models, sure, let the user run them locally, eliminating network latency.
- keyle 3y agoIn typical Apple fashion, when everyone's going bigger and stronger, they're going in a different direction... With optimising to the smallest model that can run all day without draining your battery. I love that they're almost never first to market but they find a way to distil value than others don't. It's the almond milk of technology.
- jaimex2 3y agoIn typical Apple fanboy fashion they are oblivious to what Google has been doing for over two years now.
- kkielhofner 3y agoApple put neural in their silicon a full four years before Google did: https://www.apple.com/newsroom/2017/09/the-future-is-here-iphone-x/ https://www.apple.com/newsroom/2017/09/the-future-is-here-ip... https://blog.research.google/2021/11/improved-on-device-ml-on-pixel-6-with.html https://blog.research.google/2021/11/improved-on-device-ml-o... Apple has just been more methodical to the rest of the ecosystem - essentially waiting to understand use-cases before fully embracing it across their ecosystem from Apple Silicon in Mac with neural and `device=mps`, CoreML, and now more-or-less full force with their ML studio now in Xcode. Many apps didn't wait for this and Snapchat, as one example, has been successfully taking advantage of Apple Neural since the early early days. There are many reasons Apple has been the world's most valuable company for over a decade. One certainly is that they have a prescient ability to predict where the market is going, even if they have to use their ever-building position to drag things that way. As demonstrated by the facts and references I provided above, not to mention the iPhone essentially re-defining what a smartphone is way back when Blackberry was the current market leader.
- flangola7 3y agoHow does Snapchat use it?
- kkielhofner 3y ago
- ricardobeat 3y agoI don’t think you can make a fair comparison like this. The examples at the end are essentially praising GPT-2 for hallucinating - is it better when it’s suggesting completely irrelevant text to your sentences? Apple’s approach can’t generate full sentences on its own, but that’s not the goal anyway.
- jackcook 3y agoWow, it was quite the surprise to wake up to seeing this post near the top of HN! I wrote the post, happy to answer questions if anyone is wondering about any details.
- dkga 3y agoWell, that was pretty cool. Plus, I got to know about the way processes communicate via xpc and that opened a whole new rabbit whole!
- QuadrupleA 3y agoEveryone's asking tech details and "how", but I wonder about the "why". Do we want LLMs to always write for us, or whisper in our ear what to say? By design LLMs tend toward the most commonplace, mainstream ideas and ways of saying things. They're not much for originality or human idiosyncracy. Are we engineering a bland world full of pablum?
- tempaccount420 3y agoI don't think people write much original thought on phones anyway.
- tmpX7dMeXU 3y agoYeah. I can’t wait to see what your parents said about desktop computers.
- dartos 3y agoMost sort messages people send on Apple devices aren’t meant to be original, just to quickly convey some information.
- elicash 3y ago"Bing Chat's" implementation already allows you to select more creative generation of text. It's just a radio button option. There are also different technical solutions for the LLM to select which word to generate that allow either for more interesting, or more predictable, words. This isn't to say a human element doesn't have a ton to offer! Just to say that we aren't necessarily engineering a bland world of pablum, either.
- QuadrupleA 3y agoThat's just temperature, which evens out the random probability a little of the N most probable next words. It's still vastly favoring the N most common ones based on the training corpus, and will have a hard time producing uncommon ones. E.g. try asking an LLM to name a real, non-famous person. The internet and it's training corpus is full of regular people, but you won't have much luck - they're statistically too uncommon to remember.
- smpanaro 3y agoI've been looking at these files too and have another data point for unilm.bundle being the new text prediction. If you take an iOS simulator, turn off "Settings > General > Keyboard > Predictive", reboot it and then watch the console logs as you turn that switch back on, you'll see the "kbd" process load the models out of that bundle.
- barathr 3y agoWe might ask ourselves at some point: if the point of simple text prediction is to speed up text entry, and not generate large amounts of text from scratch, then that's a sign the input interfaces are the bottleneck. If we had a way to get text from our brains to the computer faster, we wouldn't need this kind of prediction.
- xp84 3y agoI want to know if this will be used to improve all the places where Apple devices attempt to interpret what you might mean to type, including the swipe keyboard. I've been suffering for years with their terrible, unusable swipe typing. You can't even get it to swipe "I love you" because it always prioritizes "your" over you, regardless of the context. I've even experimented extremely slowly and taken screen recordings to ensure the swipes are accurate. There are a couple of other completely astonishing common words that it 100% of the time gets wrong. I guess GPT3 would probably tax the battery, but on the other hand, I'd like to have the freedom to try it, because it would let me finish tasks much easier if text entry wasn't like fighting with an insane overconfident toddler. Honestly, I don't know what happened. Text entry with iPhones 10 years ago was far less infuriating.
- pmcf 3y agoYou have to be ducking kidding. Works great for me.
- tornato7 3y agoI love you I love you I love you I love you Seems to work fine. Though I typically use Gboard because the swipe typing is much better. I find the most restrictive thing about Apple's autocorrect and speech to text to be the limited vocabulary. Once you start using any industry terms it completely fails.
- gorenb 3y agowhy are you using a <p> tag
- xp84 3y agoEspecially to whomever downvoted me: there are at least 64 of us with this same bug.[1] — and that’s just the ones who bothered to waste time on Apple’s official discussion board. I reset my keyboard dictionary again on iOS 17, still exact same reproducible bug. It’s been at least 2 years. [1]. https://discussions.apple.com/thread/253315276 https://discussions.apple.com/thread/253315276
- r0m4n0 3y agoiPhone’s auto complete and form fills drive me absolutely mad. It’s honestly just basic stuff that should have just worked a decade ago. I’m not even begging them for LLMs or some fancy ML, just learn basic words I type out all the time. For example my full email address? how about my last name? This seems like basic stuff
- ezfe 3y agoMy iPhone has always suggested those things...
- r0m4n0 3y agoSometimes on obvious fields like an email input on a web page (not all though). Not in this comment box for example
- callalex 3y agoThat doesn’t match my experience at all. I agree with you that it should “just work” and you shouldn’t have to think about it, but do you have a contact for yourself saved, and have you set it as your “me” contact in Settings>Contacts>My Info?
- r0m4n0 3y agoYes I do. But also, my wife has a different name, do I have to type out her full name every time? There is no stored learning that I have experienced whatsoever on normal typing. The little contact auto fill that you both are referring to is extremely limited, the form field literally needs to say “email” and doesn’t work out of context. As I said, did you try to type your email out in this comment response box?
- idk1 3y agoDoes anyone know if the regular spell checker has been improved. I'm terrible at spelling and I find Apple's to be the worst, for example in the link I've shown what happens when I type "nessisary" [1] (lastest iOS 16) which is a word I often get wrong and have to Google. I feel as though this is a very clear and obvious typeo that should be picked up. If anyone is wondering what the word I'm trying to spell is, it's - necessary. Google picked this up first time. 1 - https://i.imgur.com/jcfbMLn.jpeg https://i.imgur.com/jcfbMLn.jpeg
- jingles_dev 3y ago+1
- catiopatio 3y agoI can only hope so. Apple’s spell checker is terrible. My word is “bureaucrat”. No matter how many times I look it up, I can never remember how to spell it. I always try some variation of “beurocrat”, find that spell-check is of no use whatsoever, paste it into Google, and get the proper spelling instantly.