19 ms·
20B-parameter Alexa model sets new marks in few-shot learning
- zaroth 4y ago20 billion parameters and the UI for voice is still cringe level terrible. Or is it just me and I’ve turned into a get-off-my-lawn curmudgeon when it comes to audio interfaces? > Find a reservation far from my work location in eight hours for 8 people at Union Auto Company. Said absolutely no one ever, right? I guess if this is what it’s trained on it’s no wonder.
- creeble 4y agoI can't parse that sentence.
- NegativeLatency 4y agoYeah many options: - (Find a reservation) (far from my work location in eight hours) (for 8 people at Union Auto Company). - (Find a reservation far from my work location) (in eight hours) (for 8 people at Union Auto Company). - (Find a reservation far from my work location in eight hours for 8 people) (at Union Auto Company). lots more options but it is confusing if you start to dig in
- nl 4y agoHow interesting! (BTW I don't see where this sentence is coming from, and I looked in the papers too) Here's how Spacy sees it: [1] So in that case it thinks the command is "in 8 hours find a reservation that is far from my work location and is for 8 people at Union Auto Company". This parse is almost certainly incorrect. [1] https://explosion.ai/demos/displacy?text=Find%20a%20reservation%20far%20from%20my%20work%20location%20in%20eight%20hours%20for%208%20people%20at%20Union%20Auto%20Company.&model=en_core_web_sm&cpu=1&cph=1 https://explosion.ai/demos/displacy?text=Find%20a%20reservat...
- NegativeLatency 4y agoFrom one of the parents: https://news.ycombinator.com/item?id=32323847 https://news.ycombinator.com/item?id=32323847
- nl 4y agoYeah but where did they get that sentence?
- zaroth 4y agoIt's in the figure - in the yellow bubble under "Few input examples in English". Interestingly on iPhone I could copy/paste the text, but on desktop Chrome I can't, and Ctrl-F doesn't work on it. Vector text in a .webp file?
- nl 4y agoThanks! It's not vector text (webp doesn't do that). You are seeing the magic text recognition that iOS can do on images.
- zaroth 4y agoWow, that is a very pleasing and seamless UX to just treat it like actual embedded text. Not often am I actually pleasantly surprised by something new in UI. Apple Pay on the Web might have been my last ‘Woah’ moment actually.
- nl 4y agoYeah it's pretty nice isn't it. I was confused because I'd done Cmd-F search on every page - maybe Firefox (which I use) needs to incorporate a similar feature. Hmm.
- jonathankoren 4y agoThis is the type of thing you learn to say when you're dealing with a slot filling algorithm that allows for overspecification. By putting it in one utterance, you avoid it saying coming back and saying "Where do you want a reservation?" "When do you want your reservations?" "How many people are in your party?"
- buscoquadnary 4y agoThat's actually an interesting thought I've been having, will audio interfaces eventually train humans to speak in a certain way?
- saalweachter 4y agoA boss of mine was adamant that it took humans a year or so to learn to reliably communicate in a new workgroup, for one person to say a thing and have everyone else understand what was meant.
- lazide 4y agoOh we try - it still doesn’t work.
- ctoth 4y ago> We follow Hoffmann et al. (2022) and pre-train the model for roughly 1 Trillion tokens (longer than the 300B token updates of GPT-3). If I'm understanding the discussion of the Chinchilla paper correctly[0] then this should offer a significantly better boost than increasing the number of parameters would have. Also really cool that they make the model easy(ish) to run and play with! [0]: https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla-s-wild-implications https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla...
- rafaelero 4y agoNot sure how much scaling laws apply here, since this is a seq-to-seq model instead of a autoregressive causal model. It's interesting to see AlexaTM performing better than GPT-3 on SuperGLUE and SQuADv2, but it fails on Chain of Thought prompt, which is a bummer. So, is it because it's a different model or because it is positively leveraging multilingual tokens? I wish they compared this architecture to a classic GPT family model.
- transcriptase 4y agoQuestion for those familiar with the backend of things like Alexa, Google Home, Siri: At what point can we say things like “turn off the bedroom light in 5 minutes” or “stop music and turn off all the lights”? Even something like “keep the lights on” in a motion sensor system is impossible it seems. Because to me they feel like low hanging fruit, and yet despite all the advances in machine learning and these systems being around for the better part of a decade, anything but the simplest single-task no-modifier commands result in a “sorry… I didn’t understand that” or completely unpredictable results. Is there something inherently difficult about these types of queries?
- runnerup 4y agoAlso why doesn’t Siri work at all in Honda Civics and Honda HR-V’s when connected to car play and driving on the highway with no radio playing? Google assistant works fine, for the most part anyways.
- jeffbee 4y agoWeird. Works fine in an Insight, which is in all respects a Civic Hybrid.
- Closi 4y agoThese are all subtly different problems I think, but in general most of these architectures currently assume there is a single intent. > stop music and turn off all the lights This is probably the easiest of the bunch because you are asking it to perform two distinct actions. > turn off the bedroom light in 5 minutes This is much more complex, because you are asking the application to setup some sort of workflow - after it understands what you want it to do, it then has to work out how to execute that, which will be utilising the device API's / services. This is a simple example, but there are lots of permutations of different actions here, for example you might want to say "turn off the sound system once this song finishes playing" which assumes that the assistant has the capability to then understand you want it to create a task specifically waiting for the trigger of a particular song finishing playing, and that it has the ability to setup that trigger. > "keep the lights on” in a motion sensor system Now this is where the orchestration gets tricky - The assistant has to: * Work out that the lights are being affected by a motion sensor system, which is likely outside it's own platform. * Work out that your intent is that you want the assistant to override that. * Understand how to connect to the platform in order to control it. * Work out what parameter it is supposed to alter to achieve this task. * Override the existing users settings, and presumably reinstate the settings after some portion of time.
- xxpor 4y agoAs a complete outsider: has ML research just become a phallus measuring contest to see who can stuff the most parameters into a model? In other words, who can acquire the most Nvidia cards? The model size seems to always be the headline in stuff I see on HN.
- visarga 4y agoThis is a small model keeping up with the big guys. 20B parameters might fit in 2 beefy GPUs, that's a bargain compared to GPT-3.
- cuuupid 4y ago+1, also this is a teacher model. The implications are huge here as AWS will likely spin this into an offering like they did with their other AI products. Building a model downstream of GPT-3 is difficult and usually yields suboptimal results; however 20b is small enough that it would be easy to finetune this on a smaller dataset for a specific task. You could then distill that model and end up with something that’s a fraction of the size (6b parameters for example, just under 1/3, would fit on commercial GPUs like 3090s). There are some interesting examples of this with smaller models like BERT/BART or PEGASUS in Huggingface Transformer’s seq2seq distillation examples.
- pjfin123 4y agoYeah this is the opposite, they did impressively well with fewer parameters. In general larger models and more data has been an effective strategy for getting better performance but getting the right ratio is also important: https://www.deepmind.com/publications/an-empirical-analysis-of-compute-optimal-large-language-model-training https://www.deepmind.com/publications/an-empirical-analysis-...
- kajecounterhack 4y agoAs others pointed out, this paper tries to do more with fewer params. But you've identified a trend does actually describe large language models for the past few years (they've been getting bigger, and bigger has been better). Like microprocessors have the famous tick/tock cycle (https://en.wikipedia.org/wiki/Tick%E2%80%93tock_model https://en.wikipedia.org/wiki/Tick%E2%80%93tock_model), I think models might see be seeing something similar emerge naturally (make models bigger --> make models better (shrink) --> make models bigger again --> make models better (shink again)). Also, most of this LLM stuff is probably not trained on NVidia hardware -- at scale it's probably cost prohibitive if not also hard to set up. Google's TPUs, MSFT/Amazon's equivalent custom hardware, or other specialized accelerators are more economical overall.
- pjfin123 4y agoThe most notable thing about this model is that they use fewer parameters (20 billion) than many of the other LLM which makes it less resource intensive to train and easier to run. They also use an encoder-decoder architecture, which is common for machine translation, unlike most large language models which are decoder-only. https://community.libretranslate.com/t/alexatm-a-20b-multilingual-language-model-by-amazon/288 https://community.libretranslate.com/t/alexatm-a-20b-multili...
- mrlonglong 4y agoShould we worry if it achieves sentience? Any reason why we shouldn't?
- nomel 4y agoI would assume consciousness/sentience requires significant feedback loops, so the "thought process" can "keep going". I don't think any of these models have real feedback loops.
- bioemerl 4y agoThey do during training, but not when you use them
- Planktonne 4y agoNo more than you should worry about your phone, your toaster, or a boulder suddenly becoming self-aware. Extremely theoretically, a consciousness could spontaneously form at any point [1]. In practice, there is no reason to worry about this - it's not a likely event. There is nothing about this model (or any other one in existence) that increases the likelihood of it achieving sentience when compared to anything else ever. [1] https://en.wikipedia.org/wiki/Boltzmann_brain https://en.wikipedia.org/wiki/Boltzmann_brain
- MichaelCollins 4y agoAssuming the premise of this achieving sentience... > Should we worry if it achieves sentience? No, because it's easy to kill.
- taeric 4y agoSo should we worry if it learns to propagate itself? :)
- MichaelCollins 4y agoI don't really buy the whole "it teaches itself to hack itself out" idea. I think a sentient AI would be able to introspect and manipulate its operating environment if it was specifically designed to have that capability. Another scenario might be that somebody installs the sentient AI inside an drone submarine with nuclear armed SLBMs. You can contrive scenarios where the AI is given capabilities that make it hard to kill. But baring such scenarios, if the program scares you then just SIGKILL it.
- daoist_shaman 4y agoThis blows my mind. How is it even possible to validate a model that incorporates 20B parameters? How do you even test something this complex and non-deterministic? I assume some kind of infallible automated tooling is used to write tests that validate this monster. I would LOVE to see what that tooling looks like.
- taylorius 4y agoHow do you define validate? These models aren't formally proven to work in all cases or anything. They're just tested on a load of data, and if it's found that they work pretty well, then they get released.
- lionkor 4y agoSo, like 90% of the non-FAANG software that we use every day
- kajecounterhack 4y agoIt _is_ deterministic (same input gives same output). You typically don't "test" pairs of inputs/outputs for a model. Instead you measure its performance by defining metrics e.g. "what's the ROUGE-2 score on summarization after fine-tuning AlexaTM 20B using N examples from dataset Y" You can test some aspects of ML models, like sync testing (if you train on hardware A and run on hardware B, their results are not always the same). But generally you test the code that embeds the model, not the model itself.
- pfisherman 4y agoOk now how about having Alexa turn off the music, when I say “Alexa, turn off the music”?
- nycdotnet 4y agoHow about the wakeup word even working reliably anymore or not offering me an upsell on every other interaction.
- _virtu 4y agoAlexa is trash.
- deleted 4y ago[deleted]
- trention 4y ago>"excels other large language models" It's also good that its announcement excels at having a grammatically correct subtitle.
- conrad247 4y agohttps://www.merriam-webster.com/dictionary/excel https://www.merriam-webster.com/dictionary/excel