4 ms·
So it's not measuring output tokens/s, just how long it takes to start generating tokens. Seems we'll have to wait for independent benchmarks to get useful numb
by fulafel 8mo ago
So it's not measuring output tokens/s, just how long it takes to start generating tokens. Seems we'll have to wait for independent benchmarks to get useful numbers.
- dotancohen 8mo agoFor many workflows involving real time human interaction, such as voice assistant, this is the most important metric. Very few tasks are as sensitive to quality, once a certain response quality threshold has been achieved, as is the software planning and writing tasks that most HN readers are likely familiar.
- raw_anon_1111 8mo agoThe way that voice assistants work even in the age of LLMs are: Voice —> Speech to Text -> LLM to determine intent -> JSON -> API call -> response -> LLM -> text to speech. TTFT is irrelevant, you have to process everything through the pipeline before you can generate a response. A fast model is more important than a good model Source: I do this kind of stuff for call centers. Yes I know modern LLMs don’t go through the voice -> text -> LLM -> text -> voice anymore. But that only works when you don’t have to call external sources
- garbageman 8mo agoI'm curious, what does the 'determine intent' mean in this case?
- raw_anon_1111 8mo agoAn “intent” is something that a person wants to do - set a timer, get directions, etc. A “slot” is the variable part of an intent. For instance “I want directions to 555 MockingBird Lane”. Would trigger a Directions intent that required where you are coming from and where you are going. Of course in that case it would assume your location. Back in the pre LLM days and the way that Siri still works, someone had to manually list all of the different “utterances” that should trigger the intent - “Take me to {x}”,”I want to go to {x}” in every supported language and then had to have follow up phrases if someone just said something like “I need directions” to ask them something like “Where are you trying to go”. Now you can do that with an LLM and some prompting and the LLM will keep going back and forth until all of the slots are filled and then tell it to create a JSON response when it has all of the information your API needs and you call your API. This us what a prompt would look like to use a book a flight tool. https://chatgpt.com/share/69a7d19f-494c-8010-8e9e-4e450f0bf05f https://chatgpt.com/share/69a7d19f-494c-8010-8e9e-4e450f0bf0... You also get the benefit of this works in any language not just English.
- germinalphrase 8mo agoCan you recommend any good resources that discuss structure and performance improvement of these types of systems?
- raw_anon_1111 8mo agoUnfortunately, I don’t know of any. Using LLMs for voice assistants is relatively new at scale that’s the difference between Alexa and Alexa+ and Gemini powered Google Assistant and what Apple has been trying to do with Siri for two years. It’s really just using LLMs for tool calling. It is just call centers were mostly built before the age of LLMs and companies are slow to update
- germinalphrase 8mo agoUnderstood. This overlaps with a side project where I’m getting acceptable (but not polished) results, so trying to do some digging about optimizations. Thanks!
- raw_anon_1111 8mo agoOne of my niches is Amazon Connect - the AWS version of Amazon’s internal call center. It uses Amazon Lex for voice to text. Amazon Lex is still the same old intent based system I mentioned. If it doesn’t find an intent, it goes to the “FallbackIntent” and you can get the text transcription from there and feed it into a Lambda and from the Lambda call a Bedrock hosted LLM. I have found that Nova Lite is the fastest LLM. It’s much faster than Anthropic or any of the other hosted ones.
- Art9681 8mo agoIt's going to be faster no matter what. My M3 MAX prints tokens faster than I can read for the new MoE models. It's the prompt processing that kills it when the context grows beyond a threshold which is easy to do in the modern agentic loops.
- fulafel 8mo agoIf your computer was faster at it, you could run more capable models at the same token rate.
- oofbey 8mo agoToken/s is entirely determined by memory bandwidth. TTFT is compute bound.
- fulafel 8mo agoThis is broadly correct for currently favoured software, but in computer science optimization problems you can usually trade off compute for memory and vice versa. For example just now from the front page: https://news.ycombinator.com/item?id=47242637 https://news.ycombinator.com/item?id=47242637 "Speculative Speculative Decoding" Or this: https://openreview.net/forum?id=960Ny6IjEr https://openreview.net/forum?id=960Ny6IjEr "Low-Rank Compression of Language Models Via Differentiable Rank Selection"
- saagarjha 8mo agoNone of these really change the fundamental shape of the problem.
- oofbey 8mo agoGood point on speculative decoding techniques. I'd forgotten about them, and they're good. Would love to see some of these get into llama.cpp and friends, but it does require somebody to come up with a distilled draft model. But low rank compression isn't trading off compute for memory - it's just compressing the model. And critically, that's lossy compression. That's primarily a trade-off of quality for speed/size, with a little bit of added compute. Same goals as quantization. If there was some compute-intensive lossless compression of parameters, lots of people would be happy. But those floating point values look a lot like gaussian noise, making them extremely difficult to compress.