5 ms·

My question is what changes about LLM use cases when you’re getting 1000 tok/s? Models in silicon might dramatically change how we think about them.

RussianCow 2mo ago
That likely isn't as relevant for on-device iPhone usage as it is for Real Work™. I won't notice the difference between 50tps and 1000tps when asking Siri a question.
- spijdar 2mo ago
  I don't know. As others have said, the Taalas chip wasn't small, or particularly low power, so it's hard to "imagine" what that tech in an cell phone chip might look like. But if the basic premise of "good enough LLM at insane throughput" holds, I think it could qualitatively change local uses of LLMs. At a certain speed point, you're able to move from request -> response to a cascade of tool calling and "subagents", which could allow a small model to be much more useful, if provided with a lot of local data and tool calls. That said, this is assuming you could stuff a "good enough" model into a phone with Taalas-like technology. The Taalas tech demo was an 8B parameter model and required hundreds of watts (IIRC) to run. The efficiency was good given the speed (as I understand), but it's not clear at all that the approach scales small enough to be a sensible coprocessor on an iPhone or whatever.
  - intrasight 2mo ago
    Box that plugs into my desktop would be fine. Or perhaps in SSF form factor.
  - manmal 2mo ago
    > At a certain speed point, you're able to move from request -> response to a cascade of tool calling and "subagents" That also needs server class hardware though. A phone won’t happily service the insane amount of IO, compute, and network that this cascade would require.
- retatop 2mo ago
  But wouldn't higher tps allow for more reasoning or other hidden processes, potententially making a smarter model?
  - dabbz 2mo ago
    This is my thought as well. Models have to be intentional about which tokens they burn because there's a real lag time. If you can just fork out 10 different reasoning sessions at once with no regard for token waste/lag, you can compensate a smaller model with just doing more at once with it. No idea if this is reasonably true though.
    - manmal 2mo ago
      I think that only works if you have checkpoints where all that reasoning can be checked against reality. Otherwise you get an army of armchair experts. LLMs are hilariously bad at home improvement advice btw, where reasoning alone won’t get you far.
- nvme0n1p1 2mo ago
  That order of magnitude could be the difference between "the users wants me to open the notes app, let's open it" and "I've scanned all your notes before you could blink and found what you're looking for".
- p1esk 2mo ago
  If Siri is using a 3T model in high reasoning mode to answer your question you will.
in_a_society 2mo ago
The best way I can explain it is that it's the same feeling when I upgraded from 56k dialup to cable broadband.
HDBaseT 2mo ago
In the case on on-device/self-hosted LLMs. You ask your agent to implement xyz feature 10 times and use a model to compare the outputs and combine the best results. Raw intelligence becomes slightly less important when you can iterate and improve automatically. You can still claim it was "one shot" even when 30 different implementations were made then combined.
- manmal 2mo ago
  Problem is, there exists no judge model that will really pick the same winner that you would.
QuiDortDine 2mo ago
Did you use chatjimmy? It's somewhat terrifying to use when you think of the potential results with a better model. Ok, real life example: I now spend most of my time, as a developer, waiting for the agent to do its thing (after careful prompting, I'm also thinking about work stuff, don't worry I'm not useless). What if it gave back the same excellent results, but instantaneously? Why, then, I certainly would become the bottleneck. So, quite possibly, my last work task would be to plug this agent directly into the ticket system where the domain experts input their feature requests. Maybe we still need 1 developer out of 100, to coordinate releases and all that (ok, say 1 out of 10). But that's not taking things far enough: why do we need these domain experts at all? Our pitch is clear, and all software-enabled, though it took years to develop. We can just have the clients express their concerns to the AI, directly or indirectly. Have multiple lighting-fast agents with different roles (refactoring agent, new features agent, debugger agent, domain expert agent, etc.). So we fire everyone, maybe keep 1 product owner / devops to keep the trolls out. The cost is still probably 100 times less than it used to be (beyond the initial cost of acquisition of the magic machine or whatever). But one of these clients, surely, will realize that these 10 years of manual and slowly-automated development can now be emulated in very, very little time. Why not just, say, take screenshots of the entire app and feed them into the magic machine? Why, this way, they could have the service for a tenth of the yearly cost, forever! And then the economy implodes. I'm not saying it's THE most likely version of things, I'm saying that at a certain level, quantity (or rather, speed) is a quality all its own. And this new quality might change the world. Let's hope it's for the better!
- wsintra2022 2mo ago
  I think this reads like Ray Kurzwheil (sorry not able to spell that off top of my head, that bloke who wrote that book about the future) .. But yeah very dystopian and totally realistic. Not if but when..
  - QuiDortDine 2mo ago
    I LOVE Kurzwheil! Thank you for the compliment, I'm very far from having his writing skills. But yes sci-fi is looking more and more like, well, sci.
LarsDu88 2mo ago
Massive economic simulations with thousands if not millions of agents to front run the global economy and stock market. Fully interactive realtime NPCs in videogames at scale. Recommender systems that simulate individual consumers. Crazy shit
- manmal 2mo ago
  About your first example, isn’t the butterfly effect preventing this from being useful? One agent in your simulation decides to sell, and starts an avalanche, that won’t happen in reality?
  - bostik 2mo ago
    When you run tens of thousands of simulations for complex economic models, you actually do want to see the extreme outliers too. I can't recall who said it, but in finance the interconnected incentives make so-called Black Swan events much more likely and frequent than models or theories can comfortably account for. In a way... when it's finance, they should be maybe called Gray'ish Swans?
  - LarsDu88 2mo ago
    Run the sim many times... faster than it can run on actual humans and compute a probability density for specific events. Better yet use it to dimulate counterfactual phenomena like market manipulations ypu intend to enact...
TeMPOraL 2mo ago
It's pretty clear if you see what's happening on current phones. Autocorrect that works. Reply suggestions that almost work, just need to be tad more accurate (probably more of a data access issue than model) and a tad faster to look completely seamless. Screenshots with automated text detection and OCR and automatic interpretation (different suggested actions for when something on the picture looks like a web link, phone number, postal address, e-mail, or QR code, or an event poster). That's just a fraction of things I saw showing up on my Samsung phone over the last 6 months. For over a year now, you could get a much better autocorrect and spell/grammar check, and a translator all in one, if you just pasted your text to a frontier model and asked it to check for errors or translate into target language. Now imagine being able to go through a round of such checks in a 1/100 of a second. You could have this running every keystroke, and suddenly the inline autocorrect/checks would not suck anymore. Auto-linkifying that can correct for typos and doesn't need careful regex tuning because it understands from context what is meant to be a link or not. That's just one of many obvious things possible once you get local models running fast enough. Tip of an iceberg, and the first step to imagining all the other potential uses is to let go of the two mistaken beliefs people hold on to: 1. That LLMs are about written language. They're not; ever since "multimodal models" became a thing, tokenization extended to visual and audio space, and now textual and visual and aural inputs are all just regular, first-class tokens. 2. That chatting with the models is the only optimal way for end users to interact with AI. That's just artificially limiting yourself to the space of chat-based UI.
- Godsend69 2mo ago
  [dead]
- spacer-dinos 2mo ago
  [flagged]

RussianCow 2mo ago

That likely isn't as relevant for on-device iPhone usage as it is for Real Work™. I won't notice the difference between 50tps and 1000tps when asking Siri a question.

spijdar 2mo ago
I don't know. As others have said, the Taalas chip wasn't small, or particularly low power, so it's hard to "imagine" what that tech in an cell phone chip might look like. But if the basic premise of "good enough LLM at insane throughput" holds, I think it could qualitatively change local uses of LLMs. At a certain speed point, you're able to move from request -> response to a cascade of tool calling and "subagents", which could allow a small model to be much more useful, if provided with a lot of local data and tool calls. That said, this is assuming you could stuff a "good enough" model into a phone with Taalas-like technology. The Taalas tech demo was an 8B parameter model and required hundreds of watts (IIRC) to run. The efficiency was good given the speed (as I understand), but it's not clear at all that the approach scales small enough to be a sensible coprocessor on an iPhone or whatever.
- intrasight 2mo ago
  Box that plugs into my desktop would be fine. Or perhaps in SSF form factor.
- manmal 2mo ago
  > At a certain speed point, you're able to move from request -> response to a cascade of tool calling and "subagents" That also needs server class hardware though. A phone won’t happily service the insane amount of IO, compute, and network that this cascade would require.
retatop 2mo ago
But wouldn't higher tps allow for more reasoning or other hidden processes, potententially making a smarter model?
- dabbz 2mo ago
  This is my thought as well. Models have to be intentional about which tokens they burn because there's a real lag time. If you can just fork out 10 different reasoning sessions at once with no regard for token waste/lag, you can compensate a smaller model with just doing more at once with it. No idea if this is reasonably true though.
  - manmal 2mo ago
    I think that only works if you have checkpoints where all that reasoning can be checked against reality. Otherwise you get an army of armchair experts. LLMs are hilariously bad at home improvement advice btw, where reasoning alone won’t get you far.
nvme0n1p1 2mo ago
That order of magnitude could be the difference between "the users wants me to open the notes app, let's open it" and "I've scanned all your notes before you could blink and found what you're looking for".
p1esk 2mo ago
If Siri is using a 3T model in high reasoning mode to answer your question you will.

in_a_society 2mo ago

The best way I can explain it is that it's the same feeling when I upgraded from 56k dialup to cable broadband.

HDBaseT 2mo ago

In the case on on-device/self-hosted LLMs. You ask your agent to implement xyz feature 10 times and use a model to compare the outputs and combine the best results. Raw intelligence becomes slightly less important when you can iterate and improve automatically. You can still claim it was "one shot" even when 30 different implementations were made then combined.

manmal 2mo ago
Problem is, there exists no judge model that will really pick the same winner that you would.

QuiDortDine 2mo ago

Did you use chatjimmy? It's somewhat terrifying to use when you think of the potential results with a better model. Ok, real life example: I now spend most of my time, as a developer, waiting for the agent to do its thing (after careful prompting, I'm also thinking about work stuff, don't worry I'm not useless). What if it gave back the same excellent results, but instantaneously? Why, then, I certainly would become the bottleneck. So, quite possibly, my last work task would be to plug this agent directly into the ticket system where the domain experts input their feature requests. Maybe we still need 1 developer out of 100, to coordinate releases and all that (ok, say 1 out of 10). But that's not taking things far enough: why do we need these domain experts at all? Our pitch is clear, and all software-enabled, though it took years to develop. We can just have the clients express their concerns to the AI, directly or indirectly. Have multiple lighting-fast agents with different roles (refactoring agent, new features agent, debugger agent, domain expert agent, etc.). So we fire everyone, maybe keep 1 product owner / devops to keep the trolls out. The cost is still probably 100 times less than it used to be (beyond the initial cost of acquisition of the magic machine or whatever). But one of these clients, surely, will realize that these 10 years of manual and slowly-automated development can now be emulated in very, very little time. Why not just, say, take screenshots of the entire app and feed them into the magic machine? Why, this way, they could have the service for a tenth of the yearly cost, forever! And then the economy implodes. I'm not saying it's THE most likely version of things, I'm saying that at a certain level, quantity (or rather, speed) is a quality all its own. And this new quality might change the world. Let's hope it's for the better!

wsintra2022 2mo ago
I think this reads like Ray Kurzwheil (sorry not able to spell that off top of my head, that bloke who wrote that book about the future) .. But yeah very dystopian and totally realistic. Not if but when..
- QuiDortDine 2mo ago
  I LOVE Kurzwheil! Thank you for the compliment, I'm very far from having his writing skills. But yes sci-fi is looking more and more like, well, sci.

LarsDu88 2mo ago

Massive economic simulations with thousands if not millions of agents to front run the global economy and stock market. Fully interactive realtime NPCs in videogames at scale. Recommender systems that simulate individual consumers. Crazy shit

manmal 2mo ago
About your first example, isn’t the butterfly effect preventing this from being useful? One agent in your simulation decides to sell, and starts an avalanche, that won’t happen in reality?
- bostik 2mo ago
  When you run tens of thousands of simulations for complex economic models, you actually do want to see the extreme outliers too. I can't recall who said it, but in finance the interconnected incentives make so-called Black Swan events much more likely and frequent than models or theories can comfortably account for. In a way... when it's finance, they should be maybe called Gray'ish Swans?
- LarsDu88 2mo ago
  Run the sim many times... faster than it can run on actual humans and compute a probability density for specific events. Better yet use it to dimulate counterfactual phenomena like market manipulations ypu intend to enact...

TeMPOraL 2mo ago

It's pretty clear if you see what's happening on current phones. Autocorrect that works. Reply suggestions that almost work, just need to be tad more accurate (probably more of a data access issue than model) and a tad faster to look completely seamless. Screenshots with automated text detection and OCR and automatic interpretation (different suggested actions for when something on the picture looks like a web link, phone number, postal address, e-mail, or QR code, or an event poster). That's just a fraction of things I saw showing up on my Samsung phone over the last 6 months. For over a year now, you could get a much better autocorrect and spell/grammar check, and a translator all in one, if you just pasted your text to a frontier model and asked it to check for errors or translate into target language. Now imagine being able to go through a round of such checks in a 1/100 of a second. You could have this running every keystroke, and suddenly the inline autocorrect/checks would not suck anymore. Auto-linkifying that can correct for typos and doesn't need careful regex tuning because it understands from context what is meant to be a link or not. That's just one of many obvious things possible once you get local models running fast enough. Tip of an iceberg, and the first step to imagining all the other potential uses is to let go of the two mistaken beliefs people hold on to: 1. That LLMs are about written language. They're not; ever since "multimodal models" became a thing, tokenization extended to visual and audio space, and now textual and visual and aural inputs are all just regular, first-class tokens. 2. That chatting with the models is the only optimal way for end users to interact with AI. That's just artificially limiting yourself to the space of chat-based UI.

Godsend69 2mo ago
[dead]
spacer-dinos 2mo ago
[flagged]