4 ms·

I think that any workflow that requires the user to stare at the tokens being generated live is using it wrong. Delegate, don't stare! https://mikeveerman.gith

I think that any workflow that requires the user to stare at the tokens being generated live is using it wrong. Delegate, don't stare! https://mikeveerman.github.io/tokenspeed/?rate=10&mode=text https://mikeveerman.github.io/tokenspeed/?rate=10&mode=text You think of an idea that you want to have the LLM process, queue it up, and go back to what you were doing. Once you've finished reading the next article on HN about a 5 tps Xeon, your task will be complete. It's kind of like using a 3D printer: It doesn't matter if a print takes 10 hours, because when you come back in the morning it will be done. Yes, with top-tier GPU farms you can hit hundreds of tokens per second. But if the old Xeon in the closet can get useful work done at 5 tokens per second, there are lots of people and lots of use cases where a free, unlimited 5 TPS stream is worth more than paying a dollars per day to get access to a 500 TPS source.

bitpush 3mo ago
is there a good tool to manage these workloads? batch process a bunch, handle failures, retry things etc?
adastra22 3mo ago
We aren’t there yet. Not for frontier development work at least.
ctoth 3mo ago
[dead]
allknowingfrog 3mo ago
We clearly have different goals. I want an LLM to review my code, not the other way around.
- nolok 3mo ago
  It's still the same thing, you can ask it to do a full on report give explanation and details be thorough and then go do something else, another task a lunch break whatever and it will be done when you're back
  - allknowingfrog 3mo ago
    How do you maintain a flow state during a lunch break? I'm looping with Claude on a scale of minutes. While you're waiting, I'm iterating.
    - deleted 3mo ago
      [deleted]
    - vitally3643 3mo ago
      This is like comparing a hammer to a screwdriver and feeling smug because you can hammer nails faster than someone else can drive screws. These are fundamentally different tools for entirely different applications. They only look similar to people who don't understand the tools or their purpose.
      - allknowingfrog 3mo ago
        This thread started with me saying "we clearly have different goals" and then being told that I just need to hold the screwdriver differently...
    - fwip 3mo ago
      You don't "maintain flow." You eat lunch. I swear, tech culture has gotten people wanting to work for the machines, rather than the other way round.
      - ClikeX 3mo ago
        Right? Tech should make my work easier. Not have me stressing out even more. Let the machine do its work while I relax, I’ll check up on it later.
      - allknowingfrog 3mo ago
        This was a discussion about LLM usage patterns. I'm not opposed to lunch breaks. I'm opposed to being required to take the equivalent of 12 lunch breaks a day while I wait for slow responses.
- pmarreck 3mo ago
  I'm sure this exact topic has been argued hundreds of times already on HN, but I think I have a new "possibly agreeable to both sides" perspective on this after having lost man-years to retired corporate code aka "FAIAP, throwaway code" Let LLMs write the corpo code, as it will be unlikely to still be running in 5-10 years. Frontier AI is already at the point where it writes fewer bugs per LOC than humans. By a lot. Go ahead and do your bespoke coding on your side-project loves and core libraries... The stuff that will last, anyway. But if you're working for a corpo and still doing bespoke... That's... not gonna last, I'm afraid. Well, either you remaining there, or that, as it were.
  - allknowingfrog 3mo ago
    There's a whole spectrum of employment between faceless corporations and personal side projects. AI will replace humans because giant business believe they can do the same work, not because they will actually be able to. The correctness of an application is limited by your ability to understand and describe what you need. We have a word for an application specification tool so detailed it eliminates all ambiguity. It's called a "programming language". The mistakes are always in the transfer from human to machine. I still find a high-level programming language to be the best way to express my intent. Humans will make mistakes in the hand-off to AI just like they make mistakes in the hand-off to code, but at least code is deterministic.
    - pmarreck 3mo ago
      > AI will replace humans because giant business believe they can do the same work, not because they will actually be able to. This has been being claimed for at least 2 years now. Wouldn't we already be seeing disasters if this was the case? It's certainly been around long enough to cause some real damage. Instead there is a slow trickle of things that makes the news by people who didn't bother instituting a single fucking control. For example, I am completely immune to `rm -rf` style fuckups because I wrote rm-safe years ago and it is mapped to `rm` in any environment that the LLM will run in: https://github.com/pmarreck/rm_safe https://github.com/pmarreck/rm_safe In any event, it's not a replacer, it's an augmenter. Nothing will replace humans, because we are the stakeholders; it may shift them around, though. > The correctness of an application is limited by your ability to understand and describe what you need. If you write code in Lean 4 or Idris 2, you may not completely understand why it is or isn't correct, but their respective compilers will certainly prove it to you one way or the other. We already are perfectly functional with incomplete understandings. An entire generation of web developers have had successful careers without having a single clue how compilers work or how machine code works. Now is there a type of problem that sometimes comes along that DOES require deep understanding? Of course, that's when you call in the heavy artillery. Or the advanced LLM. Or both. See my point?
RussianCow 3mo ago
Once you've used a model that runs at hundreds of TPS, it's hard to go back. Everything completes so quickly that you can iterate without breaking out of flow state. My biggest gripe with slow (<50tps) LLMs is that I've lost all the mental context I built up by the time it's done, which makes it extremely difficult to explore or iterate on solutions.
- ngcc_hk 3mo ago
  In 1980s ibm has studied and said why sub-second response needed to maintain the mental flow. That time you send a whole screen unlike unix like character by character. This proves very true even when you deal with form processing. I think that we are dealing with the same issue here. Keep your mental context in your brain is critical
- hx8 3mo ago
  I'd rather have slower and better output than worse and faster output.
  - RussianCow 3mo ago
    It depends. For something high stakes or inherently complex, sure, you don't want to have to clean up the agent's mess afterwards. But for many tasks like building web UIs, the difference in output quality is going to be small enough that iteration speed will win over quality. With a fast enough model, I can iterate on the UI of a given screen 4-5 times before Opus finishes its first attempt.
- nmfisher 3mo ago
  Completely agree. Slow but smart models (Fable, Sol, GLM5.2 etc) are great, but they leave me with zero mental model of the code that's been written. Most of the time my mind wanders off and I go check social media or fire off a prompt for some other random project, it's a big productivity drain. Working with models that are super fast, but slightly dumber (like mimo-v2.5-pro-ultraspeed) is amazing, I feel like I'm still the one that's actually making every decision.
Aurornis 3mo ago
> Once you've finished reading the next article on HN about a 5 tps Xeon, your task will be complete. If I spend 10 minutes reading an article, that would only generate 3000 tokens. That’s not counting the prompt processing time. We have very different expectations for LLMs if your tasks only take a couple thousand tokens and you’re happy waiting 10 minutes for it. > Yes, with top-tier GPU farms you can hit hundreds of tokens per second My 5090 gets hundreds of tokens per second with this model. No farm needed. I’d have to double check but I think even a $1000 Intel B70 might break 100 tokens per second. > But if the old Xeon in the closet can get useful work done at 5 tokens per second, there are lots of people and lots of use cases where a free, unlimited 5 TPS stream is worth more than paying a dollars per day to get access to a 500 TPS source. If that old Xeon pulls 200W from the wall and you pay national average electricity costs, it’s going to cost $0.90 per day to run it. I would rather pay a dollar per day, get my answers 100X faster, and not have an old Xeon heating up my house.
- zoobab 3mo ago
  You could suspend it to ram, and only wake it up on request, it takes 2 seconds on my box.
  - Aurornis 3mo ago
    It’s not a cost savings relative to paying API prices even if you’re suspending it. This is an option if you must run local inference, you’re not sensitive to speed, and the budget is low. It’s not going to be cheaper than paying API prices for the model though.
bredren 3mo ago
Except often queued agentic flows must be checked in on. Or to use the comparison, 3D printers are not immune to making spaghetti all night when something goes wrong. (I’m not a 3d printing expert so maybe that is solved now) It is common for agents to just stop because overload or some API error hijinks. Or you get a TUI question that is blocking. In general you’re right though, staring at tokens from agentic is not time well spent. Some of these I’ve built custom harness around in iterm2 though.
varispeed 3mo ago
Filament snaps at 1am and then you have to run print again. 10 hours turn into many days potentially. I watch tokens to see if it goes in right direction. If model goes off the rails, then it is time to stop and adjust prompt.
dofm 3mo ago
> It's kind of like using a 3D printer: It doesn't matter if a print takes 10 hours, because when you come back in the morning it will be done. This is how I used to think about my 3D printer, but FWIW the way my actual thinking and planning works, print speed really matters. Not for the final print, but for iterative work and test parts, it is obvious that either having a fast printer helps. Having multiple slow printers also helps, but there are only so many areas of a design you can iterate on at once. At the moment my own LLM use is experimental and iterative, and I definitely favour the faster MoE models for much of what I am doing, even if I might in principle prefer to get the final work done in the slower ones.
parineum 3mo ago
> there are lots of people and lots of use cases where a free, unlimited 5 TPS stream is worth more than paying a dollars per day to get access to a 500 TPS source. That Xeon is going to be devouring power.

bitpush 3mo ago

is there a good tool to manage these workloads? batch process a bunch, handle failures, retry things etc?

adastra22 3mo ago

We aren’t there yet. Not for frontier development work at least.

ctoth 3mo ago

[dead]

allknowingfrog 3mo ago

We clearly have different goals. I want an LLM to review my code, not the other way around.

nolok 3mo ago
It's still the same thing, you can ask it to do a full on report give explanation and details be thorough and then go do something else, another task a lunch break whatever and it will be done when you're back
- allknowingfrog 3mo ago
  How do you maintain a flow state during a lunch break? I'm looping with Claude on a scale of minutes. While you're waiting, I'm iterating.
  - deleted 3mo ago
    [deleted]
  - vitally3643 3mo ago
    This is like comparing a hammer to a screwdriver and feeling smug because you can hammer nails faster than someone else can drive screws. These are fundamentally different tools for entirely different applications. They only look similar to people who don't understand the tools or their purpose.
    - allknowingfrog 3mo ago
      This thread started with me saying "we clearly have different goals" and then being told that I just need to hold the screwdriver differently...
  - fwip 3mo ago
    You don't "maintain flow." You eat lunch. I swear, tech culture has gotten people wanting to work for the machines, rather than the other way round.
    - ClikeX 3mo ago
      Right? Tech should make my work easier. Not have me stressing out even more. Let the machine do its work while I relax, I’ll check up on it later.
    - allknowingfrog 3mo ago
      This was a discussion about LLM usage patterns. I'm not opposed to lunch breaks. I'm opposed to being required to take the equivalent of 12 lunch breaks a day while I wait for slow responses.
pmarreck 3mo ago
I'm sure this exact topic has been argued hundreds of times already on HN, but I think I have a new "possibly agreeable to both sides" perspective on this after having lost man-years to retired corporate code aka "FAIAP, throwaway code" Let LLMs write the corpo code, as it will be unlikely to still be running in 5-10 years. Frontier AI is already at the point where it writes fewer bugs per LOC than humans. By a lot. Go ahead and do your bespoke coding on your side-project loves and core libraries... The stuff that will last, anyway. But if you're working for a corpo and still doing bespoke... That's... not gonna last, I'm afraid. Well, either you remaining there, or that, as it were.
- allknowingfrog 3mo ago
  There's a whole spectrum of employment between faceless corporations and personal side projects. AI will replace humans because giant business believe they can do the same work, not because they will actually be able to. The correctness of an application is limited by your ability to understand and describe what you need. We have a word for an application specification tool so detailed it eliminates all ambiguity. It's called a "programming language". The mistakes are always in the transfer from human to machine. I still find a high-level programming language to be the best way to express my intent. Humans will make mistakes in the hand-off to AI just like they make mistakes in the hand-off to code, but at least code is deterministic.
  - pmarreck 3mo ago
    > AI will replace humans because giant business believe they can do the same work, not because they will actually be able to. This has been being claimed for at least 2 years now. Wouldn't we already be seeing disasters if this was the case? It's certainly been around long enough to cause some real damage. Instead there is a slow trickle of things that makes the news by people who didn't bother instituting a single fucking control. For example, I am completely immune to `rm -rf` style fuckups because I wrote rm-safe years ago and it is mapped to `rm` in any environment that the LLM will run in: https://github.com/pmarreck/rm_safe https://github.com/pmarreck/rm_safe In any event, it's not a replacer, it's an augmenter. Nothing will replace humans, because we are the stakeholders; it may shift them around, though. > The correctness of an application is limited by your ability to understand and describe what you need. If you write code in Lean 4 or Idris 2, you may not completely understand why it is or isn't correct, but their respective compilers will certainly prove it to you one way or the other. We already are perfectly functional with incomplete understandings. An entire generation of web developers have had successful careers without having a single clue how compilers work or how machine code works. Now is there a type of problem that sometimes comes along that DOES require deep understanding? Of course, that's when you call in the heavy artillery. Or the advanced LLM. Or both. See my point?

RussianCow 3mo ago

Once you've used a model that runs at hundreds of TPS, it's hard to go back. Everything completes so quickly that you can iterate without breaking out of flow state. My biggest gripe with slow (<50tps) LLMs is that I've lost all the mental context I built up by the time it's done, which makes it extremely difficult to explore or iterate on solutions.

ngcc_hk 3mo ago
In 1980s ibm has studied and said why sub-second response needed to maintain the mental flow. That time you send a whole screen unlike unix like character by character. This proves very true even when you deal with form processing. I think that we are dealing with the same issue here. Keep your mental context in your brain is critical
hx8 3mo ago
I'd rather have slower and better output than worse and faster output.
- RussianCow 3mo ago
  It depends. For something high stakes or inherently complex, sure, you don't want to have to clean up the agent's mess afterwards. But for many tasks like building web UIs, the difference in output quality is going to be small enough that iteration speed will win over quality. With a fast enough model, I can iterate on the UI of a given screen 4-5 times before Opus finishes its first attempt.
nmfisher 3mo ago
Completely agree. Slow but smart models (Fable, Sol, GLM5.2 etc) are great, but they leave me with zero mental model of the code that's been written. Most of the time my mind wanders off and I go check social media or fire off a prompt for some other random project, it's a big productivity drain. Working with models that are super fast, but slightly dumber (like mimo-v2.5-pro-ultraspeed) is amazing, I feel like I'm still the one that's actually making every decision.

Aurornis 3mo ago

> Once you've finished reading the next article on HN about a 5 tps Xeon, your task will be complete. If I spend 10 minutes reading an article, that would only generate 3000 tokens. That’s not counting the prompt processing time. We have very different expectations for LLMs if your tasks only take a couple thousand tokens and you’re happy waiting 10 minutes for it. > Yes, with top-tier GPU farms you can hit hundreds of tokens per second My 5090 gets hundreds of tokens per second with this model. No farm needed. I’d have to double check but I think even a $1000 Intel B70 might break 100 tokens per second. > But if the old Xeon in the closet can get useful work done at 5 tokens per second, there are lots of people and lots of use cases where a free, unlimited 5 TPS stream is worth more than paying a dollars per day to get access to a 500 TPS source. If that old Xeon pulls 200W from the wall and you pay national average electricity costs, it’s going to cost $0.90 per day to run it. I would rather pay a dollar per day, get my answers 100X faster, and not have an old Xeon heating up my house.

zoobab 3mo ago
You could suspend it to ram, and only wake it up on request, it takes 2 seconds on my box.
- Aurornis 3mo ago
  It’s not a cost savings relative to paying API prices even if you’re suspending it. This is an option if you must run local inference, you’re not sensitive to speed, and the budget is low. It’s not going to be cheaper than paying API prices for the model though.

bredren 3mo ago

Except often queued agentic flows must be checked in on. Or to use the comparison, 3D printers are not immune to making spaghetti all night when something goes wrong. (I’m not a 3d printing expert so maybe that is solved now) It is common for agents to just stop because overload or some API error hijinks. Or you get a TUI question that is blocking. In general you’re right though, staring at tokens from agentic is not time well spent. Some of these I’ve built custom harness around in iterm2 though.

varispeed 3mo ago

Filament snaps at 1am and then you have to run print again. 10 hours turn into many days potentially. I watch tokens to see if it goes in right direction. If model goes off the rails, then it is time to stop and adjust prompt.

dofm 3mo ago

> It's kind of like using a 3D printer: It doesn't matter if a print takes 10 hours, because when you come back in the morning it will be done. This is how I used to think about my 3D printer, but FWIW the way my actual thinking and planning works, print speed really matters. Not for the final print, but for iterative work and test parts, it is obvious that either having a fast printer helps. Having multiple slow printers also helps, but there are only so many areas of a design you can iterate on at once. At the moment my own LLM use is experimental and iterative, and I definitely favour the faster MoE models for much of what I am doing, even if I might in principle prefer to get the final work done in the slower ones.

parineum 3mo ago

> there are lots of people and lots of use cases where a free, unlimited 5 TPS stream is worth more than paying a dollars per day to get access to a 500 TPS source. That Xeon is going to be devouring power.