3 ms·

I've been using Qwen3.6 35B A3B, and with reasoning turned on, I'd say 2/3 (give or take) of the tokens for a response are thinking tokens. Which at 70+ tps loc

I've been using Qwen3.6 35B A3B, and with reasoning turned on, I'd say 2/3 (give or take) of the tokens for a response are thinking tokens. Which at 70+ tps locally, that isn't that awful. I run an 80k context across 4-10 "agents" for my solo TTRPG, where Qwen is the GM, each NPC at a location, the director, and the narrator. Each turn is about 45-60 seconds to generate all of the various responses. The GM and director have reasoning on, and the NPCs/Location/Narrator do not. It's a fairly good "engine" for that. I'm not sure how a denser Qwen would do here regarding speed.

jakswa 2mo ago
I like the tabletop RPG use case, and wanted to say: If your hardware likes it you should check out Gemma 4 for creative DMing use case. I found it to be much better at holding the plotlines and being creative on gaming turns. My experimental case was an audio-only Zork and Gemma 12B and even E4B were pretty good!
toyg 2mo ago
Is there some sort of dedicated tool for this type of setup, or did you hand-craft it ?
- seanmcdirmid 2mo ago
  Not parent, but I use Goose for my non-handcrafted Qwen use cases, I’m also working on handcrafting as well. Goose was the only harness that didnt bloat context too much with system prompts (like openclaw) and I could get reasonable web search working with Qwen.
- jermaustin1 2mo ago
  Somewhat hand rolled, somewhat claude coded. Back in 2023 I started my own C# LLM library for doing tool calls and structured output, and over the years it has morphed bigger and bigger, and that is the backbone of almost all of my LLM-based projects. I've never released it, but its easy to understand, and simple to add your own tools: [AIDescription("Get current weather for a location")] static string GetWeather( [AIDescription("The city name")] string city, [AIDescription("The country name")] string country, [AIDescription("Temperature unit", ["C", "F"])] string unit = "C") { // make some API call to a weather API and return a string to the LLM return $"The weather in {city}, {country} is 22°{unit} and sunny"; } var chat = client.StartConversation("You are a helpful assistant with access to weather data."); var response = await chat.SendAsync<string>("What's the weather in London?", GetWeather); I'm sure plenty of better libraries exist for this now, but in 2023, I don't think any existed in the dotnet ecosystem. I've never released it though, because I've never "finished" it.
  - rhplus 2mo ago
    It looks like you’re describing something like an MCP server and a client model. If you’re in the C# ecosystem you could consider converting your APIs to MCP format tools using the MCP SDK. https://devblogs.microsoft.com/dotnet/build-a-model-context-protocol-mcp-server-in-csharp/ https://devblogs.microsoft.com/dotnet/build-a-model-context-... And then leveraging Microsoft Agent Framework for the client and orchestration side of things: https://learn.microsoft.com/en-us/agent-framework/ https://learn.microsoft.com/en-us/agent-framework/
lostmsu 2mo ago
Are you running inference in parallel? 70 tps seems low for parallel execution.
- jermaustin1 2mo ago
  It is on a single 3090, and that seems to be where it averages out. I'll get 85tps on turn 0, but then it settles down to low 70s within a few turns, but holds steady at that. My issue currently is KV Cache, because I can't keep enough parallel caches running (4 is where I'm at), so TTFT (is that the initialism?) can be long when I have a particularly large scene (basically more than 2 NPCs). But my harness does let me offload to any OpenAI compatible endpoint, I just prefer local cuz free.
crorella 2mo ago
This sound very interesting, do you have any resource I could look at? Me and my son did a very rudimentary (compared to yours) setup to play Paranoia, but this is at another level.
- jermaustin1 2mo ago
  I'll go over my repo, and see if it is hiding any API keys and maybe make it public. The issue I have is it relies on a nuget package that also isn't live (its in my local nuget feed). I'm not sure what all is needed to make that work for people.
makr17 2mo ago
I'm working on something similar. My biggest annoyance is that the overly-helpful LLM was making every die roll succeed. I ended up building some tooling around rolling dice. Also some tooling around character stats and inventory management, so those don't get lost in context compression.
sanxiago 2mo ago
I was also playing on something similar a few months ago: https://github.com/sanxiago/kastsaga https://github.com/sanxiago/kastsaga Anyone else working on this?

jakswa 2mo ago

I like the tabletop RPG use case, and wanted to say: If your hardware likes it you should check out Gemma 4 for creative DMing use case. I found it to be much better at holding the plotlines and being creative on gaming turns. My experimental case was an audio-only Zork and Gemma 12B and even E4B were pretty good!

toyg 2mo ago

Is there some sort of dedicated tool for this type of setup, or did you hand-craft it ?

seanmcdirmid 2mo ago
Not parent, but I use Goose for my non-handcrafted Qwen use cases, I’m also working on handcrafting as well. Goose was the only harness that didnt bloat context too much with system prompts (like openclaw) and I could get reasonable web search working with Qwen.
jermaustin1 2mo ago
Somewhat hand rolled, somewhat claude coded. Back in 2023 I started my own C# LLM library for doing tool calls and structured output, and over the years it has morphed bigger and bigger, and that is the backbone of almost all of my LLM-based projects. I've never released it, but its easy to understand, and simple to add your own tools: [AIDescription("Get current weather for a location")] static string GetWeather( [AIDescription("The city name")] string city, [AIDescription("The country name")] string country, [AIDescription("Temperature unit", ["C", "F"])] string unit = "C") { // make some API call to a weather API and return a string to the LLM return $"The weather in {city}, {country} is 22°{unit} and sunny"; } var chat = client.StartConversation("You are a helpful assistant with access to weather data."); var response = await chat.SendAsync<string>("What's the weather in London?", GetWeather); I'm sure plenty of better libraries exist for this now, but in 2023, I don't think any existed in the dotnet ecosystem. I've never released it though, because I've never "finished" it.
- rhplus 2mo ago
  It looks like you’re describing something like an MCP server and a client model. If you’re in the C# ecosystem you could consider converting your APIs to MCP format tools using the MCP SDK. https://devblogs.microsoft.com/dotnet/build-a-model-context-protocol-mcp-server-in-csharp/ https://devblogs.microsoft.com/dotnet/build-a-model-context-... And then leveraging Microsoft Agent Framework for the client and orchestration side of things: https://learn.microsoft.com/en-us/agent-framework/ https://learn.microsoft.com/en-us/agent-framework/

lostmsu 2mo ago

Are you running inference in parallel? 70 tps seems low for parallel execution.

jermaustin1 2mo ago
It is on a single 3090, and that seems to be where it averages out. I'll get 85tps on turn 0, but then it settles down to low 70s within a few turns, but holds steady at that. My issue currently is KV Cache, because I can't keep enough parallel caches running (4 is where I'm at), so TTFT (is that the initialism?) can be long when I have a particularly large scene (basically more than 2 NPCs). But my harness does let me offload to any OpenAI compatible endpoint, I just prefer local cuz free.

crorella 2mo ago

This sound very interesting, do you have any resource I could look at? Me and my son did a very rudimentary (compared to yours) setup to play Paranoia, but this is at another level.

jermaustin1 2mo ago
I'll go over my repo, and see if it is hiding any API keys and maybe make it public. The issue I have is it relies on a nuget package that also isn't live (its in my local nuget feed). I'm not sure what all is needed to make that work for people.

makr17 2mo ago

I'm working on something similar. My biggest annoyance is that the overly-helpful LLM was making every die roll succeed. I ended up building some tooling around rolling dice. Also some tooling around character stats and inventory management, so those don't get lost in context compression.

sanxiago 2mo ago

I was also playing on something similar a few months ago: https://github.com/sanxiago/kastsaga https://github.com/sanxiago/kastsaga Anyone else working on this?