4 ms·

Have been letting it spin pretty hard (~$12.50 for 2B, 50% cache hits) on my traffic simulator/distributed physics engine all day, it's found some pretty signif

Have been letting it spin pretty hard (~$12.50 for 2B, 50% cache hits) on my traffic simulator/distributed physics engine all day, it's found some pretty significant gains without introducing any new problems. I'm happy

blahyawnblah 2mo ago
Can you tell me about your engine?
- monster_truck 2mo ago
  It's ~a traffic simulator with the fidelity of a rally sim like DiRT Rally or Asseto. Full engine/drivetrain, grip modelling with tire deformation, suspension, aero, collisions, etc. I just really like Rally Racing and Car Accidents The absurd goal was to be able to simulate all of the active traffic in NYC, so 3-500,000 cars, without using any of the macro flow corner cutting that you see commercially or academically. Initially thought that one machine was not going to be enough to do this in realtime so I spun off a very big fork and built out a webtransport stack to split the effort over a local network. It was promising until I also wanted to cover the highway in thousands of giant beach balls [1] [2]. In the process of leveraging codegen to shrink the data that needed to be relayed by >10000x (packing and dynamically updating sparse continous arrays of floats), threw in a fuckton of LOD work (both spatial & temporal), statistical aggregation, and a lot of differential equation bullshit to derive LUTs. It's at the point where a base model M2 mini can handle much more than that by itself. All of the networking effort paid large dividends in cross thread coordination and lock-free data passing. Went from struggling to fit each of the sim kernels for just 32 cars @ 60hz (~11ms) to 0.03, 0.0003ms p99s with the corresponding jumps in car count (north of 32k/thread). Multiplayer works well enough, it falls apart where it should (~128 people or LLMs driving around in the same square quarter mile, and a great deal more NPCs, latency permitting) The remaining work is making it look and sound cool as fuck. Building out a physically-based audio synthesis engine that simulates the pulses of exhaust gas starting from the cylinder count/size/firing order, intake and exhaust count and size, header and exhaust configuration, and another one for the tire sounds, and another one for the collisions, and then rendering those out to wavetables so it scales and frees up the cycles for occlusion. And then getting it to visually render and control performantly in a browser tab, lots of instancing and shader work. There's also some bullshit cooking that uses the motion estimation built into GPU video compression engines for ... other purposes at stupid low latency. I figured that's what Waymo had to be doing so I let it rip It's fucking nuts, I'm having so much fun :) It will be done when it's finished [1]: https://i.imgur.com/BDQSuLv.png https://i.imgur.com/BDQSuLv.png [2]: https://i.imgur.com/CpreOWT.png https://i.imgur.com/CpreOWT.png
p1necone 2mo ago
50% cache hit is really low - in a standard agentic loop you should expect like 99%+ cache hit percentage (which should also lower that $12.50 to like a couple of $ for the same amount of tokens). If you're using a customised harness you should make sure you don't have something that's e.g. changing your system prompt on some requests or rewriting history - it can be tempting to do stuff like strip old thinking tokens or compact tool call results to reduce context size but it's a trap - you want to never change history because of how cheap cache is, even more so with deepseek because their cache hit pricing is so low compared to most other models.
- Lalabadie 2mo ago
  In my experience, that's the OpenRouter tax. Even a session that does everything right to remain sticky ends up getting moved between providers on a few requests, which bills you the full context as input every time the switch happens. I assume it's done as load balancing/latency mitigation, but it's put me off of OpenRouter for my use cases (limited use, limited need for changing models).
  - grono 2mo ago
    Why not pin to specific openrouter provider and disable fallback?
  - julianz 2mo ago
    There is only one provider for this model, so shouldn't be running into that.
  - p1necone 2mo ago
    This has not been my experience. Generally I do pin to 1 provider, or 1 provider with a couple fallbacks (especially with deepseek - most providers are 10x the cached token price compared to deepseek themselves), but even when I don't I still usually see 99%+ cache hit percentage. Specifically using pi with various ad-hoc customisations (that I was careful not to break prompt caching with).
    - 2mo ago
fooblaster 2mo ago
Can you explain how you used 12 billion tokens to do useful work?
- eru 2mo ago
  (Not the original commenter.) You can rack up quite a lot of tokens if you ask it to try out a lot of things, eg for performance investigations and trying out optimisation ideas.
  - fooblaster 2mo ago
    is there a standard pattern for this? Like spawn an agent for each technique to try?
    - lightbulbish 2mo ago
      maybe this? https://github.com/karpathy/autoresearch https://github.com/karpathy/autoresearch
    - eru 2mo ago
      Not sure. I usually tell the agent to spawn subagents at will. (And they are doing that on their own anyway.)
    - laichzeit0 2mo ago
      I regularly do a “go to DynaTrace, look at how this service gets used in production then use a profiler and see if there’s any low hanging optimisations we could make” on stuff. LLMs are really good at doing everything that was fun about software development.
      - eru 2mo ago
        Ha, I have a whole hobby work-stream going on about finding low-hanging fruit in various open source projects to turn into valuable contributions. Two premier sources: (1) look at good contributions someone already tried to make, but that got stuck in review or were otherwise abandoned. (2) look at user reported bugs and see if we can find a user reported bugs, and see if we can reproduce and fix cheaply. Both are explicitly scoped as best-effort affairs: move on, if you can't quickly make progress. Most of the work I have to do as a human is review and navigating the submission process: tokens are cheap these days, so you really need to make sure the contribution is actually worth someone's time to review.

blahyawnblah 2mo ago

Can you tell me about your engine?

monster_truck 2mo ago
It's ~a traffic simulator with the fidelity of a rally sim like DiRT Rally or Asseto. Full engine/drivetrain, grip modelling with tire deformation, suspension, aero, collisions, etc. I just really like Rally Racing and Car Accidents The absurd goal was to be able to simulate all of the active traffic in NYC, so 3-500,000 cars, without using any of the macro flow corner cutting that you see commercially or academically. Initially thought that one machine was not going to be enough to do this in realtime so I spun off a very big fork and built out a webtransport stack to split the effort over a local network. It was promising until I also wanted to cover the highway in thousands of giant beach balls [1] [2]. In the process of leveraging codegen to shrink the data that needed to be relayed by >10000x (packing and dynamically updating sparse continous arrays of floats), threw in a fuckton of LOD work (both spatial & temporal), statistical aggregation, and a lot of differential equation bullshit to derive LUTs. It's at the point where a base model M2 mini can handle much more than that by itself. All of the networking effort paid large dividends in cross thread coordination and lock-free data passing. Went from struggling to fit each of the sim kernels for just 32 cars @ 60hz (~11ms) to 0.03, 0.0003ms p99s with the corresponding jumps in car count (north of 32k/thread). Multiplayer works well enough, it falls apart where it should (~128 people or LLMs driving around in the same square quarter mile, and a great deal more NPCs, latency permitting) The remaining work is making it look and sound cool as fuck. Building out a physically-based audio synthesis engine that simulates the pulses of exhaust gas starting from the cylinder count/size/firing order, intake and exhaust count and size, header and exhaust configuration, and another one for the tire sounds, and another one for the collisions, and then rendering those out to wavetables so it scales and frees up the cycles for occlusion. And then getting it to visually render and control performantly in a browser tab, lots of instancing and shader work. There's also some bullshit cooking that uses the motion estimation built into GPU video compression engines for ... other purposes at stupid low latency. I figured that's what Waymo had to be doing so I let it rip It's fucking nuts, I'm having so much fun :) It will be done when it's finished [1]: https://i.imgur.com/BDQSuLv.png https://i.imgur.com/BDQSuLv.png [2]: https://i.imgur.com/CpreOWT.png https://i.imgur.com/CpreOWT.png

p1necone 2mo ago

50% cache hit is really low - in a standard agentic loop you should expect like 99%+ cache hit percentage (which should also lower that $12.50 to like a couple of $ for the same amount of tokens). If you're using a customised harness you should make sure you don't have something that's e.g. changing your system prompt on some requests or rewriting history - it can be tempting to do stuff like strip old thinking tokens or compact tool call results to reduce context size but it's a trap - you want to never change history because of how cheap cache is, even more so with deepseek because their cache hit pricing is so low compared to most other models.

Lalabadie 2mo ago
In my experience, that's the OpenRouter tax. Even a session that does everything right to remain sticky ends up getting moved between providers on a few requests, which bills you the full context as input every time the switch happens. I assume it's done as load balancing/latency mitigation, but it's put me off of OpenRouter for my use cases (limited use, limited need for changing models).
- grono 2mo ago
  Why not pin to specific openrouter provider and disable fallback?
- julianz 2mo ago
  There is only one provider for this model, so shouldn't be running into that.
- p1necone 2mo ago
  This has not been my experience. Generally I do pin to 1 provider, or 1 provider with a couple fallbacks (especially with deepseek - most providers are 10x the cached token price compared to deepseek themselves), but even when I don't I still usually see 99%+ cache hit percentage. Specifically using pi with various ad-hoc customisations (that I was careful not to break prompt caching with).
  - 2mo ago

fooblaster 2mo ago

Can you explain how you used 12 billion tokens to do useful work?

eru 2mo ago
(Not the original commenter.) You can rack up quite a lot of tokens if you ask it to try out a lot of things, eg for performance investigations and trying out optimisation ideas.
- fooblaster 2mo ago
  is there a standard pattern for this? Like spawn an agent for each technique to try?
  - lightbulbish 2mo ago
    maybe this? https://github.com/karpathy/autoresearch https://github.com/karpathy/autoresearch
  - eru 2mo ago
    Not sure. I usually tell the agent to spawn subagents at will. (And they are doing that on their own anyway.)
  - laichzeit0 2mo ago
    I regularly do a “go to DynaTrace, look at how this service gets used in production then use a profiler and see if there’s any low hanging optimisations we could make” on stuff. LLMs are really good at doing everything that was fun about software development.
    - eru 2mo ago
      Ha, I have a whole hobby work-stream going on about finding low-hanging fruit in various open source projects to turn into valuable contributions. Two premier sources: (1) look at good contributions someone already tried to make, but that got stuck in review or were otherwise abandoned. (2) look at user reported bugs and see if we can find a user reported bugs, and see if we can reproduce and fix cheaply. Both are explicitly scoped as best-effort affairs: move on, if you can't quickly make progress. Most of the work I have to do as a human is review and navigating the submission process: tokens are cheap these days, so you really need to make sure the contribution is actually worth someone's time to review.