storystarling

10 ms·

91.
by storystarling 9mo ago
Curious how the token economics compare here to a standard agent loop. It seems like if you're using the LLM as a JIT to optimize the algorithm as the game evolves, the context accumulation would get expensive fast even with Flash pric
92.
by storystarling 9mo ago
The danger is that we bridge that gap with backend complexity. I spent weeks over-engineering a chain of evaluators and retries to get reliable outputs from cheaper models, thinking I was optimizing margins. Then a smarter model dropped tha
93.
by storystarling 9mo ago
I tried this recently and found the token overhead makes it prohibitive for any non-trivial schema. Dumping the full introspection result into the context window gets expensive fast and seems to increase hallucination rates compared to just
94.
by storystarling 9mo ago
Nice writeup. I've been looking at this for a print-on-demand project and found that physical ink bleed changes the constraints quite a bit compared to e-paper. In my experience error diffusion often gets muddy due to dot gain, whereas
95.
by storystarling 9mo ago
I suspect this is primarily a unit economics problem. To get context beyond the diff you really need the full repository or a robust AST, but the token costs to load that state for every PR make the margins impossible right now.
96.
by storystarling 9mo ago
The unit economics seem tough at that price for a 1T parameter model. Even with MoE sparsity you are still VRAM bound just keeping the weights resident, which is a much higher baseline cost than serving a smaller model like Haiku.
97.
by storystarling 9mo ago
The practical limit is the latency and inference cost. A full planning and validation loop burns a lot of tokens, and waiting for that cycle breaks flow compared to just writing the code.
98.
by storystarling 9mo ago
You are likely just seeing the host topology. Even if the container reports 56 cores, the actual compute is almost certainly throttled via cgroups to keep the unit economics viable. I would be surprised if you can sustain more than a fracti
99.
by storystarling 9mo ago
Fuzzing handles the logic verification, but I'd be more worried about the architectural debt of mapping GC patterns to Rust. You often end up with a mess of Arc/Mutex wrappers and cloning just to satisfy the borrow checker, which
100.
by storystarling 9mo ago
That assumes inference efficiency is static, which isn't really the case. Between aggressive quantization, speculative decoding, and better batching strategies, the cost per token can vary wildly on the exact same hardware. I suspect t
101.
by storystarling 9mo ago
Matches my experience with print-on-demand workflows. I tried using vision models to validate things like ICC profiles and total ink density, but they usually just hallucinate that the file is compliant. I ended up giving the agent access t
102.
by storystarling 9mo ago
complaining customers...
103.
by storystarling 9mo ago
nope - Temporal is indeed a good option for that.
104.
by storystarling 9mo ago
yes, though this is highly subjective - it 'feels' like that to me as well (comapred to Gemini 3, GPT 5.2, Opus 4.5).
105.
by storystarling 9mo ago
It seems solvable if you treat it as an architecture problem. I've been using LangGraph to force the model to extract and cite evidence before it runs any scoring logic. That creates an audit trail based on the flow rather than just op
106.
by storystarling 9mo ago
It is worth noting the distinction between display acceleration and compute support here. While the desktop rendering is impressive, for local AI or LLM inference the Linux stack on M-series is still significantly behind Metal/MPS on m
107.
by storystarling 9mo ago
Nice work on the pure Go implementation. I built a similar pipeline for generating audio versions of articles and the pricing trade-offs are tough at scale. ElevenLabs is obviously the quality winner but their per-character pricing eats up
108.
by storystarling 9mo ago
I've had better results separating these concerns rather than trying to stuff it all into one prompt. In my backend workflows (using LangGraph), I treat generation and critique as distinct agents where the second one explicitly challen
109.
by storystarling 9mo ago
I suspect Kagi is running a multi-step agentic loop there, maybe something like a LangGraph implementation that iterates on the context. That burns a lot of inference tokens and adds latency, which works for a paid subscription but probably
110.
by storystarling 9mo ago
I had to go down this path for a print-on-demand book project. If you need high-DPI assets for physical print the commercial static map APIs are prohibitively expensive or restrict usage rights for resale. Self-hosting was basically the onl
111.
by storystarling 9mo ago
Curious if the advanced indexing strategies cover vector workloads. I've been trying to consolidate my GenAI stack to avoid maintaining a separate vector database, but performance has been a bit of a bottleneck. Would be great to hear
112.
by storystarling 9mo ago
It's also the only medium where semantic reasoning and indexing at scale makes financial sense. I can run RAG over millions of text rows in Postgres for pennies, but the compute costs to process and embed video content are still prohib
113.
by storystarling 9mo ago
It feels like France is actually leading on the infrastructure side of things right now. With Mistral and Hugging Face both in Paris, the open source AI ecosystem is pretty heavily concentrated there.
114.
by storystarling 9mo ago
The trade-off is replacing deterministic code with probabilistic agents. I've found you still need a heavy orchestration layer—I'm using LangGraph and Celery—just to handle retries and ensure idempotency when the agent inevitably
115.
by storystarling 9mo ago
That might be true for chat, but for building actual products the prompting feels like a solved problem compared to the plumbing. I've spent way more time lately fighting with Celery queues and LangGraph state management than on the pr
116.
by storystarling 9mo ago
I ended up overengineering a LangGraph workflow to handle this. It forces the LLM to generate and pass its own tests in a sandbox before I even see the PR. The API costs are significantly higher because of the retry loops, but it filters ou
117.
by storystarling 9mo ago
Curious how the token optimization presets balance context window costs against the depth of call graph analysis. I've found that aggressively pruning context to save on input tokens often degrades reasoning quality pretty quickly when
118.
by storystarling 9mo ago
I suspect the curse of dimensionality makes this an optimization dead end. You hit prohibitive latency limits on retrieval long before the resolution approaches human perception. Even with current dimensions, the trade-off between index siz
119.
by storystarling 9mo ago
I came to the same conclusion and ended up wiring a custom pipeline with LangGraph and Celery. The markup on the SaaS options is hard to justify given the raw API costs. The main benefit of rolling it yourself seems to be the control over c
120.
by storystarling 9mo ago
I'm curious if the 16ms budget is actually driven by the token stream here. Even with the fastest providers right now, you are looking at maybe 100-150 tokens per second. That is orders of magnitude below the 30MB/s terminal parsi

More ›