storystarling

10 ms·

181.
by storystarling 9mo ago
The 5B text encoder feels disproportionate for a 2B video model. If the text portion is dominating your VRAM usage it really hurts the inference economics. Have you tried quantizing the T5? In my experience you can usually run these encoder
182.
by storystarling 9mo ago
I tried to build a similar local stack recently to save on API costs. In practice I found the hardware savings are a bit of a mirage for coding workflows. The local models hallucinate just enough that you end up spending more in lost time d
183.
by storystarling 9mo ago
The inference is cheap, but the context window costs for iteratively debugging architecture issues add up fast. Things like state management or migrations usually require feeding the whole stack back in multiple times, which blows past that
184.
by storystarling 9mo ago
I suspect the main trade-off is structured data versus text parsing. While CLIs are composable, relying on stdout is brittle for anything complex. MCP enforces a schema (types), which acts as a contract between the model and your backend. I
185.
by storystarling 9mo ago
I'd be careful sending raw audio to public APIs given the sensitive commercial info. A local pipeline with Whisper and Llama 3 is viable now and solves the privacy issue. It also keeps the long-term inference costs much lower.
186.
by storystarling 9mo ago
I've found that architecting around that reliability wall is where the margins fall apart. You end up chaining verification steps and retries to get a usable result, which multiplies inference costs until the business case just doesn&#
187.
by storystarling 9mo ago
An RTF above 1 for just 0.6B parameters suggests the bottleneck isn't the GPU, even on a 1080. The raw compute should be much faster. I'd bet it's mostly CPU overhead or an issue with the serving implementation.
188.
by storystarling 9mo ago
This seems solvable today if you treat it as an architecture problem rather than relying on the model's weights. I'm using LangGraph to force function calls to Crossref or OpenAlex for a similar workflow. As long as you keep the f