storystarling

5 ms·

61.
by storystarling 9mo ago
That 19 EUR figure is basically subscription arbitrage. If you ran that volume through the API with xhigh reasoning the cost would be significantly higher. It doesn't seem scalable for non-interactive agents unless you can stay on the
62.
by storystarling 9mo ago
Curious about the memory bandwidth constraints here. 20B parameters at 20fps seems like it would saturate the bandwidth of a single GPU unless you are running int4. I assume this requires an H100?
63.
by storystarling 9mo ago
Electricity is negligible but the dominant cost is the hardware depreciation itself. Also inference is typically memory bandwidth bound so you are limited by how fast you can move weights rather than raw compute efficiency.
64.
by storystarling 9mo ago
I suspect the endgame is confidential computing for distributed systems. If you are running high value workloads like LLMs in untrusted environments you need to verify integrity. Right now guaranteeing that the compute context hasn't b
65.
by storystarling 9mo ago
I've found LLMs perform surprisingly well here if you target CSG or OpenSCAD. It seems to frame the 3D modeling challenge as a logic and syntax problem rather than a spatial one, which plays to the model's strengths. You avoid the
66.
by storystarling 9mo ago
Right, but how does it know which files to pick? I'm curious if you're using a dependency graph or embeddings for that discovery step, since getting the agent to self-select the right scope is usually the main bottleneck.
67.
by storystarling 9mo ago
That explains the coherence, but I'm curious about the mechanics of the retrieval. Is it AST-based to map dependencies or are you just using vector search? I assume you still have to filter pretty aggressively to keep the token costs v
68.
by storystarling 9mo ago
I solved the port 80 issue by adding AmbientCapabilities=CAP_NET_BIND_SERVICE to the Service section of the unit file. That lets you bind privileged ports while still defining a User= line to run non-root. The lifecycle management seems sol
69.
by storystarling 9mo ago
Fair, it worked, but podman generate systemd is deprecated now. I found the generated unit files pretty brittle to maintain compared to just having a declarative config that handles the lifecycle.
70.
by storystarling 9mo ago
Quadlet actually solves this. It's the newer way to define containers for systemd and handles the rootless user case properly. I migrated my services to it recently and it's much more robust than the old generate scripts.
71.
by storystarling 9mo ago
The fine-tuning overhead is definitely a factor, but for smaller shops the hard constraint is usually inference VRAM. Running a 32B model locally or on a rented GPU is surprisingly expensive if you aren't saturating it. Even at 4-bit q
72.
by storystarling 9mo ago
How did you handle the context window for 20k lines? I assume you aren't feeding the whole codebase in every time given the API costs. I've struggled to keep agents coherent on larger projects without blowing the budget, so I&#x27
73.
by storystarling 9mo ago
I built a similar gateway for my own stack and thought it would be a quick project, but the complexity is hidden in the details. A basic proxy is simple enough, but getting accurate token counts for streaming responses turned out to be a hu
74.
by storystarling 9mo ago
I suspect this is just an LLM hallucinating generic thread-safety boilerplate. In an async serverless runtime like Workers this pattern creates blocking risks and doesn't actually solve the distributed consistency problem.
75.
by storystarling 9mo ago
Cloning gets you the raw text objects directly. If you scrape the web UI you're dealing with a lot of markup overhead that just burns compute during ingestion. For training data you usually want the structure to be as clean as possible
76.
by storystarling 9mo ago
The unit economics seem pretty rough though. You're locking up 8xH100s for the compute of ~32B active parameters. I guess memory is the bottleneck but hard to see how the margins work on that.
77.
by storystarling 9mo ago
Serverless GPU providers like Modal or RunPod are probably the closest thing. You pay for execution time rather than tokens so the unit economics are deterministic, and you don't have to manage the underlying capacity or OS. It is stil
78.
by storystarling 9mo ago
Fair point. I suspect if you priced that workload out on per-token API costs, it would be completely unviable for a bootstrapped business. The flat-rate subscription is really the only thing making it accessible right now.
79.
by storystarling 9mo ago
True, but that's still 1,500 inference cycles. Even without external API fees, the latency and compute burden seems huge. I don't see how the economics work there without significant subsidies.
80.
by storystarling 9mo ago
I guess it works well until you hit a stateful failure. My concern would be Day 2 operations—debugging a database issue or networking partition without a mental model of the underlying architecture seems pretty painful.
81.
by storystarling 9mo ago
I ran into this exact trade-off when migrating a legacy inventory system to event sourcing. The refinement mapping concept is elegant, but implementing it as a live SQL view over the event table killed our read performance once the volume p
82.
by storystarling 9mo ago
I suspect the noise is largely an artifact of cost optimization. Most tools restrict the context to just the diff to save on input tokens, rather than traversing the full dependency graph. Without seeing the actual definitions or call sites
83.
by storystarling 9mo ago
uv is great for resolution, but it seems like it doesn't really address the build complexity for heavy native dependencies. If you are doing any serious work with torch or local LLMs, you still run into issues where wheels aren't
84.
by storystarling 9mo ago
Might be worth checking if they are appending random query strings to force cache misses. Usually you can normalize the request at the edge to strip those out and protect the origin.
85.
by storystarling 9mo ago
1,500 tool calls per task sounds like a nightmare for unit economics though. I've been optimizing my own agent workflows and even a few dozen steps makes it hard to keep margins positive, so I'm not sure how this is viable for any
86.
by storystarling 9mo ago
The supervisor-worker architecture is standard for distributed systems, but I'm not sure the unit economics make sense yet. Given current latency and inference costs, that specific pattern seems significantly more expensive and slower
87.
by storystarling 9mo ago
The US CLOUD Act explicitly asserts jurisdiction over data controlled by US companies regardless of the physical server location. So relying on AWS Frankfurt doesn't actually protect the data from US warrants if the provider is a US en
88.
by storystarling 9mo ago
SQLite solves the storage layer but I suspect you run into a pretty big impedance mismatch on the graph traversals. For heavy DAG operations like history rewriting, a custom structure seems way more efficient than trying to model that relat
89.
by storystarling 9mo ago
I found that the only way to get true fixed costs is renting the GPUs and self-hosting. The unlimited API plans usually come with strict rate limits or concurrency caps that make them unusable for production traffic. You basically have to c
90.
by storystarling 9mo ago
I'm not sure I agree about the lack of a technical moat. While spinning up a basic WebRTC wrapper is trivial, the real challenge is the global distributed systems engineering required for low latency and reliability at scale. You need

More ›