storystarling

5 ms·

31.
by storystarling 9mo ago
Isolated state is the sensible choice for embedding; fighting a GIL from the host language is miserable. I guess those shared-memory primitives are the escape hatch for when message passing serialization gets too expensive? The documentatio
32.
by storystarling 9mo ago
It is less about the raw transfer speed and more about the synchronization and kernel launch overheads. If you profile a standard inference loop with a batch size of 1 you see the GPU spending a lot of time idle waiting for the CPU to dispa
33.
by storystarling 9mo ago
I assume the use case here is mostly for backend infrastructure rather than consumer devices. You want to verify that a machine has booted a specific signed image before you release secrets like database keys to it. If you can't attest
34.
by storystarling 9mo ago
I've found they are actually quite good at semantic geometry even if they struggle with visual or pixel-based reasoning. Since this is parametric the agent just needs to understand the API and constraints rather than visualize the fina
35.
by storystarling 9mo ago
The killer app here is likely LLM inference loops. Currently you pay a PCIe latency penalty for every single token generated because the CPU has to handle the sampling and control logic. Moving that logic to the GPU and keeping the whole ge
36.
by storystarling 9mo ago
The hardest part of a rewrite like this is usually maintaining bug-for-bug compatibility with the legacy parser rather than the actual Rust implementation. Most real-world media files are malformed in some way that the C++ code implicitly h
37.
by storystarling 9mo ago
Curious how you handle sharding and KV cache pressure for a 120b model. I guess you are doing tensor parallelism across consumer cards, or is it a unified memory setup?
38.
by storystarling 9mo ago
I ran into similar issues with context rot on a larger backend project recently. I ended up writing a tool that parses the AST to strip out function bodies and only feeds the relevant signatures and type definitions into the prompt. It cuts
39.
by storystarling 9mo ago
Fair point. I suspect it comes down to how they handle retries. If an upload times out but the counter already incremented, the system sees the space as used until an async cleanup job runs. It is really common to have ghost usage in eventu
40.
by storystarling 9mo ago
Makes sense regarding the MoE performance. I am not sure the cost argument holds up for high volume workloads though. If you are running batch jobs 24/7 the hardware pays for itself in a few months compared to API opex. It really just
41.
by storystarling 9mo ago
Isolated state seems like the right call. I am curious how you implemented the shared memory primitives though. I spent a while trying to get zero-copy buffer sharing right in a previous project and usually ended up complicating the host AP
42.
by storystarling 9mo ago
Thanks for the link. I will take a look at the internals. The trade-off between global locks and isolated states is usually the most critical decision for embeddable languages so I am curious to see how the implementation handles it.
43.
by storystarling 9mo ago
How are you handling shared state with the new concurrency primitives? Since it is embeddable I am curious if you went with a global lock approach like Python or isolated states similar to Lua. Managing thread safety while keeping the C API
44.
by storystarling 9mo ago
Does the AI rewriting actually run on-device? The site mentions local storage but I wasn't sure if that implies local inference for the processing modes too. If it is local, curious what models you're using to handle the structure
45.
by storystarling 9mo ago
I've mostly switched to raw SQL for these kinds of queries too. It seems like the maintenance burden is actually lower when you can see the explicit query plan, rather than trying to reverse engineer what the ORM is doing with the join
46.
by storystarling 9mo ago
Have you tried running any local LLMs via llama.cpp? I am curious if that high RAM is effectively usable as unified memory for larger models. I wonder if the memory bandwidth is sufficient to get decent performance on something like a 70b m
47.
by storystarling 9mo ago
This sounds like a classic consistency vs latency trade-off. Enforcing strict quotas across distributed services usually requires coordination that kills performance. They likely rely on asynchronous counters that drift, meaning the fronten
48.
by storystarling 9mo ago
I have found modular monoliths are basically a requirement if you are bootstrapping and want to keep margins healthy. The coordination overhead of microservices kills velocity. That said, I would be careful about rolling your own infrastruc
49.
by storystarling 9mo ago
Curious how you handle the region files. Standard git tends to fall over with the size of world history given they are large binaries, so I assume there is some custom diffing logic involved?
50.
by storystarling 9mo ago
I used Hilbert curves for data locality in a distributed system a few years back. It’s great for optimizing range queries across multi-dimensional data. Does your implementation use SIMD for the bit-interleaving? I remember that being the m
51.
by storystarling 9mo ago
It seems the main issue is they lack a system-level mental model. I have found I need to define strict boundaries and interfaces first to get a usable result. You still have to do the architecture work upfront.
52.
by storystarling 9mo ago
The happy path is trivial now but I've found the gap between prototype and production is actually wider. You end up spending all your time handling non-determinism and latency issues that simply didn't exist with deterministic cod
53.
by storystarling 9mo ago
The unit price looks low but the cost driver in these apps is usually the context window. You have to re-send the full history every turn to maintain state, so you end up paying for the same tokens dozens of times. By the end of a long sess
54.
by storystarling 9mo ago
I run a small print-on-demand platform and this is exactly what we're seeing. The submissions used to be easy to filter with basic heuristics or cheap classifiers, but now the grammar and structure are technically perfect. The problem
55.
by storystarling 9mo ago
Fast feedback is key, but I'm skeptical of the $100 figure for training nanoGPT. If you use spot instances on Lambda or RunPod you can train a model that size for less than a dollar. I've been running similar experiments recently
56.
by storystarling 9mo ago
The reasoning gains make sense but I am wondering about the production economics. Running four distinct agent roles per update seems like a huge multiplier on latency and token spend. Does the claimed efficiency actually offset the aggregat
57.
by storystarling 9mo ago
3-8 t/s seems pretty sluggish for interactive coding. I guess it works for background agents but for a human loop that latency is tough. Also hard to justify the $50k capex compared to just hitting the Anthropic API. You'd need ma
58.
by storystarling 9mo ago
Curious how you handle context window management here. I've been building something similar and found balancing latency with accuracy to be the hardest part. Are you doing any AST parsing to reduce the payload or just relying on larger
59.
by storystarling 9mo ago
I am surprised you went with ElevenLabs for the transcription layer. In my experience the pricing there makes the unit economics pretty tough for high volume calls. We moved a similar stack to Deepgram recently and the latency is significan
60.
by storystarling 9mo ago
The bottleneck with git is actually the on-the-fly packfile generation. The server has to burn CPU calculating deltas for every clone. For a distributed system it seems much better to use a simple content-addressable store where you just se

More ›