3 ms·

does the reflash actually stall every live connection, or just the ones whose request lands during that window? if the whole board goes dark for the full ~25s w

by useiris 2mo ago

does the reflash actually stall every live connection, or just the ones whose request lands during that window? if the whole board goes dark for the full ~25s while any request is queued behind it, you could probably hide most of that behind partial reconfiguration, reflashing only the region holding the model weights while the sequencer and I/O logic on the rest of the fabric stay live and keep draining the queue. that's obviously a much bigger lift than what you've built here, but it would turn a hard stop into something closer to a brief latency bump for whoever's unlucky enough to hit it, rather than a shared 25s wall for everyone behind them in line.

deleted 2mo ago
[deleted]
mikeayles 2mo ago
Good question. short version: it's a shared wall, not a per-request bump. one KV260, one model on the fabric at a time, so the rotation is global. but the ~25s is mostly software, not fabric-dark time. the PL reconfigure is ~1-3s via fpgautil, the rest is orchestration (daemon teardown, streaming the weights back into URAM, clock change, coherence check, chat-server reconnect). DFX keeping the sequencer + I/O live only recovers that 1-3s slice, and it fights the design, since the weights are smeared across every URAM column while a reconfigurable partition wants a contiguous pblock. The swap needs a fresh bitstream at all only because four little ROMs are baked at synthesis: layernorm gammas, the activation-quant scale, two dequant tables. the embeddings already ride the boot-streamed weight image. make those four boot-loadable through the same path and a same-arch model hot-swaps with no reconfigure. There is basically zero margin at the moment to implement this. which is a decent excuse for a 24h telemetry update. Peak 46 concurrent connections monday afternoon, 61 uniques in the busiest half hour, ~6,600 requests / ~400k tokens total. At no instant did more than 2 inferences overlap (although it came close once). the fabric finishes a ~100-char reply in ~2ms of compute (end-to-end round-trip is ~20-60ms once you add network + server), so even with 40+ people connected the queue sits empty ~100% of the time. Worst 30-second window of the whole day was p95 59.9ms, and the heaviest burst (~200 requests in 5 minutes from one enthusiastic visitor) didn't even produce it. still ticking at ~37 connections this morning, latency flat. That's the counterintuitive bit about a 20k tok/s model with short replies: the "single stream" is busy for milliseconds per request, so it timeshares across human typing speed essentially for free. the synthetic ceiling (2,000 concurrent requesters, p95 ~9.4s, zero errors) is ~40x past anything real traffic did. every actual visitor had the chip to themselves. After this is all over, I'll do a post-mortem to see how it held up. I was really hoping for FP with the amount of effort this one took compared to my other submissions, which really would have stressed it, but I guess it was just a bit too niche!

by useiris 2mo ago

deleted 2mo ago

[deleted]

mikeayles 2mo ago

Good question. short version: it's a shared wall, not a per-request bump. one KV260, one model on the fabric at a time, so the rotation is global. but the ~25s is mostly software, not fabric-dark time. the PL reconfigure is ~1-3s via fpgautil, the rest is orchestration (daemon teardown, streaming the weights back into URAM, clock change, coherence check, chat-server reconnect). DFX keeping the sequencer + I/O live only recovers that 1-3s slice, and it fights the design, since the weights are smeared across every URAM column while a reconfigurable partition wants a contiguous pblock. The swap needs a fresh bitstream at all only because four little ROMs are baked at synthesis: layernorm gammas, the activation-quant scale, two dequant tables. the embeddings already ride the boot-streamed weight image. make those four boot-loadable through the same path and a same-arch model hot-swaps with no reconfigure. There is basically zero margin at the moment to implement this. which is a decent excuse for a 24h telemetry update. Peak 46 concurrent connections monday afternoon, 61 uniques in the busiest half hour, ~6,600 requests / ~400k tokens total. At no instant did more than 2 inferences overlap (although it came close once). the fabric finishes a ~100-char reply in ~2ms of compute (end-to-end round-trip is ~20-60ms once you add network + server), so even with 40+ people connected the queue sits empty ~100% of the time. Worst 30-second window of the whole day was p95 59.9ms, and the heaviest burst (~200 requests in 5 minutes from one enthusiastic visitor) didn't even produce it. still ticking at ~37 connections this morning, latency flat. That's the counterintuitive bit about a 20k tok/s model with short replies: the "single stream" is busy for milliseconds per request, so it timeshares across human typing speed essentially for free. the synthetic ceiling (2,000 concurrent requesters, p95 ~9.4s, zero errors) is ~40x past anything real traffic did. every actual visitor had the chip to themselves. After this is all over, I'll do a post-mortem to see how it held up. I was really hoping for FP with the amount of effort this one took compared to my other submissions, which really would have stressed it, but I guess it was just a bit too niche!