4 ms·

Show HN: HighSNR – Cut length and noise from your LLM context

6 points by gskm 7mo ago

kostas77 7mo ago
Really high potential service to reduce OPEX of AI services and increase the context window! And really straightforward to test it.
- gskm 7mo ago
  Thank you! That's exactly the goal, drop-in token savings without changing your LLM pipeline. If you give it a spin, I'd love to hear how it works on your data. We're actively tuning the ranking based on early feedback, so any input helps shape the product.
gskm 7mo ago
[dead]
gskm 7mo ago
Update on the benchmark numbers: the results in the original post were computed with a looser tokenizer, making the budget less strict than it should be. We've since improved that — the budget is now accurate end-to-end. Corrected numbers at 90% budget: HotpotQA F1 71.57 vs full-context baseline 69.71 — beating it by a wider margin than previously reported. Qasper 46.25 vs 47.22 (~98% of full-context quality). Updated results and scripts: https://github.com/HighSNRInc/highsnr-benchmarks https://github.com/HighSNRInc/highsnr-benchmarks One thing worth clarifying: there's no model in the processing pipeline. The ranking is fully deterministic — same input always produces the same output. This means it's fast enough for synchronous calls, runs well on commodity CPUs without GPUs, and can handle high throughput without the latency or cost overhead of an inference step.
imitliagkas 7mo ago
Very useful! I really don't like paying for unnecessary tokens. And works very fast for me. Do you have some standard practices to recommend with respect to the optional hint in the call? It seems to help significantly.
- gskm 7mo ago
  Thanks! Glad it's working well for you. A few practical tips: 1. Pass the user's query directly. In the benchmark, the hint is literally the question. That's the simplest and most effective approach for RAG. 2. Keep it concise (a sentence or two). Natural language works fine. 3. Skip it for summarization. When there's no specific query, omitting the hint lets the optimizer select for overall document coverage, which is probably what you want. 4. Biggest impact at lower budgets. The hint shines most when the optimizer has to be selective, e.g., at 50% budget on Qasper, hint adds nearly 6 F1 points (41.27 vs 35.35).

6 points by gskm 7mo ago

Really high potential service to reduce OPEX of AI services and increase the context window! And really straightforward to test it.

gskm 7mo ago
Thank you! That's exactly the goal, drop-in token savings without changing your LLM pipeline. If you give it a spin, I'd love to hear how it works on your data. We're actively tuning the ranking based on early feedback, so any input helps shape the product.

gskm 7mo ago

[dead]

gskm 7mo ago

Update on the benchmark numbers: the results in the original post were computed with a looser tokenizer, making the budget less strict than it should be. We've since improved that — the budget is now accurate end-to-end. Corrected numbers at 90% budget: HotpotQA F1 71.57 vs full-context baseline 69.71 — beating it by a wider margin than previously reported. Qasper 46.25 vs 47.22 (~98% of full-context quality). Updated results and scripts: https://github.com/HighSNRInc/highsnr-benchmarks https://github.com/HighSNRInc/highsnr-benchmarks One thing worth clarifying: there's no model in the processing pipeline. The ranking is fully deterministic — same input always produces the same output. This means it's fast enough for synchronous calls, runs well on commodity CPUs without GPUs, and can handle high throughput without the latency or cost overhead of an inference step.

imitliagkas 7mo ago

Very useful! I really don't like paying for unnecessary tokens. And works very fast for me. Do you have some standard practices to recommend with respect to the optional hint in the call? It seems to help significantly.

gskm 7mo ago
Thanks! Glad it's working well for you. A few practical tips: 1. Pass the user's query directly. In the benchmark, the hint is literally the question. That's the simplest and most effective approach for RAG. 2. Keep it concise (a sentence or two). Natural language works fine. 3. Skip it for summarization. When there's no specific query, omitting the hint lets the optimizer select for overall document coverage, which is probably what you want. 4. Biggest impact at lower budgets. The hint shines most when the optimizer has to be selective, e.g., at 50% budget on Qasper, hint adds nearly 6 F1 points (41.27 vs 35.35).