dipampaul17

4 ms·

1.
Show HN: Democlean – Score robot demos by motion quality(github.com)
1 points by dipampaul17 8mo ago|0 comments
2.
Show HN: AgentGuard – Auto-kill AI agents before they burn through your budget(github.com)
47 points by dipampaul17 1y ago|26 comments
3.
by dipampaul17 1y ago
For 30-40B parameter models, you'll see two types of performance impacts: First, there's a direct throughput improvement – our benchmarks show a 14.5% speed increase with K8V4 versus FP16. This comes from better memory bandwidth u
4.
by dipampaul17 1y ago
Yup, this approach would likely work on NVIDIA/AMD GPUs as well - the underlying principle that keys require higher precision than values is hardware-independent. The CUDA backend in llama.cpp already supports separate cache type setti
5.
by dipampaul17 1y ago
You're right to question the perplexity impact - 0.86% isn't negligible. Our extended testing shows this impact remains fairly consistent across context lengths up to 16K, which was our test limit. We haven't benchmarked at 6
6.
by dipampaul17 1y ago
The memory savings from KVSplit scale proportionally with context length, so higher-RAM Macs (64GB/128GB) benefit even more in absolute terms. On a 128GB Mac Studio, you could potentially handle context windows in the hundreds of thous
7.
by dipampaul17 1y ago
Great question about the intuition! The difference comes from the core roles these components play in attention. Keys determine which tokens to attend to - they create the actual attention pattern through similarity calculations. Values o
8.
by dipampaul17 1y ago
With the K8V4 configuration providing 59% memory savings, you can effectively run contexts 2.4× longer on the same hardware. A model with a 2048 token context can now handle about 5000 tokens, while an 8K context model can reach approximate
9.
by dipampaul17 1y ago
Yes, that's one of the key benefits - KVSplit works with any existing .gguf model without requiring reconstruction or special conversion. The quantization happens at runtime on the KV cache, not during model loading or conversion. This
10.
Show HN: KVSplit – Run 2-3x longer contexts on Apple Silicon(github.com)
272 points by dipampaul17 1y ago|40 comments