kenmu

5 ms·

1. 1.▲
   by kenmu 6mo ago

   Love your idea. We have timeout mechanisms and we originally would be pretty aggressive with timeouts based on both time and response length to balance accuracy and speed. There’s research that longer responses tend to be less accurate (whe
2. 2.▲
   by kenmu 6mo ago

   As of right now, we do not. I'm working on these other benchmarks, but unfortunately they cost quite a bit of money to run, which I'm hoping will come from many people using Sup :)
3. 3.▲
   by kenmu 6mo ago

   I mentioned in another comment that I make sure the cost/time is within 1.25x of the next best single-model run. So it's not perfect, but I think that aspect will only get better with time. Of course I'm biased, but using Sup
4. 4.▲
   by kenmu 6mo ago

   Love the idea of learning the skill of vibe coding. The subtitle of the book, "Idea → Prompt → AI → Edit → Ship" doesn't quite go with the premise, "There is no magic prompt that turns your idea into a finished product o
5. 5.▲
   by kenmu 6mo ago

   Well-written article. Does a great job walking through why any robust system will need what DSPy provides. Though there are many libraries and frameworks that will provide the basics, RAG, exponential back-off, etc. DSPy's real value i
6. 6.▲
   by kenmu 6mo ago

   Is their scaffold available? Does it do anything special beyond feeding the warmup, single challenge, and full problem to an LLM? Because it's interesting that GPT-5.2 Pro, arguably the best model until a few months ago, couldn't