janaksunil

3 ms·

1.
by janaksunil 2mo ago
do you have some time to chat? janak@withspecific.com
2.
by janaksunil 2mo ago
would love to chat and learn more about your set up! here's my email - janak@withspecific.com
3.
by janaksunil 2mo ago
we manually vet all codebases and companies
4.
by janaksunil 2mo ago
yes!
5.
by janaksunil 2mo ago
that's fair - for long horizon engineering tasks would speed still matter?
6.
by janaksunil 2mo ago
this is a good question. what would make you reject an otherwise working PR on design grounds?
7.
by janaksunil 2mo ago
we reached out to companies that were willing to license their codebases. every codebase we used had real users, one of them had 200k+ users and is currently top 100 on the app store.
8.
by janaksunil 2mo ago
i appreciate the feedback, the benchmark is primarily long horizon real world engineering tasks on big private codebases.
9.
by janaksunil 2mo ago
we're going to open source some of our tasks and model trajectories as well
10.
by janaksunil 2mo ago
all the codebases were written pre-2023, so pre when AI got good at coding
11.
by janaksunil 2mo ago
would love to learn why?
12.
by janaksunil 2mo ago
will do! happy to chat more on janak@withspecific.com as well
13.
by janaksunil 2mo ago
we've done our best to use the native provider's harness. all models were run on 'high' reasoning. this is still v1 and tons of room for improvement - really appreciate your feedback!
14.
by janaksunil 2mo ago
makes sense
15.
by janaksunil 2mo ago
the tasks on the benchmar are long horizon swe tasks - where GLM does surprisingly well
16.
by janaksunil 2mo ago
here's where all the models messed up! its under this section 'Missed requirements are the most common failure' on realswe.withspecific.com we also have the setup in the blog. the reason for lower success rates is that we gav
17.
by janaksunil 2mo ago
nope, these were private codebases
18.
by janaksunil 2mo ago
hey this seems really interesting - what prompted you to test multiple agents on your codebase?
19.
by janaksunil 2mo ago
i'm janak, cofounder of Specific Labs (YC F25) and one of the authors of Real-SWE. if i can help answer any questions please feel free to email me at janak@withspecific.com, happy to send over my phone number as well :)
20.
Show HN: MCP server that finds dev tool credits in your workflow
1 points by janaksunil 8mo ago|1 comments