swyx

8 ms·

91.
by swyx 4mo ago
to you it may do idk. note that if you scroll past fig 1 you get into a nice data explorer that breaks out pass@5 by reasoning level with token and $ and step cost visualized. i think some other commenters on this hn thread got very worked
92.
by swyx 4mo ago
ok i mean i agree, how is it a gaping hole when its literally the second (and third and fourth..) chart on the post? yes token cost and reasoning efficiency is important, hence the 2D pareto charts
93.
by swyx 4mo ago
no, single customer focus would be bad for a number of reasons. but frontiercode-finance? thatd be cool…
94.
by swyx 4mo ago
i agree it would be interesting but apart from the fact that its be harder to measure and automate, theres real alpha in being the best truly async, hands off model/agent, which is what cog has been working on for 2 years now. its not
95.
by swyx 4mo ago
this gives my non tired self a chance to fix the typo: - “ ON TOP of that, 40+ hours of real human work to turn…” + “ ON TOP of that, 40+ hours of real human work PER TASK to turn…”
96.
by swyx 4mo ago
> Do you plan on releasing the tasks publicly? yep
97.
by swyx 4mo ago
yes well aware :) numbers shown are on "house" harnesses eg codex with gpt and claude code with opus. fwiw we have examples of each model doing better on NON-house harnesses too - speaking jsut for myself i think the "the la
98.
by swyx 4mo ago
thanks - credit to silas, eric, ben, and team for the depth of the evals, and the rest of the research team for doing the transcript reading parties lol by nature of being based on open source, frontiercode public will saturate very very qu
99.
by swyx 4mo ago
*50 unique problems but 20-40 rubrics per problem (something I had to keep reminding people internally who were unimpressed with the N) simple answer is our reporting was pass@5. feel like you'd need like 50+ runs to have reasonable co
100.
by swyx 4mo ago
see Beyond Unit Tests and Novel Grading Methods in TFA. i think something like ~60% llm as judge rubrics and the rest as described. every rubric validated by maintainer. 3000 rubrics
101.
by swyx 4mo ago
:wave: i was on the team! AMA. some headlines - 3000 rubrics on code quality. First benchmark to measure: "would this code get actually merged?" - 20+ expert open-source maintainer created tasks on their own repos to capture their
102.
by swyx 4mo ago
and it is a capital N nonprofit https://latent.space/p/biohub
103.
by swyx 4mo ago
ya sorry we upgraded our thing after this
104.
by swyx 4mo ago
ha ok but why do that for a post you didnt write?
105.
by swyx 4mo ago
time of day also matters
106.
by swyx 4mo ago
we interviewed Ryan here: https://www.latent.space/p/harness-eng and he gave a talk version of it in london: https://www.youtube.com/watch?v=am_oeAoUhew
107.
by swyx 4mo ago
we interviewed Alex Rives, cofounder of EvoScale and Head of Science at BioHub - here https://www.latent.space/p/esmfold2 also 3 paper coauthors walked thru it with us: https://youtu.be/4g1bURdKN0Q all
108.
by swyx 4mo ago
agree reasoning fixed a lot of it. but the inherent path dependence of autoregression is one reason i was excited for text diffusion models ( https://www.youtube.com/watch?v=r305-aQTaU0 ) instead of going left to right, even
109.
by swyx 4mo ago
> If you can see that these models empirically get better with scale, why would you swap the main architecture? Those events will be pretty rare c.f. hardware lotter https://arxiv.org/abs/2009.06489
110.
by swyx 4mo ago
not exactly, bitter lesson is one meta-level up from "scale eats everything". this is a common misunderstanding of bitter lesson that rich sutton has been fighting ever since the thing was written. in rich's own words[1], the
111.
by swyx 4mo ago
why would a "live cam show" company need jiggle physics?
112.
by swyx 4mo ago
i mean right but even you hold back some stuff because youre a Nice Person yknow? haha
113.
by swyx 4mo ago
theres always a story behind the story. wondering why he chose today to say these
114.
How to Stop Shipping Low-Quality RL Environments (With Examples)(latent.space)
1 points by swyx 4mo ago|0 comments
115.
by swyx 4mo ago
as a singaporean its actually kinda funny to hear everyone here describe him as a neoliberal. not quite our lived experience
116.
by swyx 4mo ago
let me try a different tack - 99% of the value is in the affirmation of the response and the demnonstration of responsiveness, not the actual content value of the 5 words. you could send a thumbs up emoji, you could send a ".", ju
117.
by swyx 4mo ago
good luck sir. if you ever want a job i'm happy to refer you to cognition/devin.
118.
by swyx 4mo ago
then you dont deal in high enough email volume
119.
by swyx 4mo ago
very nice and well executed product for what i assume is a small team! (of one?) i guess the obvious question is... if its FOSS and local only, whats the business game plan? you've impressed me that you can ship, but you've basica
120.
by swyx 4mo ago
what i'm hearing is i need to watch this show, it might be the new Silicon Valley

More ›