10 ms·

Without reliable benchmarks, they are Mythos-like only in the sense that they accept text as input and produce text as output.

by glimshe 4mo ago

irthomasthomas 4mo ago
They provide benchmarks in the paper https:// arxiv.org/abs/2606.21228
- khurs 4mo ago
  Think 'indepenent' was implied when glimshe wrote 'reliable' benchmarks. Needs to be on the usual leaderboards.
theplumber 4mo ago
Well if they are hyped like Mythos then we can add that to the list of “like Mythos”. Perhaps what’s missing is their CEO warning the world that their model is too unsafe to be released on the internet and someone must stop them before it’s too late.
chrsw 4mo ago
I don't even look at benchmarks anymore. I just try different models as they're released on our large, proprietary, systems software codebases in real, shipping products or projects that will ship eventually. It's pretty clear which models help me do my job better or faster. I'm fortunate enough to have the token budget to use basically as much as I need, for now. No need for benchmarks, evals, marketing, system cards or anything like that. I read the web for tips, practices and release announcements. My colleagues and I share our experiences with each other but beyond that, everything else is just noise.
- BlaDeKke 4mo ago
  This is the way. Not that big of budget here. But if there’s something promising, I just try that for a month or so. But even then… at this moment I’m using z.ai models and those do the job. No need for anything else. So I’m staying until there is something new, same affordability, but a lot better. (Using a coding plan)
- buthowjejddjeu 4mo ago
  I am not against AI but I do wonder how you guys handle the fact this leaks all your code and is stored forever on servers belonging to God knows who? I “trust” OpenAI and Anthropic (somewhat) but to be honest I still feel only safe using it on code without any secret sauce whatsoever. Luckily that’s a lot of code, but still. I wonder how others are looking at this? (FYI I feel the same about Github and we also don’t store our code there)
  - chrsw 4mo ago
    > how you guys handle the fact this leaks all your code and is stored forever on servers belonging to God knows who? I don't. My company doesn't host any code on GitHub, we have our own Git servers. Regarding the privacy issues with OpenAI, Anthropic or anyone else, we engineers are just using the tools we've been authorized to use by the company. If there's a security issue, that would need to be worked out between the company and the model providers. Would I use external model providers through an API for personal projects? For everything I'm doing now, probably. Could I see a day where I'm working on something too sensitive to be willing to give any data to these companies? Possibly.

Without reliable benchmarks, they are Mythos-like only in the sense that they accept text as input and produce text as output.

by glimshe 4mo ago

Without reliable benchmarks, they are Mythos-like only in the sense that they accept text as input and produce text as output.

irthomasthomas 4mo ago

They provide benchmarks in the paper https:// arxiv.org/abs/2606.21228

khurs 4mo ago
Think 'indepenent' was implied when glimshe wrote 'reliable' benchmarks. Needs to be on the usual leaderboards.

theplumber 4mo ago

Well if they are hyped like Mythos then we can add that to the list of “like Mythos”. Perhaps what’s missing is their CEO warning the world that their model is too unsafe to be released on the internet and someone must stop them before it’s too late.

chrsw 4mo ago

I don't even look at benchmarks anymore. I just try different models as they're released on our large, proprietary, systems software codebases in real, shipping products or projects that will ship eventually. It's pretty clear which models help me do my job better or faster. I'm fortunate enough to have the token budget to use basically as much as I need, for now. No need for benchmarks, evals, marketing, system cards or anything like that. I read the web for tips, practices and release announcements. My colleagues and I share our experiences with each other but beyond that, everything else is just noise.

BlaDeKke 4mo ago
This is the way. Not that big of budget here. But if there’s something promising, I just try that for a month or so. But even then… at this moment I’m using z.ai models and those do the job. No need for anything else. So I’m staying until there is something new, same affordability, but a lot better. (Using a coding plan)
buthowjejddjeu 4mo ago
I am not against AI but I do wonder how you guys handle the fact this leaks all your code and is stored forever on servers belonging to God knows who? I “trust” OpenAI and Anthropic (somewhat) but to be honest I still feel only safe using it on code without any secret sauce whatsoever. Luckily that’s a lot of code, but still. I wonder how others are looking at this? (FYI I feel the same about Github and we also don’t store our code there)
- chrsw 4mo ago
  > how you guys handle the fact this leaks all your code and is stored forever on servers belonging to God knows who? I don't. My company doesn't host any code on GitHub, we have our own Git servers. Regarding the privacy issues with OpenAI, Anthropic or anyone else, we engineers are just using the tools we've been authorized to use by the company. If there's a security issue, that would need to be worked out between the company and the model providers. Would I use external model providers through an API for personal projects? For everything I'm doing now, probably. Could I see a day where I'm working on something too sensitive to be willing to give any data to these companies? Possibly.