3 ms·

It seems like if Anthropic released a super cool and useful _free_ utility (like a compiler, for example) that was better than existing counterparts or solved

It seems like if Anthropic released a super cool and useful _free_ utility (like a compiler, for example) that was better than existing counterparts or solved a problem that hadn’t been solved before[0] and just casually said “Here is this awesome thing that you should use every day. By the way our language model made this.” it would be incredible advertising for them. But they instead made a blog post about how it would cost you twenty thousand dollars to recreate a piece of software that they do not, with a straight face, actually recommend that you use in any capacity beyond as a toy. [0] I am categorically not talking about anything AI related or anything that is directly a part of their sales funnel. I am talking about a piece of software that just efficiently does something useful. GCC is an example, Everything by voidtools is an example, Wireshark is an example, etc. Claude is not an example.

tjr 9mo ago
I wonder how much more it would take Anthropic to make CCC on par with, or even better than, GCC.
- hulitu 9mo ago
  At the current state of AI, some centuries. As far as I underestand from the comments, Anthropic released a "compiler" that translates C code to some assembly, which might or might not be valid input for a linker. They claimed they were able to compile the Linux kernel (which version ? which config) and boot it (was the boot successfull ? were all devices correctly initialized ? Is userland running without problems ?) At the moment it really looks like a political farce with no real outcome except some promisses. Is there a code repository so i can test it ? What is the licence of this compiler ?
vidarh 9mo ago
They made a blog post about it because it's an amazing test of the abilities of the models to deliver a working C-compiler, even with lots of bugs and serious caveats, for $20k of tokens, without a human babysitting it. I'd challenge anyone who are negative to this to try to achieve what they did by hand, with the same restrictions (e.g. generating full SSA form instead of just directly emitting code, capable of compiling Linux), and log their time doing it. Having written several compilers, I'll say with some confidence that not many developers would succeed. Far fewer would succeed fast enough to compete with $20k cost. Even fewer would do that and deliver decent quality code. Now notice the part where they've done this experiment before. This is the first time it succeeded. Give it another model iteration or two, and expect quality to increase, and price to drop. This is the new floor.
- jrflowers 9mo ago
  >without a human babysitting it. From the original blog post: >Every agent would hit the same bug, fix that bug, and then overwrite each other's changes. Having 16 agents running didn't help because each was stuck solving the same task. >The fix was to use GCC as an online known-good compiler oracle to compare against. I wrote a new test harness that randomly compiled most of the kernel using GCC The blog post used the word autonomous a lot, which I suppose is true if Nicholas Carlini is not a human being but in fact a Claude agent. >I'd challenge anyone who are negative to this to try to achieve what they did by hand, with the same restrictions (e.g. generating full SSA form instead of just directly emitting code, capable of compiling Linux), and log their time doing it. Why would anyone do that? My point was that why does the company _not_ make a useful tool? I feel like that is a much more interesting topic of discussion than “why aren’t people that aren’t impressed by this spending their time trying to make this company look good?” >This is the new floor. Aside from the notion that they maybe intentionally set out to create the least useful or valuable output from their tooling (eg ‘the floor’) when they did not say that they did that, my question was “Why do they not make something genuinely useful?”. Marketing speak and imaginary engineers failing at made up challenges does not answer that question.
  - vidarh 9mo ago
    > The blog post used the word autonomous a lot, which I suppose is true if Nicholas Carlini is not a human being but in fact a Claude agent. Nothing in the article suggests it did not autonomously do the work. > Why would anyone do that? Because a lot of naysayers here pretend as if this is somehow trivial. > My point was that why does the company _not_ make a useful tool? Useful to whom? This is a researcher testing the limits of the models. Knowing those limits is highly useful to Anthropic. And it's highly useful to lots of others too, like me, as a means of understanding the capabilities of these models. What, exactly would such a tool that'd somehow make the people dismissing this change their minds look like? Because I don't think anything would. They could produce lots of useful tools, if they aimed lower than testing the limits of the model. But it would not achieve what they set out to do, and it would not tell us anything useful. I produce "useful tools" with Claude every day. That's not interesting. Anyone who actually uses these tools properly will develop a good understanding of the many things that can be achieved with them. Most of us can't spend $20k figuring out where the limits are, however. > I feel like that is a much more interesting topic of discussion than “why aren’t people that aren’t impressed by this spending their time trying to make this company look good?” This is a ridiculous misrepresentation of the point. The point is that the people who aren't impressed by this very clearly and obviously do not have an understanding of the complexity of what they achieved, and are making ignorant statements about it. > Aside from the notion that they maybe intentionally set out to create the least useful or valuable output from their tooling (eg ‘the floor’) Again, you're either entirely failing to understand, or wilfully misrepresenting what I said. No, their goal was not to "set out the create the least useful or valuable output". Their goal was to test the limits of what the model can achieve. They did that. That has far higher value than not testing the limits. Lots, and lots of people are building tools with Claude without testing the limits. We would not learn anything from that. > my question was “Why do they not make something genuinely useful?” Because that wasn't the purpose. The purpose was to test the limits of what the model can achieve. That you struggle to understand why what they achieved was massively impressive, does not change that.

tjr 9mo ago

I wonder how much more it would take Anthropic to make CCC on par with, or even better than, GCC.

hulitu 9mo ago
At the current state of AI, some centuries. As far as I underestand from the comments, Anthropic released a "compiler" that translates C code to some assembly, which might or might not be valid input for a linker. They claimed they were able to compile the Linux kernel (which version ? which config) and boot it (was the boot successfull ? were all devices correctly initialized ? Is userland running without problems ?) At the moment it really looks like a political farce with no real outcome except some promisses. Is there a code repository so i can test it ? What is the licence of this compiler ?

vidarh 9mo ago

They made a blog post about it because it's an amazing test of the abilities of the models to deliver a working C-compiler, even with lots of bugs and serious caveats, for $20k of tokens, without a human babysitting it. I'd challenge anyone who are negative to this to try to achieve what they did by hand, with the same restrictions (e.g. generating full SSA form instead of just directly emitting code, capable of compiling Linux), and log their time doing it. Having written several compilers, I'll say with some confidence that not many developers would succeed. Far fewer would succeed fast enough to compete with $20k cost. Even fewer would do that and deliver decent quality code. Now notice the part where they've done this experiment before. This is the first time it succeeded. Give it another model iteration or two, and expect quality to increase, and price to drop. This is the new floor.

jrflowers 9mo ago
>without a human babysitting it. From the original blog post: >Every agent would hit the same bug, fix that bug, and then overwrite each other's changes. Having 16 agents running didn't help because each was stuck solving the same task. >The fix was to use GCC as an online known-good compiler oracle to compare against. I wrote a new test harness that randomly compiled most of the kernel using GCC The blog post used the word autonomous a lot, which I suppose is true if Nicholas Carlini is not a human being but in fact a Claude agent. >I'd challenge anyone who are negative to this to try to achieve what they did by hand, with the same restrictions (e.g. generating full SSA form instead of just directly emitting code, capable of compiling Linux), and log their time doing it. Why would anyone do that? My point was that why does the company _not_ make a useful tool? I feel like that is a much more interesting topic of discussion than “why aren’t people that aren’t impressed by this spending their time trying to make this company look good?” >This is the new floor. Aside from the notion that they maybe intentionally set out to create the least useful or valuable output from their tooling (eg ‘the floor’) when they did not say that they did that, my question was “Why do they not make something genuinely useful?”. Marketing speak and imaginary engineers failing at made up challenges does not answer that question.
- vidarh 9mo ago
  > The blog post used the word autonomous a lot, which I suppose is true if Nicholas Carlini is not a human being but in fact a Claude agent. Nothing in the article suggests it did not autonomously do the work. > Why would anyone do that? Because a lot of naysayers here pretend as if this is somehow trivial. > My point was that why does the company _not_ make a useful tool? Useful to whom? This is a researcher testing the limits of the models. Knowing those limits is highly useful to Anthropic. And it's highly useful to lots of others too, like me, as a means of understanding the capabilities of these models. What, exactly would such a tool that'd somehow make the people dismissing this change their minds look like? Because I don't think anything would. They could produce lots of useful tools, if they aimed lower than testing the limits of the model. But it would not achieve what they set out to do, and it would not tell us anything useful. I produce "useful tools" with Claude every day. That's not interesting. Anyone who actually uses these tools properly will develop a good understanding of the many things that can be achieved with them. Most of us can't spend $20k figuring out where the limits are, however. > I feel like that is a much more interesting topic of discussion than “why aren’t people that aren’t impressed by this spending their time trying to make this company look good?” This is a ridiculous misrepresentation of the point. The point is that the people who aren't impressed by this very clearly and obviously do not have an understanding of the complexity of what they achieved, and are making ignorant statements about it. > Aside from the notion that they maybe intentionally set out to create the least useful or valuable output from their tooling (eg ‘the floor’) Again, you're either entirely failing to understand, or wilfully misrepresenting what I said. No, their goal was not to "set out the create the least useful or valuable output". Their goal was to test the limits of what the model can achieve. They did that. That has far higher value than not testing the limits. Lots, and lots of people are building tools with Claude without testing the limits. We would not learn anything from that. > my question was “Why do they not make something genuinely useful?” Because that wasn't the purpose. The purpose was to test the limits of what the model can achieve. That you struggle to understand why what they achieved was massively impressive, does not change that.