15 ms·
> Still a really cool project! Yeah. This test sorta definitely proves that AI is legit. Despite the millions of people still insisting it's a hoax. The fac
by qarl 9mo ago
> Still a really cool project!
Yeah. This test sorta definitely proves that AI is legit. Despite the millions of people still insisting it's a hoax.
The fact that the optimizations aren't as good as the 40 year gcc project? Eh - I think people who focus on that are probably still in some serious denial.
- PostOnce 9mo agoIt's amazing that it "works", but viability is another issue. It cost $20,000 and it worked, but it's also totally possible to spend $20,000 and have Claude shit out a pile of nonsense. You won't know until you've finished spending the money whether it will fail or not. Anthropic doesn't sell a contract that says "We'll only bill you if it works" like you can get from a bunch of humans. Do catastrophic bugs exist in that code? Who knows, it's 100,000 lines, it'll take a while to review. On top of that, Anthropic is losing money on it. All of those things combined, viability remains a serious question.
- deleted 9mo ago[deleted]
- tumdum_ 9mo ago> On top of that, Anthropic is losing money on it. It seems they are *not* losing money on inference: https://bsky.app/profile/steveklabnik.com/post/3mdirf7tj5s2e https://bsky.app/profile/steveklabnik.com/post/3mdirf7tj5s2e
- deleted 9mo ago[deleted]
- byzantinegene 9mo agono, and that is widely known. the actual problem is that the margins are not sufficient at that scale to make up for the gargantuan training costs to train their SOTA model.
- aurareturn 9mo agoSource on that? Because inference revenue is outpacing training cost based on OpenAI’s report and intuition.
- cma 9mo agoNet inference revenue would need to be outpacing to go against his think about margins.
- JamesBarney 9mo agoThey are large enough to cover their previous training costs but not their next gen training costs. i.e They made more money on 3.5 than 3.5 cost to train, but didn't make enough money on 3.5 to train 4.0.
- quikoa 9mo agoThat's for the API right? The subs are still a loss. I don't know which one of the two is larger.
- bdangubic 9mo ago> On top of that, Anthropic is losing money on it This has got to be my favorite one of them all that keeps coming up in too many comments… You know who also was losing money in the beginning?! every successful company that ever existed! some like Uber were losing billions for a decade. and when was the last time you rode in a taxi? (I still do, my kid never will). not sure how old you are and if you remember “facebook will never be able to monetize on mobile…” - they all lose money, until they do not
- PostOnce 9mo agoAre we forgetting that sometimes, they just go bankrupt?
- bdangubic 9mo agoname one with comparable number of users and revenue? not saying you are wrong but I would bet against the outcome
- svieira 9mo agoEnron
- bdangubic 9mo agoI should have guessed someone would answer this question in this thread with Enron :) I did not ask for random company that went under for any reason but specific question related to users and revenue.
- grey-area 9mo agoWell there are lots and lots of examples that don't end in bankruptcy, just a very large loss of capital for investors. The majority of the stars of the dotcom bubble just as one example: Qualcomm, pets.com, Yahoo!, MicroStrategy etc etc. Uber, which you cite as a success, is only just starting to make any money, and any original investors are very unlikely to see a return given the huge amounts ploughed in. MicroStrategy has transformed itself, same company, same founder, similar scam 20 years later, only this time they're peddling bitcoin as the bright new future. I'm surprised they didn't move on to GAI. Qualcomm is now selling itself as an AI first company, is it, or is it trying to ride the next bubble? Even if GAI becomes a roaring success, the prominent companies now are unlikely to be those with lasting success.
- qarl 9mo ago> It cost $20,000 I'm curious - do you have ANY idea what it costs to have humans write 100,000 lines of code??? You should look it up. :)
- sarchertech 9mo agoYou wouldn’t pay a human to write 100k LOC. Or at least you shouldn’t. You’d pay a human to write a working useful compiler that isn’t riddled with copyright issues. If you didn’t care about copying code, usefulness, or correctness you could probably get a human to whip you up a C compiler for a lot less than $20k.
- qarl 9mo agoAre you trolling me? Companies (made of humans) write 100,000 LOC all the time. And it's really expensive, despite your suspicions.
- sarchertech 9mo agoNo, companies don’t pay people to write 100k LOC. They pay people to write useful software. We figured out that LOC was a useless productivity metric in the 80s.
- qarl 9mo ago[flagged]
- chamomeal 9mo agoThat's a good point! Here claude opus wrote a C compiler. Outrageously cool. Earlier today, I couldn't get opus to replace useEffect-triggered-redux-dispatch nonsense with react-query calls. I already had a very nice react-query wrapper with tons of examples. But it just couldn't make sense of the useEffect rube goldberg machine. To be fair, it was a pretty horrible mess of useEffects. But just another data point. Also I was hoping opus would finally be able to handle complex typescript generics, but alas...
- ryanjshaw 9mo ago> You won't know until you've finished spending the money whether it will fail or not. How do you conclude that? You start off with a bunch of tests and build these things incrementally, why would you spend 20k before realizing there’s a problem?
- friendzis 9mo agoBecause literally no real-world non-research project starts with "we have an extremely comprehensive test suite and specification complete down to the most finite detail" and then searches for a way to turn it into code.
- galdauts 9mo agoPrecisely. Figuring out what the specification is supposed to look like is often the hardest part.
- mastermage 9mo ago100% agreed i use Claude often to just bounce ideas back and forth on specs i would like to create which I know will never gain traction because its either way too ambitious or too niche. And the amount of times Claude proposes something thats completely contradictory in the same response. Or completely does a 180 after two more responses. Is ridiculous.
- ryanjshaw 9mo agoI’ve spent nearly 20 years working as a consultant writing software, I know that. How do you think humans solve that problem?
- friendzis 9mo agoTypically by putting cost caps on deliverables.
- 9mo ago
- RA_Fisher 9mo agoProgress can be reviewed over time, and I'd think that'd take a lot of the risk out.
- georgeven 9mo agoit's 20,000 in 2026, with the price of tokens halving every year (at a given perf level), this will be around 1,000 dollars in 2030
- nly 9mo agoAlso, heaven knows if the result in maintainable or easy to change.
- kvemkon 9mo ago> optimizations aren't as good as the 40 year gcc project with all optimizations disabled: > Even with all optimizations enabled, it outputs less efficient code than GCC with all optimizations disabled.
- qarl 9mo agoThat distinction doesn't change my point. I am not surprised that a 40 year old project generates better code than this brand new one.
- charcircuit 9mo agoNot only is it new. There has been 0 performance optimization done. Well none prompted for at least. Once you give the agents a profiler and start a loop focusing on performance you'll see it start improving it.
- thesz 9mo agoWe are talking about compiler here and "performance" referred above is the performance of generated code. When you are optimizing a program, you have a specific part of code to improve. The part can be found with profiler. When you are optimizing a compiler generated code, you have many similar parts of code in many programs and not-so-specific part of compiler that can be improved.
- charcircuit 9mo agoYes, performance of the generated code. You have some benchmark of using a handful of common programs going through common workflows and you measure the performance of the generated code. As tweaks are made you see how the different performance experiments effect the overall performance. Some strategies are always a win, but things like how you layout different files and functions in memory have different trade offs and are hard to know up front without doing actual real world testing.
- LinXitoW 9mo agoHow does 20K to replicate code available in the thousands online (toy C compilers) prove anything? It requires a bunch of caveats about things that don't work, it requires a bunch of other tools to do stuff, and an experienced developer had to guide it pretty heavily to even get that lackluster result.
- thesz 9mo ago> This test sorta definitely proves that AI is legit. This is an "in distribution" test. There are a lot of C compilers out there, including ones with git history, implemented from scratch. "In distribution" tests do not test generalization. The "out of distribution" test would be like "implement (self-bootstrapping, Linux kernel compatible) C compiler in J." J is different enough from C and I know of no such compiler.
- disgruntledphd2 9mo ago> This is an "in distribution" test. There are a lot of C compilers out there, including ones with git history, implemented from scratch. "In distribution" tests do not test generalization. It's still really, really impressive though. Like, economics aside this is amazing progress. I remember GPT3 not being able to hold context for more than a paragraph, we've come a long way since then. Hell, I remember bag of words being state of the art when I started my career. We have come a really, really, really long way since then.
- thesz 9mo ago> It's still really, really impressive though. Do we know how many attempts were done to create such compiler before during previous tests? Would Anthropic report on the failed attempt? Can this "really, really impressive" thing be a result of a luck? Much like quoting Quake code almost verbatim not so long ago.
- disgruntledphd2 9mo ago> Do we know how many attempts were done to create such compiler before during previous tests? Would Anthropic report on the failed attempt? Can this "really, really impressive" thing be a result of a luck? No we don't and yeah we would expect them to only report positive results (this is both marketing and investigation). That being said, they provide all the code et al for people to review. I do agree that an out of distribution test would be super helpful, but given that it will almost certainly fail (given what we know about LLMs) I'm not too pushed about that given that it will definitely fail. Look, I'm pretty sceptical about AI boosting, but this is a much better attempt than the windsurf browser thing from a few months back and it's interesting to know that one can get this work. I do note that the article doesn't talk much about all the harnesses needed to make this work, which assuming that this approach is plausible, is the kind of thing that will be needed to make demonstrations like this more useful.
- soperj 9mo agoOnly if we take them at their word. I remember thinking things were in a completely different state when Amazon had their shop and go stores, but then finding out it was 1000s of people in Pakistan just watching you via camera.
- miohtama 9mo agoGCC had 40 years headstart
- byzantinegene 9mo agoit costs $20,000 to reinvent the wheel, that it probably trained on. If that's your definition of legit, sure
- organicUser 9mo agowell, if in this period it is a matter of cost, tomorrow won't be anymore. 4GB of RAM in the 80s would have cost tens of millions of dollars, now even your car runs 4 gb memory only for the infotainment systems, and runs dozens GBs of RAM for the most complex assistants. So i would see this achievement more as a warning, the final result is not what's concerning, it is the premonition behind it
- cardanome 9mo agoIf will write you an C compiler by hand for 19k and it will be better than what Claude made. Writing a toy C compiler isn't that hard. Any decent programmer can write one in a few weeks or months. The optimizations are the actually interesting part and Claude fails hard at that.
- qarl 9mo ago[flagged]
- dwaite 9mo agoIt is legit - with some pretty severe caveats. I am pressed to come up with an example that has more formal specification, published source implementations, and public unit test coverage than a C compiler. It is not feasible that someone will use AI to tackle genuinely new software and provide a tenth of the level of guide-rails Anthropic had for this project. They were able to keep the million monkeys on their million typewriters on an extremely short leash, and able to have it do the vast majority of iteration without human intervention.
- wqaatwt 9mo agoThe full source of several compilers being in its training set is somewhat helpful though. It’s not exactly a novel problem and those optimizations and edge cases which it seemingly is struggling are the overwhelming majority of the work anyway. Do we know it just didn’t shuffle gcc’s source code around a bit?