2 ms·
With C++ formatting is optional. A better test case for LLMs is Python where indention specifies code blocks. Even ChatGPT 3.5 got the formatting for Python a
by raw_anon_1111 10mo ago
With C++ formatting is optional. A better test case for LLMs is Python where indention specifies code blocks. Even ChatGPT 3.5 got the formatting for Python and YAML correct - now the actual code back then was often hilariously wrong.
- to11mtm 10mo agoI can't even get Github Copilot's plugin to avoid randomly trashing files with a Zero No width break space at the beginning, let alone follow formatting rules consistently...
- sothatsit 10mo ago> Github Copilot Well there’s your issue!
- raw_anon_1111 10mo agoI am the last person to say anything good about CoPilot. I used CoPilot for a minute, mostly used raw ChatGPT until last month and now use Codex with my personal subscription to ChatGPT and my personal but company reimbursed subscription to Claude.
- apaprocki 10mo agoA quick search finds many COBOL checkers. I’d be very surprised if a modern model was not able to fix its own mistakes if connected to a checker tool. Yes, it may not be able to one shot it perfectly, but if it can quickly call a tool once and it “works”, does it really matter much in the end? (Maybe it matters from a cost perspective, but I’m just referring to it solving the problem you asked it to solve.) Clearly it isn’t just “broken” for everyone, “Claude Code modernizes a legacy COBOL codebase”, from Anthropic: https://youtu.be/OwMu0pyYZBc https://youtu.be/OwMu0pyYZBc
- shakna 10mo agoTaking Anthropic reporting on Anthropic, at face value, is not something you should really do. In this case, a five stage pipeline, built on demo environments and code that were already in the training data, was successful. I see more red flags there, than green.