6 ms·

By the time these academic studies get published they are usually already several months out of date. o3-mini was released yesterday and if one wants to know ab

By the time these academic studies get published they are usually already several months out of date. o3-mini was released yesterday and if one wants to know about the limitations of current technology they are much better to check twitter than some research paper

Xmd5a 2y ago
The paper is recent and being discussed here: https://news.ycombinator.com/item?id=42889786 https://news.ycombinator.com/item?id=42889786
FuckButtons 2y ago
I think the breathless hype train of twitter is probably the worst place to get an actually grounded take on what the real world implications of the technology is. Seeing the 100th example of an llm generating some toy code for which there are a vast number of examples of approximately similar things in the training corpus doesn’t give you a clearer view of what is or isn’t possible.
- unification_fan 2y ago
  I think that most of the developers who advocate for AI coding have never worked all by themselves on projects with over 500/1000 files. Because if they had they would not advocate for AI coding.
  - AtlasBarfed 2y ago
    I posted this earlier, but I wanted a java port of sed for ... Reasons, and despite the existence of man pages and source code it couldn't do anything but the most basic flags. Imo this should be low hanging fruit. Porting non-trivial but yet 3-4 core code files that are already debugged and interface specified should be what an LLM excels at.
    - Kye 2y ago
      I tried this with Microsoft's Copilot + the Think Deeper button. That allegedly uses the new o1 model. It goes into a lot of fancy talk about...pretty much what you said older models did. Then it said "here's some other stuff you could extend this with!" and a list of all the important sed functionality. It's possible it could do it if prompted to finish the code with those things, but I don't know the secret number of fancy o1 uses I get and I don't want to burn them on something that's not for me. You should be able to access it here if you have a Microsoft account and want to try the button: https://copilot.microsoft.com/ https://copilot.microsoft.com/
anon291 2y ago
It fundamentally does not matter. Matrix multiplication does not erase the truth of Godel and Turing.
- drdeca 2y ago
  Who is trying to “erase the truth of Gödel and Turing”? (Well, some cranks are, but I don’t think that’s who you are talking about.) Gödel and Turing’s results do not appear to give any reason that a computer program can’t do what a person can do.
  - anon291 2y ago
    That's not the point. Computer program with a finite number of steps (an auto regressive LLM without chain of thought) has a limit in what it can reason in one step. This article does a lot of wordcelling to show this obvious point.
    - drdeca 2y ago
      That seems irrelevant to Gödel? If that was your point, you should have said that rather than the things about Turing and Gödel (which leads people to expect you are talking about the halting problem and incompleteness, not the limitations that come from a limited depth circuit)
- kadoban 2y ago
  Godel and Turing just proved that there are some true things that can't be proved, and things that cannot be computed. They didn't show where those boundaries are. They certainly didn't show those boundaries to be below human cognition level.
  - andrewflnr 2y ago
    Yeah, the grounded take is that Turing and Gödel apply just as much to human intelligence. If not, someone please go ahead and use this to physically prove the existence of an immortal, hypercomputational soul.
elicksaur 2y ago
Or neither. Try it yourself. For me, LLMs still don’t meet basic usefulness and are a net negative when I try to use them. I push code daily for my job.
- kybernetyk 2y ago
  I have a good use case for them: Communication with the bureaucracy of my country. I tell my LLM of choice to write a letter to $whoever about $whatever, then I print it out (yes, we still have to do this as email don't get accepted) and send if off. I don't even need to proof read it because if there's a mistake the bureaucracy will tell me in another letter. So the burden of correctness checking is on some bureaucrat which saves me time and mental resources. I wouldn't ever use a LLM for anything where correctness matters (code) because I'd spend the same amount of time checking the generated code as writing it myself. But a letter to my tax office? Heck, why not. If something goes really wrong I can always say "gee, I made a mistake let's try it again".
  - cropcirclbureau 2y ago
    So what, you use it to spam and waste other people's time? I know, dealing government bureaucracy and corruption is soul leeching but spam was always one of the golden usecases for generated AI.
    - bonesss 2y ago
      Sending official letters to the local government isn't spam, and generally not a waste of time. People with cognitive issues, issues typing, language or presentation issues, LLMs provide a massive improvement in how they are percieved and recieved by the other side. Also, immigrants or people with langauge issues aren't quite as disadvantaged and don't need to use excess time translating or risking an embarrasing misstatement. It's a night or day accommodation tool in the right circumstances.
    - kybernetyk 2y ago
      No, I don't just send them random letters. I reply to mail I get from them or when I need them to do something (like adjust my tax pre-pay). Also one could argue that bureaucracies only exist to create bullshit jobs and waste citizens' time. So I wouldn't even feel bad about spamming those assholes.

Xmd5a 2y ago

The paper is recent and being discussed here: https://news.ycombinator.com/item?id=42889786 https://news.ycombinator.com/item?id=42889786

FuckButtons 2y ago

I think the breathless hype train of twitter is probably the worst place to get an actually grounded take on what the real world implications of the technology is. Seeing the 100th example of an llm generating some toy code for which there are a vast number of examples of approximately similar things in the training corpus doesn’t give you a clearer view of what is or isn’t possible.

unification_fan 2y ago
I think that most of the developers who advocate for AI coding have never worked all by themselves on projects with over 500/1000 files. Because if they had they would not advocate for AI coding.
- AtlasBarfed 2y ago
  I posted this earlier, but I wanted a java port of sed for ... Reasons, and despite the existence of man pages and source code it couldn't do anything but the most basic flags. Imo this should be low hanging fruit. Porting non-trivial but yet 3-4 core code files that are already debugged and interface specified should be what an LLM excels at.
  - Kye 2y ago
    I tried this with Microsoft's Copilot + the Think Deeper button. That allegedly uses the new o1 model. It goes into a lot of fancy talk about...pretty much what you said older models did. Then it said "here's some other stuff you could extend this with!" and a list of all the important sed functionality. It's possible it could do it if prompted to finish the code with those things, but I don't know the secret number of fancy o1 uses I get and I don't want to burn them on something that's not for me. You should be able to access it here if you have a Microsoft account and want to try the button: https://copilot.microsoft.com/ https://copilot.microsoft.com/

anon291 2y ago

It fundamentally does not matter. Matrix multiplication does not erase the truth of Godel and Turing.

drdeca 2y ago
Who is trying to “erase the truth of Gödel and Turing”? (Well, some cranks are, but I don’t think that’s who you are talking about.) Gödel and Turing’s results do not appear to give any reason that a computer program can’t do what a person can do.
- anon291 2y ago
  That's not the point. Computer program with a finite number of steps (an auto regressive LLM without chain of thought) has a limit in what it can reason in one step. This article does a lot of wordcelling to show this obvious point.
  - drdeca 2y ago
    That seems irrelevant to Gödel? If that was your point, you should have said that rather than the things about Turing and Gödel (which leads people to expect you are talking about the halting problem and incompleteness, not the limitations that come from a limited depth circuit)
kadoban 2y ago
Godel and Turing just proved that there are some true things that can't be proved, and things that cannot be computed. They didn't show where those boundaries are. They certainly didn't show those boundaries to be below human cognition level.
- andrewflnr 2y ago
  Yeah, the grounded take is that Turing and Gödel apply just as much to human intelligence. If not, someone please go ahead and use this to physically prove the existence of an immortal, hypercomputational soul.

elicksaur 2y ago

Or neither. Try it yourself. For me, LLMs still don’t meet basic usefulness and are a net negative when I try to use them. I push code daily for my job.

kybernetyk 2y ago
I have a good use case for them: Communication with the bureaucracy of my country. I tell my LLM of choice to write a letter to $whoever about $whatever, then I print it out (yes, we still have to do this as email don't get accepted) and send if off. I don't even need to proof read it because if there's a mistake the bureaucracy will tell me in another letter. So the burden of correctness checking is on some bureaucrat which saves me time and mental resources. I wouldn't ever use a LLM for anything where correctness matters (code) because I'd spend the same amount of time checking the generated code as writing it myself. But a letter to my tax office? Heck, why not. If something goes really wrong I can always say "gee, I made a mistake let's try it again".
- cropcirclbureau 2y ago
  So what, you use it to spam and waste other people's time? I know, dealing government bureaucracy and corruption is soul leeching but spam was always one of the golden usecases for generated AI.
  - bonesss 2y ago
    Sending official letters to the local government isn't spam, and generally not a waste of time. People with cognitive issues, issues typing, language or presentation issues, LLMs provide a massive improvement in how they are percieved and recieved by the other side. Also, immigrants or people with langauge issues aren't quite as disadvantaged and don't need to use excess time translating or risking an embarrasing misstatement. It's a night or day accommodation tool in the right circumstances.
  - kybernetyk 2y ago
    No, I don't just send them random letters. I reply to mail I get from them or when I need them to do something (like adjust my tax pre-pay). Also one could argue that bureaucracies only exist to create bullshit jobs and waste citizens' time. So I wouldn't even feel bad about spamming those assholes.