4 ms·

Just because this article isn't well formed or sourced doesnt make its claim incorrect. I program daily and I use AI for it daily. For short and simple progra

Just because this article isn't well formed or sourced doesnt make its claim incorrect. I program daily and I use AI for it daily. For short and simple programs AI can do it 100x faster, but it is fundamentally limited by its context size. As a program grows in complexity AI is not currently able to create or modify the codebase with success (around 2000 lines is where I found it has a barrier). I suspect it's due to exponential complexity associated with input size. Show me an AI that can do this for a 10,000 lines complex program and I'll eat my own shorts

leblancfg 2y ago
Here's a thought experiment. Think back on how that statement would have sounded like to past-you, 3 years ago. You would probably have dismissed it as bullshit, right? We've gone a long way since then. Both in terms of better, faster and cheaper models, but also how they're being intertwined with developer tooling. Now imagine 3 years from now.
- falcolas 2y ago
  As you inadvertently pointed out, AI improvements are not linear. They depend on new discoveries more than they do iteration. We could be in either out of jobs or lamenting the stagnation of AI (again).
  - leblancfg 2y ago
    After an innovation phase there is an implementation phase. Depending on the usefulness of the innovation, the integration with existing systems takes time. It is calculated in years, tens of years. Think back on the 80-90s, where it took years to integrate PCs into offices and workspaces. From your comment, is sounds like you think that the implementation phase of LLMs is already over? And if so, how do you come to this conclusion?
    - falcolas 2y ago
      It's not as if we have no idea how to make use of AI in programming. We've been working on AI in one form or another since the 70's, and integrated them with our programming workflow almost as long (more recently in the form of autocomplete using natural language processing and machine learning models). It's already completely integrated into our IDEs, often with options to collate the output of multiple LLMs. What further implementations of integrating AI and programming workflows have LLMs shown to be missing?
- cj 2y ago
  You could have said the same for crypto/blockchain 3-4 years ago (or whenever it was at peak hype). Eventually we realized what is and isn't possible or practical to use blockchain for. It didn't really live up to all the original hype years ago, but it's still a good technology to have around. It's possible LLMs could follow a similar pattern, but who knows.
  - __loam 2y ago
    What good thing has blockchain ever done that isn't facilitating crime or tax evasion?
    - cj 2y ago
      It created a speculative asset that some people are passionate about. However, if you saw the homepage of HN during blockchain peak hype, being a speculative asset / digital currency was seen almost as a side effect of the underlying technology, but it turns out that’s pretty much all it turned out to be useful for.
- skybrian 2y ago
  You can imagine all sorts of things, and then something else might happen. You can’t rely on “proof by imagination” or “proof by lack of imagination.” We shouldn’t be highly confident in any claims about where AI will be in three years, because it depends on how successful the research is. Figuring out how to apply the technology to create successful products takes time, too.
- denismi 2y ago
  Ten years ago when Siri/Google/Alexa were launching, I really wouldn't have expected that 2024 voice assistants would be mere egg timers, and frustrating ones at that - requiring considered phrasing and regular repeating/cancelling/yelling to trick it into doing what you want. A 10x near future isn't inconceivable, but neither is one where we look back and laugh at how hyped we got at that early-20s version of language models.
  - hyperG 2y ago
    It is a great point. It also might be that the language everyone uses 20 years from now that gives a 50X from today is just being worked on right now or won't come along for another 5 years. The way people who would have thought that humans could never fly were not completely wrong before the airplane. After the airplane though, we are really talking about two different versions of a "human that can fly".
- microtonal 2y ago
  Or you are extrapolating from the exponential growth phase in a sigmoid curve. Hard to say.
- ryaniscool 2y ago
  GPT-4 has been out for 1.5 years and I haven't seen much improvement in code quality across LLM's. It's still one of the best.
- tester756 2y ago
  Same thing that can be said about autonomous cars in 2014? Not everything will grow exponentially forever
rlt 2y ago
In my very uninformed opinion, all we need is more clever indexing, prompting, and agents that can iteratively load parts of the codebase into their context and make modifications. Real engineers aren’t expected to hold 10,000 lines of exact code in their head, they know the overall structure and general patterns used throughout the codebase, then look up the parts they need to make a modification.
abdullin 2y ago
I would recommend giving a try to o1-preview in coding tasks like this one. It is one level above Claude 3.5 Sonnet, which currently is the most popular tool among my peers.
CapeTheory 2y ago
Doesn't even take that much. Today I did some basic Terraform with the help of GenAI - it can certainly print out the fundamentals (VPC, subnets) faster than I can type them myself, but the wheels came off quickly. It hallucinated 2 or 3 things (some non-existent provider features, invalid TF syntax, etc). When you take into account prompt writing time and the effort of fixing its mistakes, I would have been better off doing the whole thing by hand. To make matters worse, I find that my mental model of what was created is nowhere near as strong as it would have been if I did things myself - meaning that when I go back to the code tomorrow, I might be better off just starting from scratch.
microtonal 2y ago
I suspect it's due to exponential complexity associated with input size. I am curious how you get to exponential complexity. The time complexity of a normal transformer is quadratic. Or do you mean that the complexity of dealing with a codebase grows exponentially with the length?
- ccvannorman 2y ago
  Generally speaking, complexity grows exponentially with input size. A program of length 100 lines has the potential to be 1000x as complex as a program with length 10 lines. Consider the amount of information that could be stored in 4 bits vs 8 bits; 2^4 vs 2^8. As the potential to be complex grows, so does current AI's ability to effectively write and modify code at that scale.

leblancfg 2y ago

Here's a thought experiment. Think back on how that statement would have sounded like to past-you, 3 years ago. You would probably have dismissed it as bullshit, right? We've gone a long way since then. Both in terms of better, faster and cheaper models, but also how they're being intertwined with developer tooling. Now imagine 3 years from now.

falcolas 2y ago
As you inadvertently pointed out, AI improvements are not linear. They depend on new discoveries more than they do iteration. We could be in either out of jobs or lamenting the stagnation of AI (again).
- leblancfg 2y ago
  After an innovation phase there is an implementation phase. Depending on the usefulness of the innovation, the integration with existing systems takes time. It is calculated in years, tens of years. Think back on the 80-90s, where it took years to integrate PCs into offices and workspaces. From your comment, is sounds like you think that the implementation phase of LLMs is already over? And if so, how do you come to this conclusion?
  - falcolas 2y ago
    It's not as if we have no idea how to make use of AI in programming. We've been working on AI in one form or another since the 70's, and integrated them with our programming workflow almost as long (more recently in the form of autocomplete using natural language processing and machine learning models). It's already completely integrated into our IDEs, often with options to collate the output of multiple LLMs. What further implementations of integrating AI and programming workflows have LLMs shown to be missing?
cj 2y ago
You could have said the same for crypto/blockchain 3-4 years ago (or whenever it was at peak hype). Eventually we realized what is and isn't possible or practical to use blockchain for. It didn't really live up to all the original hype years ago, but it's still a good technology to have around. It's possible LLMs could follow a similar pattern, but who knows.
- __loam 2y ago
  What good thing has blockchain ever done that isn't facilitating crime or tax evasion?
  - cj 2y ago
    It created a speculative asset that some people are passionate about. However, if you saw the homepage of HN during blockchain peak hype, being a speculative asset / digital currency was seen almost as a side effect of the underlying technology, but it turns out that’s pretty much all it turned out to be useful for.
skybrian 2y ago
You can imagine all sorts of things, and then something else might happen. You can’t rely on “proof by imagination” or “proof by lack of imagination.” We shouldn’t be highly confident in any claims about where AI will be in three years, because it depends on how successful the research is. Figuring out how to apply the technology to create successful products takes time, too.
denismi 2y ago
Ten years ago when Siri/Google/Alexa were launching, I really wouldn't have expected that 2024 voice assistants would be mere egg timers, and frustrating ones at that - requiring considered phrasing and regular repeating/cancelling/yelling to trick it into doing what you want. A 10x near future isn't inconceivable, but neither is one where we look back and laugh at how hyped we got at that early-20s version of language models.
- hyperG 2y ago
  It is a great point. It also might be that the language everyone uses 20 years from now that gives a 50X from today is just being worked on right now or won't come along for another 5 years. The way people who would have thought that humans could never fly were not completely wrong before the airplane. After the airplane though, we are really talking about two different versions of a "human that can fly".
microtonal 2y ago
Or you are extrapolating from the exponential growth phase in a sigmoid curve. Hard to say.
ryaniscool 2y ago
GPT-4 has been out for 1.5 years and I haven't seen much improvement in code quality across LLM's. It's still one of the best.
tester756 2y ago
Same thing that can be said about autonomous cars in 2014? Not everything will grow exponentially forever

rlt 2y ago

In my very uninformed opinion, all we need is more clever indexing, prompting, and agents that can iteratively load parts of the codebase into their context and make modifications. Real engineers aren’t expected to hold 10,000 lines of exact code in their head, they know the overall structure and general patterns used throughout the codebase, then look up the parts they need to make a modification.

abdullin 2y ago

I would recommend giving a try to o1-preview in coding tasks like this one. It is one level above Claude 3.5 Sonnet, which currently is the most popular tool among my peers.

CapeTheory 2y ago

Doesn't even take that much. Today I did some basic Terraform with the help of GenAI - it can certainly print out the fundamentals (VPC, subnets) faster than I can type them myself, but the wheels came off quickly. It hallucinated 2 or 3 things (some non-existent provider features, invalid TF syntax, etc). When you take into account prompt writing time and the effort of fixing its mistakes, I would have been better off doing the whole thing by hand. To make matters worse, I find that my mental model of what was created is nowhere near as strong as it would have been if I did things myself - meaning that when I go back to the code tomorrow, I might be better off just starting from scratch.

microtonal 2y ago

I suspect it's due to exponential complexity associated with input size. I am curious how you get to exponential complexity. The time complexity of a normal transformer is quadratic. Or do you mean that the complexity of dealing with a codebase grows exponentially with the length?

ccvannorman 2y ago
Generally speaking, complexity grows exponentially with input size. A program of length 100 lines has the potential to be 1000x as complex as a program with length 10 lines. Consider the amount of information that could be stored in 4 bits vs 8 bits; 2^4 vs 2^8. As the potential to be complex grows, so does current AI's ability to effectively write and modify code at that scale.