15 ms·

I'm actually very curious why AI use is such a bi-modal experience. I've used AI to move multi thousand line codebases between languages. I've created new apps

I'm actually very curious why AI use is such a bi-modal experience. I've used AI to move multi thousand line codebases between languages. I've created new apps from scratch with it. My theory is the willingness to baby sit and the modality. I'm perfectly fine telling the tool I use its errors and working side by side with it like it was another person. At the end of the day it can belt out lines of code faster than I, or any human, can and I can review code very quickly so the overall productivity boost has been great. It does fundamentally alter my workflow. I'm very hands off keyboard when I'm working with AI in a way that is much more like working with someone or coaching someone to make something instead of doing the making myself. Which I'm fine with but recognize many developers aren't. I use AI autocomplete 0% of the time as I found that workflow was not as effective as me just writing code, but most of my most successful work using AI is a chat dialogue where I'm letting it build large swaths of the project a file or parts of a file at a time, with me reviewing and coaching.

__float 2y ago
I'm not sure how many people are like me, but my attempts to use Copilot have largely been the context of writing code as usual, occasionally getting end-of-line or handful-of-lines completions from it. I suspect there's probably a bigger shift needed, but I haven't seen anyone (besides AI "influencers" I don't trust..?) showing what their day-to-day workflows look like. Is there a Vimcasts equivalent for learning the AI editor tips and tricks?
- sbarre 2y ago
  Have you tried the chat mode? The autocomplete is somewhere between annoying and underwhelming for me, but the chat is super useful. Being able to just describe what you're thinking or what you're trying to do and having a bespoke code sample just show up (based on the code in your editor) that you can then either copy/paste in, cherry-pick from or just get inspired by, has been a great productivity booster.. Treat it like a pair programmer or a rubber duck and you might have a better experience. I did!
- whimsicalism 2y ago
  Yeah using a chat interface
zeroonetwothree 2y ago
I guess for me it actually takes longer to review code than to write it. So maybe that’s some of the difference.
0xFACEFEED 2y ago
As a programmer of over 20 years - this is terrifying. I'm willing to accept that I just have "get off my lawn" syndrome or something. But the idea of letting an LLM write/move large swaths of code seems so incredibly irresponsible. Whenever I sit down to write some code, be it a large implementation or a small function, I think about what other people (or future versions of myself) will struggle with when interacting with the code. Is it clear and concise? Is it too clever? Is it too easy to write a subtle bug when making changes? Have I made it totally clear that X is relying on Y dangerous behavior by adding a comment or intentionally making it visible in some other way? It goes the other way too. If I know someone well (or their style) then it makes evaluating their code easier. The more time I spend in a codebase the better idea I have of what the writer was trying to do. I remember spending a lot of time reading the early Redis codebase and got a pretty good sense of how Salvatore thinks. Or altering my approaches to code reviews depending on which coworker was submitting it. These weren't things I were doing out of desire but because all non-trivial code has so much subtlety; it's just the nature of the beast. So the thought of opening up a codebase that was cobbled together by an AI is just scary to me. Subtle bugs and errors would be equally distributed across the whole thing instead of where the writer was less competent (as is often the case). The whole thing just sounds like a gargantuan mess. Change my mind.
- bongodongobob 2y ago
  You. Can. Write. Tests.
  - ok_dad 2y ago
    Tests haven’t saved us so far, humans have been writing tests that passed for software with bugs for decades.
  - lanternfish 2y ago
    Tests aren't a full solution for all the considerations of the above post.
  - blitzar 2y ago
    Just let the LLM do that too.
rqmedes 2y ago
I agree. I am in a very senior role and find that working with AI the same way you do I am many times more productive. Months of work becomes days or even hours of work
bongodongobob 2y ago
My theory is grammatical correctness and specificity. I see a lot of people prompt like this: "use python to write me a prog that does some dice rolls and makes a graph" Vs "Create a Python program that generates random numbers to simulate a series of dice rolls. Export a graph of the results in PNG format." Information theory requires that you provide enough actual information. There is a minimum amount of work to supply the input. Otherwise, the gaps will get filled in with noise, working, what you want, or not. For example, maybe someday you could say "write me an OS" and it would work. However, to get exactly what you want, you still have to specify it. You can only compress so far.
2024user 2y ago
If you're doing something that appears in it's training model a lot, like building a twitter clone, then it is great. If you're using something brand new like react router 7 then it makes mistakes
xk_id 2y ago
> I'm actually very curious why AI use is such a bi-modal experience. My conspiracy theory is that the positive experiences are exaggerated and come from investors in the Nvidia stock.
- Nullabillity 2y ago
  That's a very important caveat. In our modern economy it's difficult to not be a shill in some way, shape, or form, even if you don't quite realize it consciously. It's honestly one of the most depressing things about the stock market.
  - xk_id 2y ago
    I credit my past interest in cryptocurrencies for educating me about the essence of the stock market in its purest form. And in fact there are painful parallels with the AI bubble.
  - malux85 2y ago
    Theres a big difference between being a happy customer and being a shill.
    - Nullabillity 2y ago
      Holding stock is not being a "happy customer". I may be happy with the headset that I bought, but the difference is that I don't make money if you buy an identical one.
      - malux85 2y ago
        I wasnt talking about holding stock, I was responding to this comment you said: > In our modern economy it's difficult to not be a shill in some way, shape, or form, even if you don't quite realize it consciously. Oxford dictionary defines a shill as "an accomplice of a confidence trickster or swindler who poses as a genuine customer to entice or encourage others." So the difference between someone shilling and being a satisfied customer is an intent to decieve. How is it "difficult to not pose as a genuine customer to entice or encourage others" ?
foobiekr 2y ago
The most likely explanation is that the code you are writing has low information density and is stringing things together the same way many existing apps have already done. That isn’t a judgement but trying to use the ai code completion tools for complex systems tasks is almost always a disaster.
- fullstackchris 2y ago
  Not sure what you mean by "complex systems tasks" but most of the leading models have helped me with writing concurrent go code just fine. Not sure if that counts as "complex" enough. However this was prompting, not completion. Obviously I expect something like copilot to pick the normie non-concurrent implementation
  - foobiekr 2y ago
    It doesn’t. You’re kind of illustrating what I’m getting at.
dools 2y ago
> I'm actually very curious why AI use is such a bi-modal experience I think it's just that it's better at some things than others. Lucky for people who happen to be working in python/node/php/bash/sql/java probably unlucky for people writing Go and Rust (I'm hypothesising because I don't know Go or Rust nor have I ever used them but when the AI doesn't know something it REALLY doesn't know it, like it goes from being insanely useful to utterly useless). > I use AI autocomplete 0% of the time as I found that workflow was not as effective as me just writing code, but most of my most successful work using AI is a chat dialogue where I'm letting it build large swaths of the project a file or parts of a file at a time, with me reviewing and coaching. Me too, the way I use it is more like pair programming.
Bjartr 2y ago
Interesting that you find the conversational approach effective. For me, I'd say 9 out of 10 code conversations get stuck in a loop with me telling the AI the next suggested iteration didn't actually change anything or changed it back to something that was already broken. Do you not experience that so often, of do you have a way to escape that?
- bequanna 2y ago
  I encounter that issue when the chat becomes too long. Starting a new chat with context and asking your question again typically works for me.
apwell23 2y ago
> move multi thousand line codebases between languages i am more curious about why someone do would this
andsoitis 2y ago
> I'm perfectly fine telling the tool I use its errors and working side by side with it like it was another person. This is key. Traditional computing systems are deterministic machines, but AI is a probabilistic machine. So the way you interact and the range, precision, and perspective of the output stretches over a different problem/solution space.
tivert 2y ago
> ...and I can review code very quickly so the overall productivity boost has been great. Color me skeptical. After a certain point, greater speed is achieved by sacrificing accuracy and comprehension. So, "I can review code very quickly" starts to sound like "I don't read, I skim." IMHO, reviewing code is one of the parts of the job that sucks, so I see "AI" as a wonderful technology to improve our lives by replacing fun with chores.
zkry 2y ago
I'm curious coming from the other end. I guess I can totally understand certain use cases where I'm generating fairly simple, self contained code in a language I'm unfamiliar with being good. But surely you must have experienced something where you're literally fighting with the model, where it continuously repeats its mistakes, and fixing a mistake in one place, breaks something else, and you can't seem to escape this loop. You then get desperate, invoking magic phrases like "you think through your problems step by step", or "you are a senior developer", only for it to loose the entire thread of the conversation. Then the worst part is when you finally give up, your mental state of the problem is no better than when you first started off.
- Vegenoid 2y ago
  This is my experience. I’d love to see some full streams of people building whole useful apps from scratch with an LLM, does anyone have any good examples?
Myrmornis 2y ago
I think it's bimodal because there's a roughly bimodal distribution of high level attitudes among programmers. There's one clump that are willing to be humble and interact with the AI in a thoughtful, careful manner, acknowledging that it might be smarter than them (e.g. see Terry Tao's comments regarding mathematics usage about how in order to get good results he takes care with what he puts in (and imagine what "care" means for a professional mathematician!)) and there's another clump who aren't.
DeathArrow 2y ago
>My theory is the willingness to baby sit and the modality. I'm perfectly fine telling the tool I use its errors and working side by side with it like it was another person. In my experience, baby sitting the AI takes to much time and effort. I'd rather do it myself and use AI for tasks I don't have to babysit.
hitradostava 2y ago
I agree with you and its confusing to me. I do think there is a lot of emotion at play here - rather than cold rationality. Using LLM based tools effectively requires a change in workflow that a lot of people aren't ready to try. Everyone can share their anecdote of how an LLM has produced stupid or buggy code, but there is way too much focus on what we are now, rather than the direction of travel. I think existing models are already sufficient, its just we need to improve the feedback loop. A lot of the corrections / direction I make to LLM produced code could 100% be done by a better LLM agent. In the next year I can imagine tooling that: - lets me interact fully via voice - a separate "architecture" agent ensures that any produced code is in line with the patterns in a particular repo - compile and runtime errors are automatically fed back in and automatically fixed - a refactoring workflow mode, where the aim is to first get tests written, then get the code working, and then get the code efficient, clean and with repo patterns I'm excited by this direction of travel, but I do think it will fundamentally change software engineering in a way that is scary.
- RheingoldRiver 2y ago
  > Using LLM based tools effectively requires a change in workflow that a lot of people aren't ready to try This is a REALLY good summary of it I think. If you lose your patience with people, you'll lose your patience with AI tooling, because AI interaction is fundamentally so similar to interacting with other people
  - hitradostava 2y ago
    Exactly, and LLM based tools can be very frustrating right now - but if you view the tooling as a very fast junior developer with very broad but shallow knowledge then you can develop a workflow which for many (but not all) tasks is much much faster writing code by hand.
Tainnor 2y ago
> I've created new apps from scratch with it. I don't think that "creating new apps from scratch" should be the benchmark. Unless you're doing something very novel, creating a new app/service is rather formulaic. Many frameworks even have templates / generators for that sort of thing. LLMs are maybe just better generators - which is not useless, but it's not where the real complexity of software development lies. The success stories I am looking for are things like "I migrated a Java 6 codebase with a legacy app server to Java 21", "I ripped out this unsupported library across the project and replaced it with a better one" or "I refactored the codebase so that the database access is in its own layer". If LLMs can do those tasks reliably, then I'll have another look.
- ta12653421 2y ago
  Well, for the latter parts they do this already by a perfection of 90%+, i'd say - finally you are glueing together the pieces you get from the LLM. ClaudeAI Pro rocks, since you can create Projects/Worksapces and put documents into. Let me ask: How much did you already work with LLM like Claude?
torginus 2y ago
I've tried using AI(Claude) to do refactors/move code between languages, and in my experience, it has the tendency to go off the rails and just start making up code that does something similar, essentially doing a rewrite that never works.
- disgruntledphd2 2y ago
  I like to believe (it may not be true though), that the AI has learned what code actually exists in the wild, and is doing what all of us end up doing when trying to refactor a system we don't understand, writing new, similarish code as writing code is more fun than reading it.
IgorPartola 2y ago
> I've used AI to move multi thousand line codebases between languages. And are you certain you’ve reviewed all use cases to make sure no errors were introduced? I recently tried using Google’s AI assistant for some basic things like creating a function that could parse storage size in the format of 12KB, or 34TB into an actual number of bytes. It confidently gave me amount, units = s.split() which just is not correct. Even added a comment explaining what that line is meant to do. This was an obvious case that just didn’t work. But imagine it did work but flew into an infinite loop on an input like “12KB7” or some such.
ta12653421 2y ago
+++1 THIS! exactly my style of working and how i think about that: i've also not enabled/installed CoPilot or similar, just AutoSuggestion by default in VS.NET But i use LLM heavily to get rid off all the exhausting tasks, and to generate ideas what to improve in some larger code blocks so i dont have to rewrite/refactor it on my own. It boosts my productivity by 10x at least.
mhuffman 2y ago
>I've created new apps from scratch with it. Would mind sharing those apps to view. Not the code, just the apps. I have a suspicion about the bi-modal experience.
DirkH 2y ago
I'm convinced what we are witnessing is that there are genius level engineers (lots of them) that are and many have always been sub-par communicators. I think being a good communicator tracks really well with how much someone can get out of LLMs (as does engineering competency. You need both). Great but not genius engineers who are also great communicators may broadly outperform people with only technical genius soon, but that's speculation on my part.
tartoran 2y ago
> Im actually very curious why AI use is such a bi-modal experience. I've used AI to move multi thousand line codebases between languages. I've created new apps from scratch with it. I think this depends on the nature of your work. I've been successful with using LLMS for creating things from scratch, for myself, especially in a domain I was not familiar with and am quite happy with that. Things like proof of concepts or exploring a library or a framework. But in my current work setting, relying on LLMS to do production work is only somewhat helpful here and there but nowhere near as helpful as in the first case. In some cases it hallucinated so close to what it was supposed to do that it introduced a bug I would have never created had I not used LLMs and that took a lot of effort to spot.
alaaalawi 2y ago
MMV But for me at least i tend to use it for brain storming, aka intial sailing through a subject/topic/task, getting intial idea. the idea is to use as an admin who is guided by you throgh chatting. for example im given a task to translate a user description/requirement to pull something from the database. like (simplistic example) what are the top grossing films by category within each rating. so igive the AI the database tables schema and give it literally the user requirement. and see what it gave back and compare it with how I'll do it. ask it more for optimizations what else can be done more.... etc.. keep chating with the AI until I'm bored ;)
- SergeyHack 2y ago
  The same here. It helps my slow autistic context switching a lot. It's easier for me to recognize the ideas, concepts, keywords it has written than to recall them from memory when my "memory cache" is empty.

__float 2y ago

I'm not sure how many people are like me, but my attempts to use Copilot have largely been the context of writing code as usual, occasionally getting end-of-line or handful-of-lines completions from it. I suspect there's probably a bigger shift needed, but I haven't seen anyone (besides AI "influencers" I don't trust..?) showing what their day-to-day workflows look like. Is there a Vimcasts equivalent for learning the AI editor tips and tricks?

sbarre 2y ago
Have you tried the chat mode? The autocomplete is somewhere between annoying and underwhelming for me, but the chat is super useful. Being able to just describe what you're thinking or what you're trying to do and having a bespoke code sample just show up (based on the code in your editor) that you can then either copy/paste in, cherry-pick from or just get inspired by, has been a great productivity booster.. Treat it like a pair programmer or a rubber duck and you might have a better experience. I did!
whimsicalism 2y ago
Yeah using a chat interface

zeroonetwothree 2y ago

I guess for me it actually takes longer to review code than to write it. So maybe that’s some of the difference.

0xFACEFEED 2y ago

As a programmer of over 20 years - this is terrifying. I'm willing to accept that I just have "get off my lawn" syndrome or something. But the idea of letting an LLM write/move large swaths of code seems so incredibly irresponsible. Whenever I sit down to write some code, be it a large implementation or a small function, I think about what other people (or future versions of myself) will struggle with when interacting with the code. Is it clear and concise? Is it too clever? Is it too easy to write a subtle bug when making changes? Have I made it totally clear that X is relying on Y dangerous behavior by adding a comment or intentionally making it visible in some other way? It goes the other way too. If I know someone well (or their style) then it makes evaluating their code easier. The more time I spend in a codebase the better idea I have of what the writer was trying to do. I remember spending a lot of time reading the early Redis codebase and got a pretty good sense of how Salvatore thinks. Or altering my approaches to code reviews depending on which coworker was submitting it. These weren't things I were doing out of desire but because all non-trivial code has so much subtlety; it's just the nature of the beast. So the thought of opening up a codebase that was cobbled together by an AI is just scary to me. Subtle bugs and errors would be equally distributed across the whole thing instead of where the writer was less competent (as is often the case). The whole thing just sounds like a gargantuan mess. Change my mind.

bongodongobob 2y ago
You. Can. Write. Tests.
- ok_dad 2y ago
  Tests haven’t saved us so far, humans have been writing tests that passed for software with bugs for decades.
- lanternfish 2y ago
  Tests aren't a full solution for all the considerations of the above post.
- blitzar 2y ago
  Just let the LLM do that too.

rqmedes 2y ago

I agree. I am in a very senior role and find that working with AI the same way you do I am many times more productive. Months of work becomes days or even hours of work

bongodongobob 2y ago

My theory is grammatical correctness and specificity. I see a lot of people prompt like this: "use python to write me a prog that does some dice rolls and makes a graph" Vs "Create a Python program that generates random numbers to simulate a series of dice rolls. Export a graph of the results in PNG format." Information theory requires that you provide enough actual information. There is a minimum amount of work to supply the input. Otherwise, the gaps will get filled in with noise, working, what you want, or not. For example, maybe someday you could say "write me an OS" and it would work. However, to get exactly what you want, you still have to specify it. You can only compress so far.

2024user 2y ago

If you're doing something that appears in it's training model a lot, like building a twitter clone, then it is great. If you're using something brand new like react router 7 then it makes mistakes

xk_id 2y ago

> I'm actually very curious why AI use is such a bi-modal experience. My conspiracy theory is that the positive experiences are exaggerated and come from investors in the Nvidia stock.

Nullabillity 2y ago
That's a very important caveat. In our modern economy it's difficult to not be a shill in some way, shape, or form, even if you don't quite realize it consciously. It's honestly one of the most depressing things about the stock market.
- xk_id 2y ago
  I credit my past interest in cryptocurrencies for educating me about the essence of the stock market in its purest form. And in fact there are painful parallels with the AI bubble.
- malux85 2y ago
  Theres a big difference between being a happy customer and being a shill.
  - Nullabillity 2y ago
    Holding stock is not being a "happy customer". I may be happy with the headset that I bought, but the difference is that I don't make money if you buy an identical one.
    - malux85 2y ago
      I wasnt talking about holding stock, I was responding to this comment you said: > In our modern economy it's difficult to not be a shill in some way, shape, or form, even if you don't quite realize it consciously. Oxford dictionary defines a shill as "an accomplice of a confidence trickster or swindler who poses as a genuine customer to entice or encourage others." So the difference between someone shilling and being a satisfied customer is an intent to decieve. How is it "difficult to not pose as a genuine customer to entice or encourage others" ?

foobiekr 2y ago

The most likely explanation is that the code you are writing has low information density and is stringing things together the same way many existing apps have already done. That isn’t a judgement but trying to use the ai code completion tools for complex systems tasks is almost always a disaster.

fullstackchris 2y ago
Not sure what you mean by "complex systems tasks" but most of the leading models have helped me with writing concurrent go code just fine. Not sure if that counts as "complex" enough. However this was prompting, not completion. Obviously I expect something like copilot to pick the normie non-concurrent implementation
- foobiekr 2y ago
  It doesn’t. You’re kind of illustrating what I’m getting at.

dools 2y ago

> I'm actually very curious why AI use is such a bi-modal experience I think it's just that it's better at some things than others. Lucky for people who happen to be working in python/node/php/bash/sql/java probably unlucky for people writing Go and Rust (I'm hypothesising because I don't know Go or Rust nor have I ever used them but when the AI doesn't know something it REALLY doesn't know it, like it goes from being insanely useful to utterly useless). > I use AI autocomplete 0% of the time as I found that workflow was not as effective as me just writing code, but most of my most successful work using AI is a chat dialogue where I'm letting it build large swaths of the project a file or parts of a file at a time, with me reviewing and coaching. Me too, the way I use it is more like pair programming.

Bjartr 2y ago

Interesting that you find the conversational approach effective. For me, I'd say 9 out of 10 code conversations get stuck in a loop with me telling the AI the next suggested iteration didn't actually change anything or changed it back to something that was already broken. Do you not experience that so often, of do you have a way to escape that?

bequanna 2y ago
I encounter that issue when the chat becomes too long. Starting a new chat with context and asking your question again typically works for me.

apwell23 2y ago

> move multi thousand line codebases between languages i am more curious about why someone do would this

andsoitis 2y ago

> I'm perfectly fine telling the tool I use its errors and working side by side with it like it was another person. This is key. Traditional computing systems are deterministic machines, but AI is a probabilistic machine. So the way you interact and the range, precision, and perspective of the output stretches over a different problem/solution space.

tivert 2y ago

> ...and I can review code very quickly so the overall productivity boost has been great. Color me skeptical. After a certain point, greater speed is achieved by sacrificing accuracy and comprehension. So, "I can review code very quickly" starts to sound like "I don't read, I skim." IMHO, reviewing code is one of the parts of the job that sucks, so I see "AI" as a wonderful technology to improve our lives by replacing fun with chores.

zkry 2y ago

I'm curious coming from the other end. I guess I can totally understand certain use cases where I'm generating fairly simple, self contained code in a language I'm unfamiliar with being good. But surely you must have experienced something where you're literally fighting with the model, where it continuously repeats its mistakes, and fixing a mistake in one place, breaks something else, and you can't seem to escape this loop. You then get desperate, invoking magic phrases like "you think through your problems step by step", or "you are a senior developer", only for it to loose the entire thread of the conversation. Then the worst part is when you finally give up, your mental state of the problem is no better than when you first started off.

Vegenoid 2y ago
This is my experience. I’d love to see some full streams of people building whole useful apps from scratch with an LLM, does anyone have any good examples?

Myrmornis 2y ago

I think it's bimodal because there's a roughly bimodal distribution of high level attitudes among programmers. There's one clump that are willing to be humble and interact with the AI in a thoughtful, careful manner, acknowledging that it might be smarter than them (e.g. see Terry Tao's comments regarding mathematics usage about how in order to get good results he takes care with what he puts in (and imagine what "care" means for a professional mathematician!)) and there's another clump who aren't.

DeathArrow 2y ago

>My theory is the willingness to baby sit and the modality. I'm perfectly fine telling the tool I use its errors and working side by side with it like it was another person. In my experience, baby sitting the AI takes to much time and effort. I'd rather do it myself and use AI for tasks I don't have to babysit.

hitradostava 2y ago

I agree with you and its confusing to me. I do think there is a lot of emotion at play here - rather than cold rationality. Using LLM based tools effectively requires a change in workflow that a lot of people aren't ready to try. Everyone can share their anecdote of how an LLM has produced stupid or buggy code, but there is way too much focus on what we are now, rather than the direction of travel. I think existing models are already sufficient, its just we need to improve the feedback loop. A lot of the corrections / direction I make to LLM produced code could 100% be done by a better LLM agent. In the next year I can imagine tooling that: - lets me interact fully via voice - a separate "architecture" agent ensures that any produced code is in line with the patterns in a particular repo - compile and runtime errors are automatically fed back in and automatically fixed - a refactoring workflow mode, where the aim is to first get tests written, then get the code working, and then get the code efficient, clean and with repo patterns I'm excited by this direction of travel, but I do think it will fundamentally change software engineering in a way that is scary.

RheingoldRiver 2y ago
> Using LLM based tools effectively requires a change in workflow that a lot of people aren't ready to try This is a REALLY good summary of it I think. If you lose your patience with people, you'll lose your patience with AI tooling, because AI interaction is fundamentally so similar to interacting with other people
- hitradostava 2y ago
  Exactly, and LLM based tools can be very frustrating right now - but if you view the tooling as a very fast junior developer with very broad but shallow knowledge then you can develop a workflow which for many (but not all) tasks is much much faster writing code by hand.

Tainnor 2y ago

> I've created new apps from scratch with it. I don't think that "creating new apps from scratch" should be the benchmark. Unless you're doing something very novel, creating a new app/service is rather formulaic. Many frameworks even have templates / generators for that sort of thing. LLMs are maybe just better generators - which is not useless, but it's not where the real complexity of software development lies. The success stories I am looking for are things like "I migrated a Java 6 codebase with a legacy app server to Java 21", "I ripped out this unsupported library across the project and replaced it with a better one" or "I refactored the codebase so that the database access is in its own layer". If LLMs can do those tasks reliably, then I'll have another look.

ta12653421 2y ago
Well, for the latter parts they do this already by a perfection of 90%+, i'd say - finally you are glueing together the pieces you get from the LLM. ClaudeAI Pro rocks, since you can create Projects/Worksapces and put documents into. Let me ask: How much did you already work with LLM like Claude?

torginus 2y ago

I've tried using AI(Claude) to do refactors/move code between languages, and in my experience, it has the tendency to go off the rails and just start making up code that does something similar, essentially doing a rewrite that never works.

disgruntledphd2 2y ago
I like to believe (it may not be true though), that the AI has learned what code actually exists in the wild, and is doing what all of us end up doing when trying to refactor a system we don't understand, writing new, similarish code as writing code is more fun than reading it.

IgorPartola 2y ago

> I've used AI to move multi thousand line codebases between languages. And are you certain you’ve reviewed all use cases to make sure no errors were introduced? I recently tried using Google’s AI assistant for some basic things like creating a function that could parse storage size in the format of 12KB, or 34TB into an actual number of bytes. It confidently gave me amount, units = s.split() which just is not correct. Even added a comment explaining what that line is meant to do. This was an obvious case that just didn’t work. But imagine it did work but flew into an infinite loop on an input like “12KB7” or some such.

ta12653421 2y ago

+++1 THIS! exactly my style of working and how i think about that: i've also not enabled/installed CoPilot or similar, just AutoSuggestion by default in VS.NET But i use LLM heavily to get rid off all the exhausting tasks, and to generate ideas what to improve in some larger code blocks so i dont have to rewrite/refactor it on my own. It boosts my productivity by 10x at least.

mhuffman 2y ago

>I've created new apps from scratch with it. Would mind sharing those apps to view. Not the code, just the apps. I have a suspicion about the bi-modal experience.

DirkH 2y ago

I'm convinced what we are witnessing is that there are genius level engineers (lots of them) that are and many have always been sub-par communicators. I think being a good communicator tracks really well with how much someone can get out of LLMs (as does engineering competency. You need both). Great but not genius engineers who are also great communicators may broadly outperform people with only technical genius soon, but that's speculation on my part.

tartoran 2y ago

> Im actually very curious why AI use is such a bi-modal experience. I've used AI to move multi thousand line codebases between languages. I've created new apps from scratch with it. I think this depends on the nature of your work. I've been successful with using LLMS for creating things from scratch, for myself, especially in a domain I was not familiar with and am quite happy with that. Things like proof of concepts or exploring a library or a framework. But in my current work setting, relying on LLMS to do production work is only somewhat helpful here and there but nowhere near as helpful as in the first case. In some cases it hallucinated so close to what it was supposed to do that it introduced a bug I would have never created had I not used LLMs and that took a lot of effort to spot.

alaaalawi 2y ago

MMV But for me at least i tend to use it for brain storming, aka intial sailing through a subject/topic/task, getting intial idea. the idea is to use as an admin who is guided by you throgh chatting. for example im given a task to translate a user description/requirement to pull something from the database. like (simplistic example) what are the top grossing films by category within each rating. so igive the AI the database tables schema and give it literally the user requirement. and see what it gave back and compare it with how I'll do it. ask it more for optimizations what else can be done more.... etc.. keep chating with the AI until I'm bored ;)

SergeyHack 2y ago
The same here. It helps my slow autistic context switching a lot. It's easier for me to recognize the ideas, concepts, keywords it has written than to recall them from memory when my "memory cache" is empty.