6 ms·

> AI is only as smart as the human handling it. I think I'm slowly coming around to this viewpoint too. I really just couldn't understand how so many people we

> AI is only as smart as the human handling it. I think I'm slowly coming around to this viewpoint too. I really just couldn't understand how so many people were having widely different experiences. AI isn't magic; how could I have expected all the people I've worked with who struggle to explain stuff to team members, who have near perfect context, to manage to get anything valuable across to an AI? I was original pretty optimistic that AI would allow most engineers to operate at a higher level but it really seems like instead it's going to massively exacerbate the difference between an ok engineer and a great engineer. Not really sure how I feel about that yet but at-least I understand now why some people think the stuff is useless.

katbyte 1y ago
It’s like the difference between someone who can search the internet or codebase well bs someone who can’t Using search engines is a skill
jerf 1y ago
I've been struggling to apply AI on any large scale at work. I was beginning to wonder if it was me. But then my wife sort of handed me a project that previously I would have just said no to, a particular Android app for the family. I have instances of all the various Android technologies under my belt, that is, I've used GUI toolkits, I've used general purpose programming languages, I've used databases, etc, but with the possible exception of SQLite (which even that is accessed through an ORM), I don't know any of the specific technologies involved with Android now. I have never used Kotlin; I've got enough experience that I can pretty much piece it together when I'm reading it but I can't write it. Never used the Android UI toolkit, services, permissions, media APIs, ORMs, build system, etc. I know from many previous experiences that A: I could definitely learn how to do this but B: it would be a many-week project and in the end I wouldn't really be able to leverage any of the Android knowledge I would get for much else. So I figured this was a good chance to take this stuff for a spin in a really hard way. I'm about eight hours in and nearly done enough for the family; I need about another 2 hours to hit that mark, maybe 4 to really polish it. Probably another 8-12 hours and I'd have it brushed up to a rough commercial product level for a simple, single-purpose app. It's really impressive. And I'm now convinced it's not just that I'm too old a fogey to pick it up, which is, you know, a bit of a relief. It's just that it works really well in some domains, and not so much in others. My current work project is working through decades of organically-grown cruft owned by 5 different teams, most of which don't even have a person on them that understands the cruft in question, and trying to pull it all together into one system where it belongs. I've been able to use AI here and there for some stuff that is still pretty impressive, like translating some stuff into psuedocode for my reference, and AI-powered autocomplete is definitely impressive when it correctly guesses the next 10 lines I was going to type effectively letter-for-letter. But I haven't gotten that large-scale win where I just type a tiny prompt in and see the outsized results from it. I think that's because I'm working in a domain where the code I'm writing is already roughly the size of the prompt I'd have to give, at least in terms of the "payload" of the work I'm trying to do, because of the level of detail and maturity of the code base. There's no single sentence I can type that an AI can essentially decompress into 250 lines of code, pulling in the correct 4 new libraries, and adding it all to the build system the way that Gemini in Android Studio could decompress "I would like to store user settings with a UI to set the user's name, and then display it on the home page". I think I recommend this approach to anyone who wants to give this approach a fair shake - try it in a language and environment you know nothing about and so aren't tempted to keep taking the wheel. The AI is almost the only tool I have in that environment, certainly the only one for writing code, so I'm forced to really exercise the AI.
- smokel 1y ago
  What you are describing also seems to align with the idea that greenfield projects are well-suited for AI, whereas brownfield projects are considerably more challenging.
  - QuercusMax 1y ago
    Brownfield projects are more challenging because of all the context and decisions that went into building things that are not directly defined in the code. I suspect that well-engineered projects with plenty of test coverage and high-quality documentation will be easier to use AI on, just like they're easier for humans to comprehend. But you need to have somebody with the big picture still who can make sure that you don't just turn things into a giant mess once less disciplined people start using AI on a project.
    - smokel 1y ago
      Also, as soon as the code no longer fits within the context window of an LLM, one must resort to RAG-based solutions, which often leads to a significant decline in quality.
    - ab2525 1y ago
      Well said - language (text input) is actually the vehicle you have to transfer neural state to the engine. When you are working in a greenfield project or pure-vibe project, you can get away with most of that neural state being in the "default" probability mode. But in a legacy project, you need significantly more context to contrain the probability distributions a lot closer to the decisions which were made historically otherwise you quickly get into spaghetti-ville as the AI tries to drag the codebase towards its natural ruts.
- thewebguyd 1y ago
  > try it in a language and environment you know nothing about and so aren't tempted to keep taking the wheel. That's a good insights. Its almost like to use AI tools effectively, one needs to stop caring about the little things you'd get caught up in if you were already familiar and proficient in a stack. Style guidelines, a certain idiomatic way to do things, naming conventions, etc. A lot like how I've stopped organizing digital files into folders, sub folders etc (along with other content) and now I just just rely on search. Everything is a flat structure, I don't care where its stored or how it's organized as long as I can just search for it, that's what the computer is for, to keep track for me so I don't have to waste time organizing it myself. Like wise for the code Generative AI produces. I don't need to care about the code itself. As long as its correct, not insecure, and performant, it's fine. It's not 100% there yet, I still do have to go in and touch the code, but ideally I shouldn't have to, nor should I have to care what the actual code looks like, just the result of it. Let the computer manage that, not me. My role should be the system design and specification, not writing the code.
btucker 1y ago
I've been starting to think of it like this: Great Engineer + AI = Great Engineer++ (Where a great engineer isn't just someone who is a great coder, they also are a great communicator & collaborator, and love to learn) Good Engineer + AI = Good Engineer OK Engineer + AI = Mediocre Engineer
- aydyn 1y ago
  Is there a difference between "OK" and "Mediocre"?
  - Forgeties79 1y ago
    “Ok” I generally associate with being adequate but could obviously be better. “Mediocre” is just inadequate.
    - Marsymars 1y ago
      “Mediocre” is one of those words where common parlance doesn’t quite line up with the textbook definition. e.g. from the Oxford English Dictionary: “Of middling quality; neither bad nor good...”
    - furyofantares 1y ago
      Some synonyms for mediocre: decent, middling, ordinary, so-so. It can mean inadequate, but it can also mean adequate.
      - Forgeties79 1y ago
        I’m just talking about how I view/use it.
      - rerdavies 1y ago
        Another common synonym for mediocre: has no place on a software development team. Not technically correct, admittedly, but that's how I read that word in an software engineering context. Adequate is not good enough.
btown 1y ago
One of my mental models is that the notion of "effective engineer" used to mean "effective software developer" whether or not they were good at system design. Now, an "effective engineer" can be a less battle-tested software developer, but they must be good at system design. (And by system design, I don't just mean architecture diagrams: it's a personal culture of constantly questioning and innovating around "let's think critically to see what might go wrong when all these assumptions collide, and if one of them ends up being incorrect." Because AI will only suggest those things for cut-and-dry situations where a bug is apparent from a few files' context, and no ambitious idea is fully that cut-and-dry.) The set of effective engineers is thus shifting - and it's not at all a valid assumption that every formerly good developer will see their productivity skyrocket.
- ilc 1y ago
  I suspect that truly battle tested engineers will go up in value. I don't think that it lowers the bar there, if anything the bar is far harsher. If I'm doing normal coding I make X choices per time period, with Y impacts. With AI X will go up and the Y / X ratio may ALSO go up, so making more decisions of higher leverage!
jgilias 1y ago
This fits my observations as well. With the exception that it’s sometimes the really sharp engineers who can do wonders themselves who aren’t really great at communication. AI really needs you to be verbose, and a lot of people just can’t.
oblio 1y ago
As a summary, any scaling tech greatly exacerbates differences. Nassim Taleb is the prophet of our times and he doesn't get enough credit.
felipeerias 1y ago
At the moment, AI tools are particularly useful for people who feel comfortable browsing through large amounts of text, intuitively nudging the machine this way and that until arriving at a valuable outcome. However, that way of working can be exasperating for those who prefer a more deterministic approach, and who may feel frustrated by the sheer amount of slightly incorrect stuff being generated by the machine.

katbyte 1y ago

It’s like the difference between someone who can search the internet or codebase well bs someone who can’t Using search engines is a skill

jerf 1y ago

I've been struggling to apply AI on any large scale at work. I was beginning to wonder if it was me. But then my wife sort of handed me a project that previously I would have just said no to, a particular Android app for the family. I have instances of all the various Android technologies under my belt, that is, I've used GUI toolkits, I've used general purpose programming languages, I've used databases, etc, but with the possible exception of SQLite (which even that is accessed through an ORM), I don't know any of the specific technologies involved with Android now. I have never used Kotlin; I've got enough experience that I can pretty much piece it together when I'm reading it but I can't write it. Never used the Android UI toolkit, services, permissions, media APIs, ORMs, build system, etc. I know from many previous experiences that A: I could definitely learn how to do this but B: it would be a many-week project and in the end I wouldn't really be able to leverage any of the Android knowledge I would get for much else. So I figured this was a good chance to take this stuff for a spin in a really hard way. I'm about eight hours in and nearly done enough for the family; I need about another 2 hours to hit that mark, maybe 4 to really polish it. Probably another 8-12 hours and I'd have it brushed up to a rough commercial product level for a simple, single-purpose app. It's really impressive. And I'm now convinced it's not just that I'm too old a fogey to pick it up, which is, you know, a bit of a relief. It's just that it works really well in some domains, and not so much in others. My current work project is working through decades of organically-grown cruft owned by 5 different teams, most of which don't even have a person on them that understands the cruft in question, and trying to pull it all together into one system where it belongs. I've been able to use AI here and there for some stuff that is still pretty impressive, like translating some stuff into psuedocode for my reference, and AI-powered autocomplete is definitely impressive when it correctly guesses the next 10 lines I was going to type effectively letter-for-letter. But I haven't gotten that large-scale win where I just type a tiny prompt in and see the outsized results from it. I think that's because I'm working in a domain where the code I'm writing is already roughly the size of the prompt I'd have to give, at least in terms of the "payload" of the work I'm trying to do, because of the level of detail and maturity of the code base. There's no single sentence I can type that an AI can essentially decompress into 250 lines of code, pulling in the correct 4 new libraries, and adding it all to the build system the way that Gemini in Android Studio could decompress "I would like to store user settings with a UI to set the user's name, and then display it on the home page". I think I recommend this approach to anyone who wants to give this approach a fair shake - try it in a language and environment you know nothing about and so aren't tempted to keep taking the wheel. The AI is almost the only tool I have in that environment, certainly the only one for writing code, so I'm forced to really exercise the AI.

smokel 1y ago
What you are describing also seems to align with the idea that greenfield projects are well-suited for AI, whereas brownfield projects are considerably more challenging.
- QuercusMax 1y ago
  Brownfield projects are more challenging because of all the context and decisions that went into building things that are not directly defined in the code. I suspect that well-engineered projects with plenty of test coverage and high-quality documentation will be easier to use AI on, just like they're easier for humans to comprehend. But you need to have somebody with the big picture still who can make sure that you don't just turn things into a giant mess once less disciplined people start using AI on a project.
  - smokel 1y ago
    Also, as soon as the code no longer fits within the context window of an LLM, one must resort to RAG-based solutions, which often leads to a significant decline in quality.
  - ab2525 1y ago
    Well said - language (text input) is actually the vehicle you have to transfer neural state to the engine. When you are working in a greenfield project or pure-vibe project, you can get away with most of that neural state being in the "default" probability mode. But in a legacy project, you need significantly more context to contrain the probability distributions a lot closer to the decisions which were made historically otherwise you quickly get into spaghetti-ville as the AI tries to drag the codebase towards its natural ruts.
thewebguyd 1y ago
> try it in a language and environment you know nothing about and so aren't tempted to keep taking the wheel. That's a good insights. Its almost like to use AI tools effectively, one needs to stop caring about the little things you'd get caught up in if you were already familiar and proficient in a stack. Style guidelines, a certain idiomatic way to do things, naming conventions, etc. A lot like how I've stopped organizing digital files into folders, sub folders etc (along with other content) and now I just just rely on search. Everything is a flat structure, I don't care where its stored or how it's organized as long as I can just search for it, that's what the computer is for, to keep track for me so I don't have to waste time organizing it myself. Like wise for the code Generative AI produces. I don't need to care about the code itself. As long as its correct, not insecure, and performant, it's fine. It's not 100% there yet, I still do have to go in and touch the code, but ideally I shouldn't have to, nor should I have to care what the actual code looks like, just the result of it. Let the computer manage that, not me. My role should be the system design and specification, not writing the code.

btucker 1y ago

I've been starting to think of it like this: Great Engineer + AI = Great Engineer++ (Where a great engineer isn't just someone who is a great coder, they also are a great communicator & collaborator, and love to learn) Good Engineer + AI = Good Engineer OK Engineer + AI = Mediocre Engineer

aydyn 1y ago
Is there a difference between "OK" and "Mediocre"?
- Forgeties79 1y ago
  “Ok” I generally associate with being adequate but could obviously be better. “Mediocre” is just inadequate.
  - Marsymars 1y ago
    “Mediocre” is one of those words where common parlance doesn’t quite line up with the textbook definition. e.g. from the Oxford English Dictionary: “Of middling quality; neither bad nor good...”
  - furyofantares 1y ago
    Some synonyms for mediocre: decent, middling, ordinary, so-so. It can mean inadequate, but it can also mean adequate.
    - Forgeties79 1y ago
      I’m just talking about how I view/use it.
    - rerdavies 1y ago
      Another common synonym for mediocre: has no place on a software development team. Not technically correct, admittedly, but that's how I read that word in an software engineering context. Adequate is not good enough.

btown 1y ago

One of my mental models is that the notion of "effective engineer" used to mean "effective software developer" whether or not they were good at system design. Now, an "effective engineer" can be a less battle-tested software developer, but they must be good at system design. (And by system design, I don't just mean architecture diagrams: it's a personal culture of constantly questioning and innovating around "let's think critically to see what might go wrong when all these assumptions collide, and if one of them ends up being incorrect." Because AI will only suggest those things for cut-and-dry situations where a bug is apparent from a few files' context, and no ambitious idea is fully that cut-and-dry.) The set of effective engineers is thus shifting - and it's not at all a valid assumption that every formerly good developer will see their productivity skyrocket.

ilc 1y ago
I suspect that truly battle tested engineers will go up in value. I don't think that it lowers the bar there, if anything the bar is far harsher. If I'm doing normal coding I make X choices per time period, with Y impacts. With AI X will go up and the Y / X ratio may ALSO go up, so making more decisions of higher leverage!

jgilias 1y ago

This fits my observations as well. With the exception that it’s sometimes the really sharp engineers who can do wonders themselves who aren’t really great at communication. AI really needs you to be verbose, and a lot of people just can’t.

oblio 1y ago

As a summary, any scaling tech greatly exacerbates differences. Nassim Taleb is the prophet of our times and he doesn't get enough credit.

felipeerias 1y ago

At the moment, AI tools are particularly useful for people who feel comfortable browsing through large amounts of text, intuitively nudging the machine this way and that until arriving at a valuable outcome. However, that way of working can be exasperating for those who prefer a more deterministic approach, and who may feel frustrated by the sheer amount of slightly incorrect stuff being generated by the machine.