5 ms·

What do you do if your app has a bug that your LLM isn't able to fix? is your coding experience enough to fix it, or do you ship with bugs hoping customers won'

What do you do if your app has a bug that your LLM isn't able to fix? is your coding experience enough to fix it, or do you ship with bugs hoping customers won't mind?

instalabs 2y ago
You start over from scratch /s(50%)
epolanski 2y ago
What's the point of this question? Everybody ships nasty bugs in production that he himself might find impossible to debug, everybody. Thus he will do the very same thing me, you or anybody else on this planet do, find a second pair of eyes, virtually or not, paying or not.
- LunaSea 2y ago
  > Everybody ships nasty bugs in production that he himself might find impossible to debug, everybody. No.
  - monooso 2y ago
    Some people haven't realised it yet.
    - LunaSea 2y ago
      Which would be a lot better than knowingly releasing in production code with important defects.
  - epolanski 2y ago
    I haven't seen anybody, regardless of org, procedures or whatever to never ship a bug he himself could not debug. Hell, I don't even see it happening in OS space with dozens of eyes on years-long PRs. It just happens. At some point you'll write and ship a bug that you yourself can't debug in an appropriate time frame alone and needs more eyes.
- lucianbr 2y ago
  Presumably what is possible for a person with 6 months of experience is rather limited. The idea as I understand it is that he achieved apps that he would not be able to write by himself, with the help of AI. That means that it is possible to have bugs that would be reasonable to fix for someone who built the app using their own knowledge, but for the junior they may be too hard. This is a novel situation. Just because everyone has problems sometimes does not mean problems are all the same, all the same difficulty. Like if I was building Starship, and I ran into some difficult problem, I would most likely give up, as I am way out of my league. I couldn't build a model rocket. I know nothing about rockets. My situation would not be the same as of any rocket engineer. All problems and all situations and all people are not the same, and they are not made the same by AI, despite claims to the contrary. These simplifications/generalisations "we are all stochastic parrots" "we all make mistakes just like the llms make mistakes" "we all have bugs" "we all manage somehow" are absurd. Companies do not do interviews and promote some people and not others out of a sense of whimsy. Experience and knowledge matters. We are not all interchangable. If LLMs affect this somehow, it's to be looked at. I can't believe LLMs or devs using LLMs cand suddenly do anything, without limitations. We are not all now equal to Linus and Carmack and such.
- spaceman_2020 2y ago
  I haven’t encountered any serious bugs - mostly because I know what I’m capable of and what Sonnet is capable of. I don’t tackle things that are far too ambitious and focus on ideas I want to experiment with or ideas I can build the MVP for If I do encounter situations that Sonnet can’t fix - usually because it has outdated knowledge - I just read the latest documentation
- jajko 2y ago
  Those things are not not even comparable in the quality output and if you see them as equals this seriously harm your credibility in this topic. This won't change in next decade+. For some use cases thats good enough quality, till you have any actual issues your smart code tools can't handle. Till people start suing you because your half-baked app caused them a real, serious financial loss and they have a true vengeance in their eyes (smaller companies or individuals often take such harm from outside very personally). Any serious company paying serious bucks won't accept this, in 2024 they know darn well how bad software can bite massively back, some of them like banks or whole Silicon valley run whole business on software. But its true that there is a massive space outside such cases where this cca works, I've never worked there so can't judge.
ipaddr 2y ago
What I see is people using llm to make a new app without the bug
- willsmith72 2y ago
  There's always a bug, you just haven't found it yet
jstanley 2y ago
What does anyone do if they have a bug they don't know how to fix? Find a way to work around it.
amonith 2y ago
If customers do mind then at best it's an opportunity cost (less people will buy). Shipping with bugs > not shipping, simple as.
- namaria 2y ago
  You better hope no bugs expose you to liabilities like runaway cloud costs or mishandling sensitive data
  - amonith 2y ago
    Yeah SaaS is a different beast. Lots of areas worth hardening. Not really for customers but mainly for yourself. But desktop apps, mobile apps, self-hosted stuff, games, CLIs, libraries - you don't really have to worry about much.

What do you do if your app has a bug that your LLM isn't able to fix? is your coding experience enough to fix it, or do you ship with bugs hoping customers won't mind?

instalabs 2y ago

You start over from scratch /s(50%)

epolanski 2y ago

What's the point of this question? Everybody ships nasty bugs in production that he himself might find impossible to debug, everybody. Thus he will do the very same thing me, you or anybody else on this planet do, find a second pair of eyes, virtually or not, paying or not.

LunaSea 2y ago
> Everybody ships nasty bugs in production that he himself might find impossible to debug, everybody. No.
- monooso 2y ago
  Some people haven't realised it yet.
  - LunaSea 2y ago
    Which would be a lot better than knowingly releasing in production code with important defects.
- epolanski 2y ago
  I haven't seen anybody, regardless of org, procedures or whatever to never ship a bug he himself could not debug. Hell, I don't even see it happening in OS space with dozens of eyes on years-long PRs. It just happens. At some point you'll write and ship a bug that you yourself can't debug in an appropriate time frame alone and needs more eyes.
lucianbr 2y ago
Presumably what is possible for a person with 6 months of experience is rather limited. The idea as I understand it is that he achieved apps that he would not be able to write by himself, with the help of AI. That means that it is possible to have bugs that would be reasonable to fix for someone who built the app using their own knowledge, but for the junior they may be too hard. This is a novel situation. Just because everyone has problems sometimes does not mean problems are all the same, all the same difficulty. Like if I was building Starship, and I ran into some difficult problem, I would most likely give up, as I am way out of my league. I couldn't build a model rocket. I know nothing about rockets. My situation would not be the same as of any rocket engineer. All problems and all situations and all people are not the same, and they are not made the same by AI, despite claims to the contrary. These simplifications/generalisations "we are all stochastic parrots" "we all make mistakes just like the llms make mistakes" "we all have bugs" "we all manage somehow" are absurd. Companies do not do interviews and promote some people and not others out of a sense of whimsy. Experience and knowledge matters. We are not all interchangable. If LLMs affect this somehow, it's to be looked at. I can't believe LLMs or devs using LLMs cand suddenly do anything, without limitations. We are not all now equal to Linus and Carmack and such.
spaceman_2020 2y ago
I haven’t encountered any serious bugs - mostly because I know what I’m capable of and what Sonnet is capable of. I don’t tackle things that are far too ambitious and focus on ideas I want to experiment with or ideas I can build the MVP for If I do encounter situations that Sonnet can’t fix - usually because it has outdated knowledge - I just read the latest documentation
jajko 2y ago
Those things are not not even comparable in the quality output and if you see them as equals this seriously harm your credibility in this topic. This won't change in next decade+. For some use cases thats good enough quality, till you have any actual issues your smart code tools can't handle. Till people start suing you because your half-baked app caused them a real, serious financial loss and they have a true vengeance in their eyes (smaller companies or individuals often take such harm from outside very personally). Any serious company paying serious bucks won't accept this, in 2024 they know darn well how bad software can bite massively back, some of them like banks or whole Silicon valley run whole business on software. But its true that there is a massive space outside such cases where this cca works, I've never worked there so can't judge.

ipaddr 2y ago

What I see is people using llm to make a new app without the bug

willsmith72 2y ago
There's always a bug, you just haven't found it yet

jstanley 2y ago

What does anyone do if they have a bug they don't know how to fix? Find a way to work around it.

amonith 2y ago

If customers do mind then at best it's an opportunity cost (less people will buy). Shipping with bugs > not shipping, simple as.

namaria 2y ago
You better hope no bugs expose you to liabilities like runaway cloud costs or mishandling sensitive data
- amonith 2y ago
  Yeah SaaS is a different beast. Lots of areas worth hardening. Not really for customers but mainly for yourself. But desktop apps, mobile apps, self-hosted stuff, games, CLIs, libraries - you don't really have to worry about much.