4 ms·

I have a non-technical friend who in the last two months has bootstrapped a SaaS startup using nothing but AI. He's got just over a handful of paying customers

I have a non-technical friend who in the last two months has bootstrapped a SaaS startup using nothing but AI. He's got just over a handful of paying customers at this point on a monthly subscription[0]. I asked him to show me his process[1] after trying my hand (20 year, principal) and noticed a big difference in how we used AI: I instruct the AI how to code, he asks the AI to fix problems. In other words, I have a tendency to look at the code and ask the AI to fix it in more specific and direct ways that I want it fixed. On the other hand, if something doesn't work, my friend will copy/paste the error to the AI directly out of the dev tools console and ask the AI to fix the error. The two approaches are totally different. My lesson here is that you're not meant to debug AI generated code; hand the error off to the AI and let it fix itself. I think if you're debugging AI generated code, you're doing AI generated code wrong. If you're an experienced dev picking up AI coding, I think you need to shift your mindset entirely. Ideally, someone out there will just create a closed loop where the AI can fix itself when it finds an error (integrate some browser and autonomous test loop into Cursor, for example, and let it fix its own errors). Conclusion: if you're going to use AI to code, commit to it and use AI to fix the errors as well. Use AI for every aspect of it. [0] Yes, I'm sure there are security holes and code issues galore, but those can always be fixed later when he's proven the business model. [1] Yes, I have told him that he should create a YT channel or stream on Twitch because the content itself is super interesting how well he's been able to use AI.

Madmallard 2y ago
Yeah this works for crud apps with conventional methods for accounts email payments and not at all for anything complex especially if it isn’t a super commonly used language or framework. Try coding a single game with AI that isn’t something done 10000 times already. It actually is impossible.
- llamaimperative 2y ago
  The vast majority of code in the world is the former though
  - sarchertech 2y ago
    The last 20 years of programming tells me that this isn’t the case. This is only the case for new projects which don’t yet have users. Add users to even the simplest project and it evolves into a special snowflake with never before seen edge cases. That’s why low code solutions are great for prototyping but eventually always explode into a nightmare of complexity.
abossy 2y ago
In my experience, AI isn't very good at debugging AI-generated code. If it fails to make the right insight, it loops continuously until it's completely off the rails. I'm surprised your friend hasn't fully gotten stuck with this, as it seems like a huge risk for his startup.
- CharlieDigital 2y ago
  Having had an inside view of a YC startup that went from seed to C, I can tell you that code quality means a lot less than one would think when it comes to the early days of a startup. The biggest risk to a startup is that you get the business model wrong or you don't ship code, even if it's the code is buggy and messy.
skeeterbug 2y ago
I wonder what this codebase will look like after a year or so of doing this.
- extesy 2y ago
  Bugs will escalate from syntax errors to business logic errors ("one customer was charged twice"). There won't be anything to copy/paste, no AI will be able to fix these errors and no human will touch this codebase with a long pole.
  - iamflimflam1 2y ago
    Have you seen the job market at the moment? Humans will do a lot of things to keep a roof over their heads.
arrowsmith 2y ago
> integrate some browser and autonomous test loop into Cursor Doesn't this exist yet? It's such an obvious idea I'd be astonished if no-one has done it.
- CharlieDigital 2y ago
  They exist in separate pieces; I've not seen it integrated into one loop yet. Code gen -> show the AI an example of how it's supposed to work -> error -> code gen -> AI tries it again by itself -> Code gen
arend321 2y ago
This probably only works if you glue a bunch of high level, popular APIs together. It might work, but will be fragile and expensive.
- CharlieDigital 2y ago
  Most SaaS apps today can be done by gluing together popular APIs (e.g. Stripe, Shopify, etc.). No better or worse than hiring cheap offshore contractors to do the same, IMO.
- dartos 2y ago
  > fragile and expensive Unfortunately, that’s the most common kind of software in the saas industry anyway.
bendauphinee 2y ago
As an experienced developer, that’s also how I use it. What I’m finding is that it generally rabbit holes as I give it new errors that it’s previous fix has produced. However, usually after three or four of those kind of fixes, I can walk it back to the starting point before the initial error, and I now know how to prompt it to produce correct code, because I now have a better mental model of how the thing is supposed to work. This has been super helpful in my process of learning new things, as well as relearning things I haven’t worked with in a while.
imtringued 2y ago
This only works if there is an error message. Do you instruct the AI to fill in the code with asserts and not implemented exceptions?
- CharlieDigital 2y ago
  I was only on a session with him for like 15 minutes and he showed me his prompt history. Basically when he hits an error, he will paste the error and give a simple instruction like "I'm getting this error when I click this button: <ERROR_HERE>" and then repeat until it's fixed. Nothing special; imagine a non-technical PM giving directions to a junior dev except this junior dev codes nearly instantaneously.
jfengel 2y ago
In my experience, fixing security issues after the fact is extremely challenging. A secure system has a very different architecture, with security as its fundamental function and business logic almost an afterthought. It's not impossible to fix later. But it's often more effective to scrap and rewrite. Hopefully your proven business model has yielded enough money for that, before someone else has pwned it.
jiehong 2y ago
I’ve seen some users do that, and got stuck in a loop where the AI is saying "ah, this error is because", and not fixing it properly, or fixing it and adding a different issue by modifying part of the code that is not related at the same time. Next, the code is fixed but by added the old fixed issue. I saw that with users asking VBA code to be generated by people trying to automated part of email and excel work.
- CharlieDigital 2y ago
  It's possible, but his choices are 1) hire someone else, 2) just sit there and prompt again until it's fixed. Since he's bootstrapping this with < $50/mo, the choice is simple. Also, it may be the case that the corpus of training data with VBA is not as good as it is with React these days.
Timber-6539 2y ago
One can only imagine how many corners your friend had to cut to get to the product you call finished.
- CharlieDigital 2y ago
  He's got paying customers organic inbound by word of mouth only; there must be some value there.
  - kuschku 2y ago
    > word of mouth > must be some value https://en.wikipedia.org/wiki/Parasite_(2019_film)#Plot https://en.wikipedia.org/wiki/Parasite_(2019_film)#Plot
    - CharlieDigital 2y ago
      I don't see how these two would be connected. His SaaS solves a problem for a very specific industry that's quite small (his market research yielded about 6000 customers) so it's a small, niche industry where a lot of the small business proprietors know each other through a trade group and more or less have the same problem: the alternative solutions currently in the market are expensive, entrenched, legacy providers that operate through a POS while his solution is web-based and costs less.
plagiarist 2y ago
I have tried that and the AI gets stuck just attempting whatever and writing more code that won't even compile. I have had more success trying to get it to follow steps or examples. Maybe the language your friend is using has more examples for training, or perhaps the dynamism of some languages get it to runtime errors that have better details it can work with.
- CharlieDigital 2y ago
  React and JS so I think it has some benefits since 1) it has a large corpus of recent training data, 2) the browser gives pretty good errors. I also tried it and the biggest issue I ran into is that I'm very specific about what I want. I wanted to use `nanostores` for state and routing. Problem is that the LLM keeps using code from `react-router` instead of `@nanostores/router`. As soon as I point it out, the LLM fixes it, but the first pass code generation is almost always wrong, even using an instruction file (as documented in both Cursor and GH Copilot). That's when I realized that we are using the AI in two totally different ways: he simply doesn't care about the implementation, prop drilling, any of the technical details. None of that matters to him except that when "this button is clicked, that action happens". So however complex or inefficient or imperfect the code is, he doesn't care whereas I still have a tendency to read the code and try to ask the AI to do it in specific ways.
InvOfSmallC 2y ago
I don't know which specific LLM your friend used but pasting the error to the LLM usually ends in a endless loop where they tell you to do the same thing over and over again or the solution doesn't really work or generate another error. So maybe he was lucky or he is using a very good LLM I'm not aware of.
- CharlieDigital 2y ago
  Claude Sonnet. If your choices are to pay out of pocket for an offshore contractor and wait for weeks or pay $20/mo. for an LLM, it's pretty clear that even if you have to sit there for a few days until you get what you want, using the LLM is the better bet if you're non-technical. In either case, the code would be of questionable quality and a non-technical person would not be able to tell the difference anyways. I see it as a wash.

Madmallard 2y ago

Yeah this works for crud apps with conventional methods for accounts email payments and not at all for anything complex especially if it isn’t a super commonly used language or framework. Try coding a single game with AI that isn’t something done 10000 times already. It actually is impossible.

llamaimperative 2y ago
The vast majority of code in the world is the former though
- sarchertech 2y ago
  The last 20 years of programming tells me that this isn’t the case. This is only the case for new projects which don’t yet have users. Add users to even the simplest project and it evolves into a special snowflake with never before seen edge cases. That’s why low code solutions are great for prototyping but eventually always explode into a nightmare of complexity.

abossy 2y ago

In my experience, AI isn't very good at debugging AI-generated code. If it fails to make the right insight, it loops continuously until it's completely off the rails. I'm surprised your friend hasn't fully gotten stuck with this, as it seems like a huge risk for his startup.

CharlieDigital 2y ago
Having had an inside view of a YC startup that went from seed to C, I can tell you that code quality means a lot less than one would think when it comes to the early days of a startup. The biggest risk to a startup is that you get the business model wrong or you don't ship code, even if it's the code is buggy and messy.

skeeterbug 2y ago

I wonder what this codebase will look like after a year or so of doing this.

extesy 2y ago
Bugs will escalate from syntax errors to business logic errors ("one customer was charged twice"). There won't be anything to copy/paste, no AI will be able to fix these errors and no human will touch this codebase with a long pole.
- iamflimflam1 2y ago
  Have you seen the job market at the moment? Humans will do a lot of things to keep a roof over their heads.

arrowsmith 2y ago

> integrate some browser and autonomous test loop into Cursor Doesn't this exist yet? It's such an obvious idea I'd be astonished if no-one has done it.

CharlieDigital 2y ago
They exist in separate pieces; I've not seen it integrated into one loop yet. Code gen -> show the AI an example of how it's supposed to work -> error -> code gen -> AI tries it again by itself -> Code gen

arend321 2y ago

This probably only works if you glue a bunch of high level, popular APIs together. It might work, but will be fragile and expensive.

CharlieDigital 2y ago
Most SaaS apps today can be done by gluing together popular APIs (e.g. Stripe, Shopify, etc.). No better or worse than hiring cheap offshore contractors to do the same, IMO.
dartos 2y ago
> fragile and expensive Unfortunately, that’s the most common kind of software in the saas industry anyway.

bendauphinee 2y ago

As an experienced developer, that’s also how I use it. What I’m finding is that it generally rabbit holes as I give it new errors that it’s previous fix has produced. However, usually after three or four of those kind of fixes, I can walk it back to the starting point before the initial error, and I now know how to prompt it to produce correct code, because I now have a better mental model of how the thing is supposed to work. This has been super helpful in my process of learning new things, as well as relearning things I haven’t worked with in a while.

imtringued 2y ago

This only works if there is an error message. Do you instruct the AI to fill in the code with asserts and not implemented exceptions?

CharlieDigital 2y ago
I was only on a session with him for like 15 minutes and he showed me his prompt history. Basically when he hits an error, he will paste the error and give a simple instruction like "I'm getting this error when I click this button: <ERROR_HERE>" and then repeat until it's fixed. Nothing special; imagine a non-technical PM giving directions to a junior dev except this junior dev codes nearly instantaneously.

jfengel 2y ago

In my experience, fixing security issues after the fact is extremely challenging. A secure system has a very different architecture, with security as its fundamental function and business logic almost an afterthought. It's not impossible to fix later. But it's often more effective to scrap and rewrite. Hopefully your proven business model has yielded enough money for that, before someone else has pwned it.

jiehong 2y ago

I’ve seen some users do that, and got stuck in a loop where the AI is saying "ah, this error is because", and not fixing it properly, or fixing it and adding a different issue by modifying part of the code that is not related at the same time. Next, the code is fixed but by added the old fixed issue. I saw that with users asking VBA code to be generated by people trying to automated part of email and excel work.

CharlieDigital 2y ago
It's possible, but his choices are 1) hire someone else, 2) just sit there and prompt again until it's fixed. Since he's bootstrapping this with < $50/mo, the choice is simple. Also, it may be the case that the corpus of training data with VBA is not as good as it is with React these days.

Timber-6539 2y ago

One can only imagine how many corners your friend had to cut to get to the product you call finished.

CharlieDigital 2y ago
He's got paying customers organic inbound by word of mouth only; there must be some value there.
- kuschku 2y ago
  > word of mouth > must be some value https://en.wikipedia.org/wiki/Parasite_(2019_film)#Plot https://en.wikipedia.org/wiki/Parasite_(2019_film)#Plot
  - CharlieDigital 2y ago
    I don't see how these two would be connected. His SaaS solves a problem for a very specific industry that's quite small (his market research yielded about 6000 customers) so it's a small, niche industry where a lot of the small business proprietors know each other through a trade group and more or less have the same problem: the alternative solutions currently in the market are expensive, entrenched, legacy providers that operate through a POS while his solution is web-based and costs less.

plagiarist 2y ago

I have tried that and the AI gets stuck just attempting whatever and writing more code that won't even compile. I have had more success trying to get it to follow steps or examples. Maybe the language your friend is using has more examples for training, or perhaps the dynamism of some languages get it to runtime errors that have better details it can work with.

CharlieDigital 2y ago
React and JS so I think it has some benefits since 1) it has a large corpus of recent training data, 2) the browser gives pretty good errors. I also tried it and the biggest issue I ran into is that I'm very specific about what I want. I wanted to use `nanostores` for state and routing. Problem is that the LLM keeps using code from `react-router` instead of `@nanostores/router`. As soon as I point it out, the LLM fixes it, but the first pass code generation is almost always wrong, even using an instruction file (as documented in both Cursor and GH Copilot). That's when I realized that we are using the AI in two totally different ways: he simply doesn't care about the implementation, prop drilling, any of the technical details. None of that matters to him except that when "this button is clicked, that action happens". So however complex or inefficient or imperfect the code is, he doesn't care whereas I still have a tendency to read the code and try to ask the AI to do it in specific ways.

InvOfSmallC 2y ago

I don't know which specific LLM your friend used but pasting the error to the LLM usually ends in a endless loop where they tell you to do the same thing over and over again or the solution doesn't really work or generate another error. So maybe he was lucky or he is using a very good LLM I'm not aware of.

CharlieDigital 2y ago
Claude Sonnet. If your choices are to pay out of pocket for an offshore contractor and wait for weeks or pay $20/mo. for an LLM, it's pretty clear that even if you have to sit there for a few days until you get what you want, using the LLM is the better bet if you're non-technical. In either case, the code would be of questionable quality and a non-technical person would not be able to tell the difference anyways. I see it as a wash.