6 ms·

How we built Townie – an app that generates fullstack apps

175 points by stevekrouse 2y ago

janpaul123 2y ago
Post author here! Happy to answer any questions.
deleted 2y ago
[deleted]
syspec 2y ago
Does fullstack here mean using javascript on the backend? Is it able to generate code for other backend languages?
- janpaul123 2y ago
  Yes exactly. It generates code for Val Town, which hosts Typescript backends.
  - ilaksh 2y ago
    I think that this is kind of an obvious "optimization" for making application generation much more reliable. Just because the models can generate code for one of 1000 different platforms, doesn't mean that you need all of them. Just by narrowing the scope to a particular platform makes it much more feasible to get working applications without needing manual debugging due to out of date library references etc. I think something like the approach you have demonstrated here will relatively quickly become the standard for "no-code" application development.
    - janpaul123 2y ago
      Completely agree. It's useful not just for targeting one specific language, but all the other APIs that we have, and things like RAG to search for importable modules on the platform. Duplicating all that across many platforms is a lot of work!
redfloatplane 2y ago
Reviving my long-dead account to say that I built a perfectly functional small site to help schedule my dungeons and dragons group within about 5 minutes, on my phone, from my bed. If this isn't the future I don't want to go there. Fantastic work.
- stevekrouse 2y ago
  Woo! I love this story. This is why we're building val town
  - redfloatplane 2y ago
    Insanely smooth user story. From signup to unique solution to my problem in no time. Amazing work, I am blown away.
- smeej 2y ago
  Do you already have a background as a dev? As someone who's run in software startup circles on the customer support side for years, I couldn't figure out how to make it do anything. Took several minutes even to notice the "save and preview" button to get my "edit this text" edit to work on the code editor. I first tried the "refresh" button on the preview, after realizing it didn't update automatically, but most of the text for the correct button wasn't visible. Then, when it asked me to make a "val" that did something, I tried looking at the templates, hoping to figure out what they even mean by "a val." The page loaded halfway down the screen and wouldn't scroll, so I gave up and went back to the site to see if the About page would tell me anything useful. I ended up back here hoping someone in the comments had figured it out, only to find that for some people, it clearly is entirely intuitive, which makes me think the whole platform is just for "other people," people not like me in some important way I haven't identified yet, even though it sounded like it was targeting people who have ideas for projects but don't know how to build them themselves.
  - stevekrouse 2y ago
    Sorry to hear this! We definitely have a lot of work to do to make it more intuitive. If you have 30 minutes, I'd love to do a video call where I walk you through it and take notes on what you find confusing. Hopefully it'd be helpful to us both! I'm steve@val.town, shoot me a note if you're up for a chat
01HNNWZ0MV43FF 2y ago
And they can't be self-hosted?
- maxmcd 2y ago
  There hasn’t been a ton of work in this area, but most things will work in Deno if you import them directly. You’d need to figure out blob storage and sqlite but I don’t think it would be the biggest lift to do so.
deleted 2y ago
[deleted]
devbent 2y ago
One problem that I run into with LLM code generation on large projects is that at some point the LLM runs into a problem it just cannot fix no matter how it is prompted. This manifest in a number of ways. Sometimes it is by bouncing back and forth between two invalid solutions while other times it is bouncing back and forth fixing one issue and while breaking something else in another part of the code. Another issue with complex projects is that llms will not tell you what you don't know. They will happily go about designing crappy code if you ask them for a crappy solution and they don't have the ability to recommend a better path forward unless explicitly prompted. That said, I had Claude generate most of a tile-based 2D pixel art rendering engine[1] for me, but again, once things got complicated I had to go and start hand fixing the code because Claude was no longer able to make improvements. I've seen these failure modes across multiple problem domains, from CSS (alternating between two broken css styles, neither came close to fixing the issue) to backend, to rendering code (trying to get character sprites correctly on the tiles) [1] https://www.generativestorytelling.ai/town/index.html https://www.generativestorytelling.ai/town/index.html notice the tons of rendering artifacts. I've realized I'm going to need to rewrite a lot of how rendering happens to resolve them. Claude wrote 80% of the original code but by the time I'm done fixing everything maybe only 30% or so of Claude's code will remain.
- threeseed 2y ago
  I've found that LLMs are only as good as the code it is trained on. So for basic CRUD style web apps it is great. But then again so is a template. But the minute you are dealing with newer libraries, less popular languages e.g Rust or Scala it just falls apart where for me it constantly hallucinates methods, imports etc.
  - halfmatthalfcat 2y ago
    I spent weeks trying to get GPT4 to get me through some gnarly Shapeless (Scala) issues. After that failure, I realized how real the limitations are. They really cannot produce original work and as far as niche languages, hallucinates all the time to the point of being completely unusable.
anonzzzies 2y ago
Whenever there is a 'big new AI model' launch, I try to build one of my side projects fully with the new AI. So I do not touch anything myself, I only talk english to it. I do read the generated code and instructions so I can correct them in English; no code at all. It worked twice; with chatgpt4 and the sonnet launch. All the others did not manage without significant code or ops help. It is a very annoying experience even if you know what you are doing; it is still much faster than what you would get done writing code but it is very frustrating getting the last 10% right; you spend a day on 80%, two days on 10 and a week on the last 10%. If I just jump in and fix the code myself, it is about 1 day for the same project, which is still amazing (and not imaginable before). People complaining that it sucks and it cannot figure things out often are right, however, it is a lot better than what we had before, which was doing all this by hand (causing many people to procrastinate over even starting a side project while have 1000s in mind every day). These types of services are important and I like this val.town idea. Well done and keep going.
- ben_w 2y ago
  Yup, absolutely. I've got a few side-projects that I'm doing for fun, which wouldn't have even gotten started if not for LLMs being good enough — but, like many customer-facing roles*, LLMs get a lot of slack from people who (appear to) expect perfection for free. When the AI doesn't make mistakes, humans can't continue to get paid to do those jobs. So I'm hoping AI continues to be mediocre at writing code for at least another 2 years. The other big GenAI things, I'm not sure how they're perceived. Music? I don't know well enough to tell. Images? The way people talk in public is a three-way mix of complaining they're (1) making bad art while (2) also taking away jobs, plus (3) the "this rips off the human creators" that is present for all GenAI models. Video? I'm only really seeing people mock it right now; even the praise is caveated "I can see the flaws but $something_positive". * Was thinking this yesterday WRT the police getting flack both for not doing enough and also for being too proactive; but I could say this about almost anything — a former partner could have said the same about working in a call center, a few days ago on this site there was a discussion about doctors making terrible mistakes, and there's a few current examples I don't dare write about because I know I'll get flack from "both sides" (even on this site) for even daring to use the phrase "both sides".
deckiedan 2y ago
I just played with townie AI for an hour or so... Very cool! Very fun. There's still some glitches, occasionally the entire app code would get replaced by the function the LLM was trying to update. I could fix it by telling it that's what had happened, and it would then fill everything in again... Waiting for the entire app to be rewritten each time was a bit annoying. It got the initial concepts of the app very very quickly running, but then struggled with some CSS stuff, saying it would try a different approach, or apologising for missing things repeatedly...and eventually it told me it would try more radical approaches and wrote online styles... I wonder if the single file approach has limitations in that respect. Very interesting, very fun to play with. I'm kind of concerned for security things with LLM written apps - you can ask it to do things and it says yes, without really thinking if it's a good idea or not. But cool! And anything which helps with the internet to be full of small independent quirky creative ideas, the better.
- sabbaticaldev 2y ago
  > I'm kind of concerned for security things with LLM written apps - you can ask it to do things and it says yes, without really thinking if it's a good idea or not. you can ask it to consider your concerns too
  - deckiedan 2y ago
    Well, right. If I'm using an LLM to create code, I'm going to use all my skill and experience to review and shape the code to standards I'm ok with. But for people with extremely limited experience, LLMs offer a "create an app by talking!!" Zero understanding required. So they won't know to not leak user PII in JSON responses or have publicly writable endpoints or keeping private keys for external services server side and outside of the code base, etc... Let alone anything more complex.
wonger_ 2y ago
The author's bit about "IterateGPT" reminds me of this "AI in a loop" post from last year: https://til.simonwillison.net/llms/python-react-pattern https://til.simonwillison.net/llms/python-react-pattern prompt = """ You run in a loop of Thought, Action, PAUSE, Observation. At the end of the loop you output an Answer Use Thought to describe your thoughts about the question you have been asked. Use Action to run one of the actions available to you - then return PAUSE. Observation will be the result of running those actions. ...""" Seems like a really powerful technique to have LLMs act on their own feedback.
- keymon-o 2y ago
  Interesting. Sounds like prompt engineering might just move from laughable to elitist field one day.
osigurdson 2y ago
LLMs are massively useful, just not in the way that people think they should work. No, you are not the manager with AI doing the work. Instead, it is more like someone to bounce ideas off of, a teacher (who is often wrong) and a reviewer. The human, ironically, is the specialist that can get details right, not AI (at least not for now).
thelastparadise 2y ago
Couldn't this essentially be used as a training data generator? E.g. have humans + LLMs generate a bunch of prompts that goes into this system, and it spits out a bunch of fully-fledged applications, which can be used to train an even bigger model.

175 points by stevekrouse 2y ago

janpaul123 2y ago

Post author here! Happy to answer any questions.

deleted 2y ago

[deleted]

syspec 2y ago

Does fullstack here mean using javascript on the backend? Is it able to generate code for other backend languages?

janpaul123 2y ago
Yes exactly. It generates code for Val Town, which hosts Typescript backends.
- ilaksh 2y ago
  I think that this is kind of an obvious "optimization" for making application generation much more reliable. Just because the models can generate code for one of 1000 different platforms, doesn't mean that you need all of them. Just by narrowing the scope to a particular platform makes it much more feasible to get working applications without needing manual debugging due to out of date library references etc. I think something like the approach you have demonstrated here will relatively quickly become the standard for "no-code" application development.
  - janpaul123 2y ago
    Completely agree. It's useful not just for targeting one specific language, but all the other APIs that we have, and things like RAG to search for importable modules on the platform. Duplicating all that across many platforms is a lot of work!

redfloatplane 2y ago

Reviving my long-dead account to say that I built a perfectly functional small site to help schedule my dungeons and dragons group within about 5 minutes, on my phone, from my bed. If this isn't the future I don't want to go there. Fantastic work.

stevekrouse 2y ago
Woo! I love this story. This is why we're building val town
- redfloatplane 2y ago
  Insanely smooth user story. From signup to unique solution to my problem in no time. Amazing work, I am blown away.
smeej 2y ago
Do you already have a background as a dev? As someone who's run in software startup circles on the customer support side for years, I couldn't figure out how to make it do anything. Took several minutes even to notice the "save and preview" button to get my "edit this text" edit to work on the code editor. I first tried the "refresh" button on the preview, after realizing it didn't update automatically, but most of the text for the correct button wasn't visible. Then, when it asked me to make a "val" that did something, I tried looking at the templates, hoping to figure out what they even mean by "a val." The page loaded halfway down the screen and wouldn't scroll, so I gave up and went back to the site to see if the About page would tell me anything useful. I ended up back here hoping someone in the comments had figured it out, only to find that for some people, it clearly is entirely intuitive, which makes me think the whole platform is just for "other people," people not like me in some important way I haven't identified yet, even though it sounded like it was targeting people who have ideas for projects but don't know how to build them themselves.
- stevekrouse 2y ago
  Sorry to hear this! We definitely have a lot of work to do to make it more intuitive. If you have 30 minutes, I'd love to do a video call where I walk you through it and take notes on what you find confusing. Hopefully it'd be helpful to us both! I'm steve@val.town, shoot me a note if you're up for a chat

01HNNWZ0MV43FF 2y ago

And they can't be self-hosted?

maxmcd 2y ago
There hasn’t been a ton of work in this area, but most things will work in Deno if you import them directly. You’d need to figure out blob storage and sqlite but I don’t think it would be the biggest lift to do so.

deleted 2y ago

[deleted]

devbent 2y ago

One problem that I run into with LLM code generation on large projects is that at some point the LLM runs into a problem it just cannot fix no matter how it is prompted. This manifest in a number of ways. Sometimes it is by bouncing back and forth between two invalid solutions while other times it is bouncing back and forth fixing one issue and while breaking something else in another part of the code. Another issue with complex projects is that llms will not tell you what you don't know. They will happily go about designing crappy code if you ask them for a crappy solution and they don't have the ability to recommend a better path forward unless explicitly prompted. That said, I had Claude generate most of a tile-based 2D pixel art rendering engine[1] for me, but again, once things got complicated I had to go and start hand fixing the code because Claude was no longer able to make improvements. I've seen these failure modes across multiple problem domains, from CSS (alternating between two broken css styles, neither came close to fixing the issue) to backend, to rendering code (trying to get character sprites correctly on the tiles) [1] https://www.generativestorytelling.ai/town/index.html https://www.generativestorytelling.ai/town/index.html notice the tons of rendering artifacts. I've realized I'm going to need to rewrite a lot of how rendering happens to resolve them. Claude wrote 80% of the original code but by the time I'm done fixing everything maybe only 30% or so of Claude's code will remain.

threeseed 2y ago
I've found that LLMs are only as good as the code it is trained on. So for basic CRUD style web apps it is great. But then again so is a template. But the minute you are dealing with newer libraries, less popular languages e.g Rust or Scala it just falls apart where for me it constantly hallucinates methods, imports etc.
- halfmatthalfcat 2y ago
  I spent weeks trying to get GPT4 to get me through some gnarly Shapeless (Scala) issues. After that failure, I realized how real the limitations are. They really cannot produce original work and as far as niche languages, hallucinates all the time to the point of being completely unusable.

anonzzzies 2y ago

Whenever there is a 'big new AI model' launch, I try to build one of my side projects fully with the new AI. So I do not touch anything myself, I only talk english to it. I do read the generated code and instructions so I can correct them in English; no code at all. It worked twice; with chatgpt4 and the sonnet launch. All the others did not manage without significant code or ops help. It is a very annoying experience even if you know what you are doing; it is still much faster than what you would get done writing code but it is very frustrating getting the last 10% right; you spend a day on 80%, two days on 10 and a week on the last 10%. If I just jump in and fix the code myself, it is about 1 day for the same project, which is still amazing (and not imaginable before). People complaining that it sucks and it cannot figure things out often are right, however, it is a lot better than what we had before, which was doing all this by hand (causing many people to procrastinate over even starting a side project while have 1000s in mind every day). These types of services are important and I like this val.town idea. Well done and keep going.

ben_w 2y ago
Yup, absolutely. I've got a few side-projects that I'm doing for fun, which wouldn't have even gotten started if not for LLMs being good enough — but, like many customer-facing roles*, LLMs get a lot of slack from people who (appear to) expect perfection for free. When the AI doesn't make mistakes, humans can't continue to get paid to do those jobs. So I'm hoping AI continues to be mediocre at writing code for at least another 2 years. The other big GenAI things, I'm not sure how they're perceived. Music? I don't know well enough to tell. Images? The way people talk in public is a three-way mix of complaining they're (1) making bad art while (2) also taking away jobs, plus (3) the "this rips off the human creators" that is present for all GenAI models. Video? I'm only really seeing people mock it right now; even the praise is caveated "I can see the flaws but $something_positive". * Was thinking this yesterday WRT the police getting flack both for not doing enough and also for being too proactive; but I could say this about almost anything — a former partner could have said the same about working in a call center, a few days ago on this site there was a discussion about doctors making terrible mistakes, and there's a few current examples I don't dare write about because I know I'll get flack from "both sides" (even on this site) for even daring to use the phrase "both sides".

deckiedan 2y ago

I just played with townie AI for an hour or so... Very cool! Very fun. There's still some glitches, occasionally the entire app code would get replaced by the function the LLM was trying to update. I could fix it by telling it that's what had happened, and it would then fill everything in again... Waiting for the entire app to be rewritten each time was a bit annoying. It got the initial concepts of the app very very quickly running, but then struggled with some CSS stuff, saying it would try a different approach, or apologising for missing things repeatedly...and eventually it told me it would try more radical approaches and wrote online styles... I wonder if the single file approach has limitations in that respect. Very interesting, very fun to play with. I'm kind of concerned for security things with LLM written apps - you can ask it to do things and it says yes, without really thinking if it's a good idea or not. But cool! And anything which helps with the internet to be full of small independent quirky creative ideas, the better.

sabbaticaldev 2y ago
> I'm kind of concerned for security things with LLM written apps - you can ask it to do things and it says yes, without really thinking if it's a good idea or not. you can ask it to consider your concerns too
- deckiedan 2y ago
  Well, right. If I'm using an LLM to create code, I'm going to use all my skill and experience to review and shape the code to standards I'm ok with. But for people with extremely limited experience, LLMs offer a "create an app by talking!!" Zero understanding required. So they won't know to not leak user PII in JSON responses or have publicly writable endpoints or keeping private keys for external services server side and outside of the code base, etc... Let alone anything more complex.

wonger_ 2y ago

The author's bit about "IterateGPT" reminds me of this "AI in a loop" post from last year: https://til.simonwillison.net/llms/python-react-pattern https://til.simonwillison.net/llms/python-react-pattern prompt = """ You run in a loop of Thought, Action, PAUSE, Observation. At the end of the loop you output an Answer Use Thought to describe your thoughts about the question you have been asked. Use Action to run one of the actions available to you - then return PAUSE. Observation will be the result of running those actions. ...""" Seems like a really powerful technique to have LLMs act on their own feedback.

keymon-o 2y ago
Interesting. Sounds like prompt engineering might just move from laughable to elitist field one day.

osigurdson 2y ago

LLMs are massively useful, just not in the way that people think they should work. No, you are not the manager with AI doing the work. Instead, it is more like someone to bounce ideas off of, a teacher (who is often wrong) and a reviewer. The human, ironically, is the specialist that can get details right, not AI (at least not for now).

thelastparadise 2y ago

Couldn't this essentially be used as a training data generator? E.g. have humans + LLMs generate a bunch of prompts that goes into this system, and it spits out a bunch of fully-fledged applications, which can be used to train an even bigger model.