4 ms·

This is impressive honestly. I also find it interesting that the ai code submissions like this one are generally vague about the process. This seem to be crea

by eknkc 2y ago

This is impressive honestly. I also find it interesting that the ai code submissions like this one are generally vague about the process. This seem to be created using Cline on VsCode, prompting to Gemini 2.5 Pro using OpenRouter. The commit history implies that a crude version was created by the LLM using an initial prompt and then gradually improved with features, fixes etc. Assuming ongoing conversations with the AI agent. All code in a single index.html file which might not be great for human coders but who cares to be fair. All in all, a prompt history would be really educational if anyone thinks about doing something similar.

sdesol 2y ago
> All in all, a prompt history would be really edicational if anyone thinks about doing something similar. As we start to develop AI first programs, I believe we will need to start connecting LLM conversations to code, not only for educational purpose but for maintenance as well. I'm currently experimenting with what I call Block-UUIDs, and they are designed to make it easy to trace LLM generated code. You can see what I mean in the link below, which contains a simple hello world example. https://app.gitsense.com/?chat=7d09a63f-d684-4e2c-97b2-71aa1c2b0e0d https://app.gitsense.com/?chat=7d09a63f-d684-4e2c-97b2-71aa1... Something worth noting is, you can't expect the LLM to properly generate a UUID. If you ask the LLM, it'll says it can, but I don't trust it to do it correctly all the time. Since I can't trust the LLM, I instruct the LLM to use a template string, which I can replace on the server side. I've also found LLMs will not always follow instructions and will generate UUID and how I handle this is, when the LLM stops streaming, I will validate and fix if needed, any invalid UUIDs. How I see things playing out in the future is, we will alway link to LLM conversations which will give use the Block-UUIDs generated and by looking at the code, we can see what Block-UUID was used and how it came about. Full Disclosure: This is my tool
- fer 2y ago
  [dead]
- garblegarble 2y ago
  Why not just put the conversation / prompt at the end of the commit/PR message? That way you have everything in one consistent database and don't run the risk of losing it. Additionally, you get more directly usable text out of a 'git blame'
  - sdesol 2y ago
    > Why not just put the conversation / prompt at the end of the commit/PR message? This might be something I would do. My only concern is, the conversations can be quite long and mean very long. My "Chat Flow" right now is to discuss what needs to be done. Produce the code, which can span multiple chats and then have the LLM summarize things. What I think might make sense in the future is to include detailed chat summaries in commit messages and PRs. Given that we get a lot of text diarrhea from LLMs, I think putting them in a LLM as is, may do more harm than good.
- shermantanktop 2y ago
  Capturing the prompt is probably a good idea. But if I think about the goal of "reproducible builds" and "supply chain security," the use of LLMs in a vibe-coding mode looks like a step backward. Ultimately the code needs to stand alone, but if you discover that a specific version of an LLM produced vulnerable code, you have no recourse but to try again and read the generated code more carefully. And reading code carefully is the opposite of vibe-coding.
  - sdesol 2y ago
    When dealing with prompts, there is no such thing as reproducible builds, since we are dealing with a non-deterministic system. The purpose for connecting prompts and conversation to code is to aid in debugging and developing new features. I also believe by connecting code to conversations, we will produce better LLMs in the future. I would say AI has generated about 98% of my code for my chat app in the last 3 months and it was definitely not vibe coding. Every function and feature was discussed in detail and some conversations took over a week. My reasoning for building my chat app wasn't to support vibe coding, but rather to 10x senior developers. Once you know how things work, the biggest bottleneck to a senior developers productivity is typing and documentation. The speed at which LLMs can produce code and documentation cannot be matched by humans. The only downside is, LLM don't necessary produce pretty or more readable code. The more readable code is something I would like to tackle in the future, as I believe post-processing tools can make LLM code much more readable.
re-thc 2y ago
> but who cares to be fair i.e. who cares when the LLM starts giving up when the file is too big and confusing? i.e. who cares when the LLM introduces bugs / flaws it isn't away of?
- eknkc 2y ago
  I mean I don’t think these projects will be maintained by humans anyway because they are not structured by humans. If the AI fucks up then its a lost cause. And maybe you’ll better off create a new version from scratch instead of trying to maintain one when LLM starts to fail. Just ask Gemini 3.5 this time around. The AI can write obfuscated code. Name all variables from a to z. Even emit binaries directly. Who cares if it works?
  - rom16384 2y ago
    Unfortunately I think this will the the future of vibe coding. Minified blobs of code that only the AI is able to edit. We don't usually review the compiler's use of registers or memory allocation after all.
  - re-thc 2y ago
    > If the AI fucks up then its a lost cause. And maybe you’ll better off create a new version from scratch instead of trying to maintain one So how does it save time? Do I also tell customers we're blowing things up and redoing it every few months with potential risks like data loss?
    - eknkc 2y ago
      I guess the idea is that the customers are the ones interacting with the LLM and we are homeless if this thing goes that way. Probably saves a lot of their money in exchange for some of their time.. I personally do not think any of this is a good idea but here we are. And I was making fun of AI images with the weird fingers and shit just a year ago, now it's hard to identify AI generated images. The code gen now can create a single file space invaders which is impressive but shitty according to all coding metrics. They are getting better. At some point the single file shit stew will be good enough cause the context windows and capabilities of LLMs will be able to handle those files. That's when nobody gonna care I guess.
Hojojo 2y ago
I've experimented with making it more explicit in one project I'm working on where 99% of it is written by AI, but the chat logs from Roo Code are way too verbose. Ideally, I could export a version of the chat with my prompts and metadata about which lines were changed, then I could commit that with the rest of the changes. I have every chat associated with changes made solely with AI in a chat folder named similarly to the commit, but the chats themselves are just impossible to read, so I haven't committed any of them.
- rom16384 2y ago
  I also find Roo Code chats add way too much stuff. I'd would like something that only includes to the context exactly the files I want, and only calls the model when I want. Using AI Studio with custom system prompt works, but is not as fast, but with Roo / Cline the costs can explode if you are not careful to create new tasks.
_joel 2y ago
> All code in a single index.html file which might not be great for human coders but who cares to be fair. AI won't cope well as that file gets larger, or best hope that experimental diff feature is working well. I find stuff really breaks if you don't refactor it down.
- Hojojo 2y ago
  Gemini works with large files surprisingly well, but from my limited experience, the token usage (and cost) explodes and I'm not not sure it's worth it. I try to keep my file lengths fairly small because of it, so I have more control over what is pulled into the chat context.
- vbtechguy 2y ago
  already broke up index.html so no longer an issue
froh 2y ago
I think putting prompts into the commit log, mixed with manual edits, would be a great way to document the evolution of LLM assisted engineering. I wonder if there is some tool support yet that supports that.
- vbtechguy 2y ago
  That would be one long commit description. I have had prompts in 1000s of word lengths!
  - froh 2y ago
    oh wow. were those context files? or did you write the multi page prompts by hand?
    - vbtechguy 2y ago
      At later stages of development i use Gemini 2.5 Pro for brain storm sessions to formulate what I want and once I have it fleshed out, I ask it to summarize it into AI actionable prompt that is very detailed that I can copy and paste into a new chat session with any AI llm model that would have everything needed to implement what I want
      - froh 2y ago
        TIL thanks for sharing!
      - indigodaddy 2y ago
        That's actually brilliant and a great idea, as I'm often LLM-hopping as I run out of free credits from platform to platform!
        vbtechguy 2y ago
        Same I have paid subs with ChatGPT Plus, Claude Pro and Gemini Advanced and utilise them for brainstorming part without fear of token usage billshock as they are fixed monthly subscriptions.
ehutch79 2y ago
Would a prompt history even be useful? This is all non-deterministic. the same prompt could result in wildy different code?
- onion2k 2y ago
  Would that actually matter if the result code ends up running a similar, working game?

by eknkc 2y ago

sdesol 2y ago

> All in all, a prompt history would be really edicational if anyone thinks about doing something similar. As we start to develop AI first programs, I believe we will need to start connecting LLM conversations to code, not only for educational purpose but for maintenance as well. I'm currently experimenting with what I call Block-UUIDs, and they are designed to make it easy to trace LLM generated code. You can see what I mean in the link below, which contains a simple hello world example. https://app.gitsense.com/?chat=7d09a63f-d684-4e2c-97b2-71aa1c2b0e0d https://app.gitsense.com/?chat=7d09a63f-d684-4e2c-97b2-71aa1... Something worth noting is, you can't expect the LLM to properly generate a UUID. If you ask the LLM, it'll says it can, but I don't trust it to do it correctly all the time. Since I can't trust the LLM, I instruct the LLM to use a template string, which I can replace on the server side. I've also found LLMs will not always follow instructions and will generate UUID and how I handle this is, when the LLM stops streaming, I will validate and fix if needed, any invalid UUIDs. How I see things playing out in the future is, we will alway link to LLM conversations which will give use the Block-UUIDs generated and by looking at the code, we can see what Block-UUID was used and how it came about. Full Disclosure: This is my tool

fer 2y ago
[dead]
garblegarble 2y ago
Why not just put the conversation / prompt at the end of the commit/PR message? That way you have everything in one consistent database and don't run the risk of losing it. Additionally, you get more directly usable text out of a 'git blame'
- sdesol 2y ago
  > Why not just put the conversation / prompt at the end of the commit/PR message? This might be something I would do. My only concern is, the conversations can be quite long and mean very long. My "Chat Flow" right now is to discuss what needs to be done. Produce the code, which can span multiple chats and then have the LLM summarize things. What I think might make sense in the future is to include detailed chat summaries in commit messages and PRs. Given that we get a lot of text diarrhea from LLMs, I think putting them in a LLM as is, may do more harm than good.
shermantanktop 2y ago
Capturing the prompt is probably a good idea. But if I think about the goal of "reproducible builds" and "supply chain security," the use of LLMs in a vibe-coding mode looks like a step backward. Ultimately the code needs to stand alone, but if you discover that a specific version of an LLM produced vulnerable code, you have no recourse but to try again and read the generated code more carefully. And reading code carefully is the opposite of vibe-coding.
- sdesol 2y ago
  When dealing with prompts, there is no such thing as reproducible builds, since we are dealing with a non-deterministic system. The purpose for connecting prompts and conversation to code is to aid in debugging and developing new features. I also believe by connecting code to conversations, we will produce better LLMs in the future. I would say AI has generated about 98% of my code for my chat app in the last 3 months and it was definitely not vibe coding. Every function and feature was discussed in detail and some conversations took over a week. My reasoning for building my chat app wasn't to support vibe coding, but rather to 10x senior developers. Once you know how things work, the biggest bottleneck to a senior developers productivity is typing and documentation. The speed at which LLMs can produce code and documentation cannot be matched by humans. The only downside is, LLM don't necessary produce pretty or more readable code. The more readable code is something I would like to tackle in the future, as I believe post-processing tools can make LLM code much more readable.

re-thc 2y ago

> but who cares to be fair i.e. who cares when the LLM starts giving up when the file is too big and confusing? i.e. who cares when the LLM introduces bugs / flaws it isn't away of?

eknkc 2y ago
I mean I don’t think these projects will be maintained by humans anyway because they are not structured by humans. If the AI fucks up then its a lost cause. And maybe you’ll better off create a new version from scratch instead of trying to maintain one when LLM starts to fail. Just ask Gemini 3.5 this time around. The AI can write obfuscated code. Name all variables from a to z. Even emit binaries directly. Who cares if it works?
- rom16384 2y ago
  Unfortunately I think this will the the future of vibe coding. Minified blobs of code that only the AI is able to edit. We don't usually review the compiler's use of registers or memory allocation after all.
- re-thc 2y ago
  > If the AI fucks up then its a lost cause. And maybe you’ll better off create a new version from scratch instead of trying to maintain one So how does it save time? Do I also tell customers we're blowing things up and redoing it every few months with potential risks like data loss?
  - eknkc 2y ago
    I guess the idea is that the customers are the ones interacting with the LLM and we are homeless if this thing goes that way. Probably saves a lot of their money in exchange for some of their time.. I personally do not think any of this is a good idea but here we are. And I was making fun of AI images with the weird fingers and shit just a year ago, now it's hard to identify AI generated images. The code gen now can create a single file space invaders which is impressive but shitty according to all coding metrics. They are getting better. At some point the single file shit stew will be good enough cause the context windows and capabilities of LLMs will be able to handle those files. That's when nobody gonna care I guess.

Hojojo 2y ago

I've experimented with making it more explicit in one project I'm working on where 99% of it is written by AI, but the chat logs from Roo Code are way too verbose. Ideally, I could export a version of the chat with my prompts and metadata about which lines were changed, then I could commit that with the rest of the changes. I have every chat associated with changes made solely with AI in a chat folder named similarly to the commit, but the chats themselves are just impossible to read, so I haven't committed any of them.

rom16384 2y ago
I also find Roo Code chats add way too much stuff. I'd would like something that only includes to the context exactly the files I want, and only calls the model when I want. Using AI Studio with custom system prompt works, but is not as fast, but with Roo / Cline the costs can explode if you are not careful to create new tasks.

_joel 2y ago

> All code in a single index.html file which might not be great for human coders but who cares to be fair. AI won't cope well as that file gets larger, or best hope that experimental diff feature is working well. I find stuff really breaks if you don't refactor it down.

Hojojo 2y ago
Gemini works with large files surprisingly well, but from my limited experience, the token usage (and cost) explodes and I'm not not sure it's worth it. I try to keep my file lengths fairly small because of it, so I have more control over what is pulled into the chat context.
vbtechguy 2y ago
already broke up index.html so no longer an issue

froh 2y ago

I think putting prompts into the commit log, mixed with manual edits, would be a great way to document the evolution of LLM assisted engineering. I wonder if there is some tool support yet that supports that.

vbtechguy 2y ago
That would be one long commit description. I have had prompts in 1000s of word lengths!
- froh 2y ago
  oh wow. were those context files? or did you write the multi page prompts by hand?
  - vbtechguy 2y ago
    At later stages of development i use Gemini 2.5 Pro for brain storm sessions to formulate what I want and once I have it fleshed out, I ask it to summarize it into AI actionable prompt that is very detailed that I can copy and paste into a new chat session with any AI llm model that would have everything needed to implement what I want
    - froh 2y ago
      TIL thanks for sharing!
    - indigodaddy 2y ago
      That's actually brilliant and a great idea, as I'm often LLM-hopping as I run out of free credits from platform to platform!
      - vbtechguy 2y ago
        Same I have paid subs with ChatGPT Plus, Claude Pro and Gemini Advanced and utilise them for brainstorming part without fear of token usage billshock as they are fixed monthly subscriptions.

ehutch79 2y ago

Would a prompt history even be useful? This is all non-deterministic. the same prompt could result in wildy different code?

onion2k 2y ago
Would that actually matter if the result code ends up running a similar, working game?