4 ms·

> All in all, a prompt history would be really edicational if anyone thinks about doing something similar. As we start to develop AI first programs, I believe

by sdesol 2y ago

> All in all, a prompt history would be really edicational if anyone thinks about doing something similar. As we start to develop AI first programs, I believe we will need to start connecting LLM conversations to code, not only for educational purpose but for maintenance as well. I'm currently experimenting with what I call Block-UUIDs, and they are designed to make it easy to trace LLM generated code. You can see what I mean in the link below, which contains a simple hello world example. https://app.gitsense.com/?chat=7d09a63f-d684-4e2c-97b2-71aa1c2b0e0d https://app.gitsense.com/?chat=7d09a63f-d684-4e2c-97b2-71aa1... Something worth noting is, you can't expect the LLM to properly generate a UUID. If you ask the LLM, it'll says it can, but I don't trust it to do it correctly all the time. Since I can't trust the LLM, I instruct the LLM to use a template string, which I can replace on the server side. I've also found LLMs will not always follow instructions and will generate UUID and how I handle this is, when the LLM stops streaming, I will validate and fix if needed, any invalid UUIDs. How I see things playing out in the future is, we will alway link to LLM conversations which will give use the Block-UUIDs generated and by looking at the code, we can see what Block-UUID was used and how it came about. Full Disclosure: This is my tool

fer 2y ago
[dead]
garblegarble 2y ago
Why not just put the conversation / prompt at the end of the commit/PR message? That way you have everything in one consistent database and don't run the risk of losing it. Additionally, you get more directly usable text out of a 'git blame'
- sdesol 2y ago
  > Why not just put the conversation / prompt at the end of the commit/PR message? This might be something I would do. My only concern is, the conversations can be quite long and mean very long. My "Chat Flow" right now is to discuss what needs to be done. Produce the code, which can span multiple chats and then have the LLM summarize things. What I think might make sense in the future is to include detailed chat summaries in commit messages and PRs. Given that we get a lot of text diarrhea from LLMs, I think putting them in a LLM as is, may do more harm than good.
shermantanktop 2y ago
Capturing the prompt is probably a good idea. But if I think about the goal of "reproducible builds" and "supply chain security," the use of LLMs in a vibe-coding mode looks like a step backward. Ultimately the code needs to stand alone, but if you discover that a specific version of an LLM produced vulnerable code, you have no recourse but to try again and read the generated code more carefully. And reading code carefully is the opposite of vibe-coding.
- sdesol 2y ago
  When dealing with prompts, there is no such thing as reproducible builds, since we are dealing with a non-deterministic system. The purpose for connecting prompts and conversation to code is to aid in debugging and developing new features. I also believe by connecting code to conversations, we will produce better LLMs in the future. I would say AI has generated about 98% of my code for my chat app in the last 3 months and it was definitely not vibe coding. Every function and feature was discussed in detail and some conversations took over a week. My reasoning for building my chat app wasn't to support vibe coding, but rather to 10x senior developers. Once you know how things work, the biggest bottleneck to a senior developers productivity is typing and documentation. The speed at which LLMs can produce code and documentation cannot be matched by humans. The only downside is, LLM don't necessary produce pretty or more readable code. The more readable code is something I would like to tackle in the future, as I believe post-processing tools can make LLM code much more readable.

by sdesol 2y ago

fer 2y ago

[dead]

garblegarble 2y ago

Why not just put the conversation / prompt at the end of the commit/PR message? That way you have everything in one consistent database and don't run the risk of losing it. Additionally, you get more directly usable text out of a 'git blame'

sdesol 2y ago
> Why not just put the conversation / prompt at the end of the commit/PR message? This might be something I would do. My only concern is, the conversations can be quite long and mean very long. My "Chat Flow" right now is to discuss what needs to be done. Produce the code, which can span multiple chats and then have the LLM summarize things. What I think might make sense in the future is to include detailed chat summaries in commit messages and PRs. Given that we get a lot of text diarrhea from LLMs, I think putting them in a LLM as is, may do more harm than good.

shermantanktop 2y ago

Capturing the prompt is probably a good idea. But if I think about the goal of "reproducible builds" and "supply chain security," the use of LLMs in a vibe-coding mode looks like a step backward. Ultimately the code needs to stand alone, but if you discover that a specific version of an LLM produced vulnerable code, you have no recourse but to try again and read the generated code more carefully. And reading code carefully is the opposite of vibe-coding.

sdesol 2y ago
When dealing with prompts, there is no such thing as reproducible builds, since we are dealing with a non-deterministic system. The purpose for connecting prompts and conversation to code is to aid in debugging and developing new features. I also believe by connecting code to conversations, we will produce better LLMs in the future. I would say AI has generated about 98% of my code for my chat app in the last 3 months and it was definitely not vibe coding. Every function and feature was discussed in detail and some conversations took over a week. My reasoning for building my chat app wasn't to support vibe coding, but rather to 10x senior developers. Once you know how things work, the biggest bottleneck to a senior developers productivity is typing and documentation. The speed at which LLMs can produce code and documentation cannot be matched by humans. The only downside is, LLM don't necessary produce pretty or more readable code. The more readable code is something I would like to tackle in the future, as I believe post-processing tools can make LLM code much more readable.