3 ms·

You need none of these fancy tools if you iterate over specs instead of iterating over code. I explain it all in here: https://www.cleverthinkingsoftware.com/sp

You need none of these fancy tools if you iterate over specs instead of iterating over code. I explain it all in here: https://www.cleverthinkingsoftware.com/spec-first-development-the-missing-manual-for-building-with-ai/ https://www.cleverthinkingsoftware.com/spec-first-developmen...

osigurdson 1y ago
I think a series of specs as the system instruction could help guide it. You can't just go from spec to app though, at least in my experience.
- dexterlagan 1y ago
  From my experience Claude 3.7 seems to make less mistakes when you provide detailed specs. It saves a lot of back and forth, especially at the beginning, when you're trying to 'de-noise' the idea. I find that the best way is using a hybrid approach: going between iterating over specs to iterating over code, and once I have the MVP I update the specs one last time. I pass these specs to another session and continue working. It solves multiple problems at once: context getting full, LLM getting lost etc.
esafak 1y ago
That is slow and expensive.
- dexterlagan 1y ago
  It sounds slow and expensive, but it ended up saving me a lot of tokens and time. Especially when you want to target multiple platforms at once. For me it was liberating, absolutely night and day.
- dexterlagan 1y ago
  Also, keep in mind that you can continue iterating over code while you keep the specs up to date. You can then give the final specs to another LLM to resume work or produce another version for a different platform or using a different framework. I do need to update the article to clarify this.
- datadrivenangel 1y ago
  cheaper than the alternatives
WA 1y ago
I played around with your suggestion for a day or two now. While I'm intrigued, there are some real-world issues with this approach: - The same spec is processed by the same LLM differently when implementing from scratch. This can maybe mitigated somewhat by adjusting the temperature slider. But generally speaking, the same spec won't give the same result unless you are very specific. - Same if you use different LLMs. The same spec can give entirely different results for different LLMs. - This can probably mitigated somewhat by getting more specific in the spec, but at some point, it is so specific as being the code itself. Unless of course you don't care that much about the details. But if you don't, you get a slightly different app every time you implement from scratch. - Gemini 2.5 pro has "reasoning" capabilities and introduces a lot of "thinking" tokens into the context. Let's say you start with a single line spec and iterate from there. Gemini will give you a more detailed spec based on its thinking process. But if you then take the new thinking-process spec as a new starting point for the next iteration of the spec, you get even more thinking. In short, the spec gets automatically expanded by the way of "thinking" with reasoning models. - Produced code can have small bugs, but they are not really worth to put in the spec, because they are an implementation detail. I'll keep experimenting with it, but I don't think this is the holy grail of AI assisted coding.
- dexterlagan 1y ago
  Thanks for the feedback! Agreed, the one problem with the approach is reproducibility. It can be mitigated by going temp. 0, and detailing further the specs. The one method that nearly completely solves this problem is the hybrid approach: Write detailed specs, feed them to the LLM, get an MVP (or module etc); fix any and all issues found with the MVP, implement missing/new features; ask the LLM to update the specs to take the changes in account, also recording the lessons learned - and maximize reproducibility. Treat the latest specs + code package as a checkpoint you can always resume work from.
  - WA 1y ago
    I'll give it a try, thanks. Edit: Do you use reasoning models that introduce way more tokens into the context at all or prefer simpler models?

osigurdson 1y ago

I think a series of specs as the system instruction could help guide it. You can't just go from spec to app though, at least in my experience.

dexterlagan 1y ago
From my experience Claude 3.7 seems to make less mistakes when you provide detailed specs. It saves a lot of back and forth, especially at the beginning, when you're trying to 'de-noise' the idea. I find that the best way is using a hybrid approach: going between iterating over specs to iterating over code, and once I have the MVP I update the specs one last time. I pass these specs to another session and continue working. It solves multiple problems at once: context getting full, LLM getting lost etc.

esafak 1y ago

That is slow and expensive.

dexterlagan 1y ago
It sounds slow and expensive, but it ended up saving me a lot of tokens and time. Especially when you want to target multiple platforms at once. For me it was liberating, absolutely night and day.
dexterlagan 1y ago
Also, keep in mind that you can continue iterating over code while you keep the specs up to date. You can then give the final specs to another LLM to resume work or produce another version for a different platform or using a different framework. I do need to update the article to clarify this.
datadrivenangel 1y ago
cheaper than the alternatives

WA 1y ago

I played around with your suggestion for a day or two now. While I'm intrigued, there are some real-world issues with this approach: - The same spec is processed by the same LLM differently when implementing from scratch. This can maybe mitigated somewhat by adjusting the temperature slider. But generally speaking, the same spec won't give the same result unless you are very specific. - Same if you use different LLMs. The same spec can give entirely different results for different LLMs. - This can probably mitigated somewhat by getting more specific in the spec, but at some point, it is so specific as being the code itself. Unless of course you don't care that much about the details. But if you don't, you get a slightly different app every time you implement from scratch. - Gemini 2.5 pro has "reasoning" capabilities and introduces a lot of "thinking" tokens into the context. Let's say you start with a single line spec and iterate from there. Gemini will give you a more detailed spec based on its thinking process. But if you then take the new thinking-process spec as a new starting point for the next iteration of the spec, you get even more thinking. In short, the spec gets automatically expanded by the way of "thinking" with reasoning models. - Produced code can have small bugs, but they are not really worth to put in the spec, because they are an implementation detail. I'll keep experimenting with it, but I don't think this is the holy grail of AI assisted coding.

dexterlagan 1y ago
Thanks for the feedback! Agreed, the one problem with the approach is reproducibility. It can be mitigated by going temp. 0, and detailing further the specs. The one method that nearly completely solves this problem is the hybrid approach: Write detailed specs, feed them to the LLM, get an MVP (or module etc); fix any and all issues found with the MVP, implement missing/new features; ask the LLM to update the specs to take the changes in account, also recording the lessons learned - and maximize reproducibility. Treat the latest specs + code package as a checkpoint you can always resume work from.
- WA 1y ago
  I'll give it a try, thanks. Edit: Do you use reasoning models that introduce way more tokens into the context at all or prefer simpler models?