4 ms·

> I've been using LLMs the same way this past year. As a challenge to myself, I've implemented full-stack interview evaluations using languages and tools I've n

> I've been using LLMs the same way this past year. As a challenge to myself, I've implemented full-stack interview evaluations using languages and tools I've never worked with before and see what I can do. Using LLMs, it hasn't taken me more than 50% longer to craft a solution as compared to my crafting one using tools and technologies with which I'm intimately familiar. The problem with this is that you don't know what you don't know. I've actually used Claude to build a project in a framework that I haven't worked with before and it got me pretty far. Once I got that reviewed by someone who actually knew the framework it was terribly structured, not using higher level constructs to organize things, wasn't using idiomatic approaches - stuff I could easily confirm myself after going through the official docs. I think using LLMs to skip learning from official sources is a bad move, you're bound to hit a wall with LLM (at the very least the training data cutoff could be before something changed in the framework), you'll have to deep dive into the docs and figure it out. That is to say - sure you can pass the interview with LLM only - but you'll suck at the job. If you invest some effort upfront to learn from traditional sources and then use LLM to speed you up it's going to pay off really fast IMO. The more I know about what the output should be the more useful LLMs are.

Tostino 2y ago
Right. I've had that same problem when trying to build a new app with Claude / 4o. I essentially have to go through the docs myself so I understand the framework structure enough to write up an outline for the llm to follow when developing a new component or backend structure.
- dagw 2y ago
  With Claude I've had great success with first uploading the docs so that it has them as context for the rest of the session. Without doing that I got all kinds of answers, sometimes using a different framework or an older version of the framework I was using. Once it had the documentation to reference it started giving a lot better answers
  - Bluestein 2y ago
    Great point. Great suggestion.-
cadamsdotcom 2y ago
Very interesting outcome. Shows how far it’s possible to get without being idiomatic. You say the docs contain guidance on structuring, high-level constructs, and how to make things idiomatic. It would be an interesting test to hand the unfixed revision of the code to an LLM while also giving it the docs, and say “make any fixes to make this conform to standards of the framework and libraries”. If it picks up the same things that’s great news for novice programmers and anyone new to a framework! LLMs will improve at making these fixes over time - even if they’re currently bad at it that won’t last.
- rafaelmn 2y ago
  But that's kind of my point - when I was originally working on it - I didn't know there's a more idiomatic approach - why would I copy paste the docs to ask it to improve the results ? Especially given the LLMs tendency to hallucinate and gnerate noise when asked to "improve code". Also without reading the docs would I be able to spot why the idiomatic approach is better from single use case ? You're basically depending on LLM to make choices for you - and in my experience so far it makes very suboptimal ones.
- sungho_ 2y ago
  > It would be an interesting test to hand the unfixed revision of the code to an LLM while also giving it the docs, and say “make any fixes to make this conform to standards of the framework and libraries”. For your info, a few days ago, I was trying to migrate Tailwind 3 to 4 in a codebase where I had just set up the boilerplate, and I went through hell and eventually gave up. I used the Tailwind 4 documentation and Claude 3.5 sonnet in Windsurf IDE, and the part related to migration in my codebase was probably less than 50 lines. All of those tasks would take just a few minutes if a person did them directly (excluding reading the documents)
exhaze 2y ago
Using LLMs to bootstrap development in unfamiliar frameworks is a valuable approach when balanced with pragmatic engineering principles. While framework idioms are important, they're secondary to delivering working, maintainable software that solves real problems. The focus on "terrible structure" misses the point - what matters is whether the system meets its requirements efficiently and can be maintained effectively. Have you measured any actual negative impacts on system performance, maintainability, or customer value? My experience suggests that starting with working code and iteratively improving it as patterns emerge often leads to better outcomes than delaying development for complete framework mastery. The interview analogy is particularly misleading - success in a software engineering role isn't measured by framework knowledge, but by the ability to deliver value to customers effectively. Learning framework idioms can happen in parallel with productive development, especially when using LLMs to accelerate the initial learning curve.
- bluefirebrand 2y ago
  > While framework idioms are important, they're secondary to delivering working, *maintainable* software that solves real problems Emphasis maintainable The truth is that if you are not following framework idioms, you are very likely not delivering maintainable software
- rafaelmn 2y ago
  > Using LLMs to bootstrap development in unfamiliar frameworks is a valuable approach when balanced with pragmatic engineering principles. While framework idioms are important, they're secondary to delivering working, maintainable software that solves real problems. Except that using the original approach would make it hard to navigate the project after a few weeks of development - duplicate logic all over, inconsistency in duplicates, bugs. Rewriting the code to actually use the framework as intended let me progress much faster and reuse the code. And as someone who has had gigs with 6 different languages/stacks at this point, and played with probably as much on the side - that's a nice sentiment in theory but doesn't reproduce in practice. There's definite learning curve when using a new stack/language - sure your experience makes that curve different to a newbie, but in my experience it's going to be a few months until you can move as quickly as you can with a stack you're familiar with.

Tostino 2y ago

Right. I've had that same problem when trying to build a new app with Claude / 4o. I essentially have to go through the docs myself so I understand the framework structure enough to write up an outline for the llm to follow when developing a new component or backend structure.

dagw 2y ago
With Claude I've had great success with first uploading the docs so that it has them as context for the rest of the session. Without doing that I got all kinds of answers, sometimes using a different framework or an older version of the framework I was using. Once it had the documentation to reference it started giving a lot better answers
- Bluestein 2y ago
  Great point. Great suggestion.-

cadamsdotcom 2y ago

Very interesting outcome. Shows how far it’s possible to get without being idiomatic. You say the docs contain guidance on structuring, high-level constructs, and how to make things idiomatic. It would be an interesting test to hand the unfixed revision of the code to an LLM while also giving it the docs, and say “make any fixes to make this conform to standards of the framework and libraries”. If it picks up the same things that’s great news for novice programmers and anyone new to a framework! LLMs will improve at making these fixes over time - even if they’re currently bad at it that won’t last.

rafaelmn 2y ago
But that's kind of my point - when I was originally working on it - I didn't know there's a more idiomatic approach - why would I copy paste the docs to ask it to improve the results ? Especially given the LLMs tendency to hallucinate and gnerate noise when asked to "improve code". Also without reading the docs would I be able to spot why the idiomatic approach is better from single use case ? You're basically depending on LLM to make choices for you - and in my experience so far it makes very suboptimal ones.
sungho_ 2y ago
> It would be an interesting test to hand the unfixed revision of the code to an LLM while also giving it the docs, and say “make any fixes to make this conform to standards of the framework and libraries”. For your info, a few days ago, I was trying to migrate Tailwind 3 to 4 in a codebase where I had just set up the boilerplate, and I went through hell and eventually gave up. I used the Tailwind 4 documentation and Claude 3.5 sonnet in Windsurf IDE, and the part related to migration in my codebase was probably less than 50 lines. All of those tasks would take just a few minutes if a person did them directly (excluding reading the documents)

exhaze 2y ago

Using LLMs to bootstrap development in unfamiliar frameworks is a valuable approach when balanced with pragmatic engineering principles. While framework idioms are important, they're secondary to delivering working, maintainable software that solves real problems. The focus on "terrible structure" misses the point - what matters is whether the system meets its requirements efficiently and can be maintained effectively. Have you measured any actual negative impacts on system performance, maintainability, or customer value? My experience suggests that starting with working code and iteratively improving it as patterns emerge often leads to better outcomes than delaying development for complete framework mastery. The interview analogy is particularly misleading - success in a software engineering role isn't measured by framework knowledge, but by the ability to deliver value to customers effectively. Learning framework idioms can happen in parallel with productive development, especially when using LLMs to accelerate the initial learning curve.

bluefirebrand 2y ago
> While framework idioms are important, they're secondary to delivering working, *maintainable* software that solves real problems Emphasis maintainable The truth is that if you are not following framework idioms, you are very likely not delivering maintainable software
rafaelmn 2y ago
> Using LLMs to bootstrap development in unfamiliar frameworks is a valuable approach when balanced with pragmatic engineering principles. While framework idioms are important, they're secondary to delivering working, maintainable software that solves real problems. Except that using the original approach would make it hard to navigate the project after a few weeks of development - duplicate logic all over, inconsistency in duplicates, bugs. Rewriting the code to actually use the framework as intended let me progress much faster and reuse the code. And as someone who has had gigs with 6 different languages/stacks at this point, and played with probably as much on the side - that's a nice sentiment in theory but doesn't reproduce in practice. There's definite learning curve when using a new stack/language - sure your experience makes that curve different to a newbie, but in my experience it's going to be a few months until you can move as quickly as you can with a stack you're familiar with.