5 ms·

> AI and Tooling Support: It’s worth noting that AI tools are intimately familiar with Next.js and not so much with htmx This is stated as a very matter-of-fac

by ketzo 2y ago

> AI and Tooling Support: It’s worth noting that AI tools are intimately familiar with Next.js and not so much with htmx This is stated as a very matter-of-fact downside, but this is a pretty crazy portent for the future of dev tools / libraries / frameworks / languages. Predictions: - LLMs will further amplify the existing winner-take-all, first-mover nature of dev tools - LLMs will encourage usage of open-source tools because they will be so much more useful with more/better training data

wodenokoto 2y ago
I saw a tool that had a page dedicated to AI to read. Basically you would point your llm to that page as the initial prompt and from there could start asking questions. I thought it was an interesting idea, but apparently not interesting enough to remember who did it or even check how that page looked.
- q0uaur 2y ago
  its actually a big thing i am waiting for - both websites and ai tools to agree on a way to facilitate this. i'm doing some game development in godot as a hobby, and the current llm's are really bad at it - very often i get code suggestions that use ancient versions of gdscript or the engine. I'd love to have a big enough context window and the tooling needed to go like "look at these godot docs for the current version: (insert link)" and then ask my questions, i think it would fix 99% of these issues. same with other less well-known tools and languages.
ryanchants 2y ago
I was considering this other day. AI tools are stuck at a particular point in time. And even training them on newer stuff, there's only so much information to train on. I've been exploring this being a _good_ thing. In software we spend so much time chasing the latest tooling, language features, frameworks, etc. Maybe it'll be a positive that it all stagnates a bit and we just use the tools we have to get work done instead of creating new hammers every 6 months.
- treyd 2y ago
  The problem is that for most domains our hammers are still pretty bad.
  - mattgreenrocks 2y ago
    Agree. A lot of the most popular libraries/frameworks are not necessarily the best. Removing more fitness checks will only worsen this problem.
- breuleux 2y ago
  It would be nice if some AI tools could be developed to actually evaluate new libraries and frameworks. For instance, if there are already 10 libraries to do something and I develop a new one that's objectively faster than all of them (true story), could some AI do the work of installing and benchmarking it and incorporate the results in its knowledge base? And periodically update it? Is there any way to leverage AI for discovery of solid code? I suspect this is beyond current capabilities, but one can dream.
davidhaymond 2y ago
I had written a comment addressing this as well but you beat me to it. In a way it is similar to the effect StackOverflow had on popular libraries, but amplified. Even without StackOverflow, a library can do well if it has good documentation. I'm not sure if the same holds true with LLMs.
diggan 2y ago
My prediction is that it'll be like this for a while, but as soon as tooling becomes better and the context of current APIs + local files gets better taken into consideration, this "advantage" will disappear.
ClassAndBurn 2y ago
This will not be true for future frameworks, though it is likely true for current ones. Future frameworks will be designed for AI and enablement. There will be a reversal in convention-over-configuration. Explicit referencing and configuration allow models to make fewer assumptions with less training. All current models are trained on good and bad examples of existing frameworks. This is why asking an LLM to “code like John Carmack” produces better code.. Future frameworks can quickly build out example documentation and provide it within the framework for AI tools to reference directly.
- riffraff 2y ago
  I don't think convention over configuration causes LLMs any problems, GitHub copilot generates code matching rails conventions quite easily for example.
  - freeone3000 2y ago
    Because there’s enough rails code in the training data to determine the proper conventions :) if you’re making something new without this glut of data, it’s going to be much more difficult for a coding assistant to match a convention it’s never seem before.
    - riffraff 2y ago
      true, but the conventions it has seen are the same across all similar domains not just same framework/language, copilot "picks up" the similarity. What I mean is: if you name your modules consistently, say Operation::Object::Verb or Action::ObjectVerb or ObjectManager.doSomething it's really easy for the LLM to guess the next one, just as it is easy for a human. Add a new file actions/users/update.rb and start typing "Act" and it may guess "class Actions::Users::Update, and start to fill in the code based on nearby modules, switch to the corresponding unit test and it'll fill it in too. Source: we have our own in-house conventions and it seems copilot gets them right most of the time, ymmv.
    - skydhash 2y ago
      The thing is, with some elbow grease, you can write a great plugin for your preferred editor. No need for dubious LLMs results, especially when the difficult part, code intellisense, is already solved with LSP. If you're a shop that has invested in a framework, it would be cheaper and more productive.
jimbokun 2y ago
And: - developers will be incapable of writing or debugging code for development stacks or project types without LLMs trained on lots of matching examples
spankalee 2y ago
I'm not too worried about this, and I think Gumroad's concern is likely overblown. I can't tell from their comment whether they actually experienced AI being bad at HTMX, or if they transitioned to talking about other resources. LLMs are often wildly good at being universal translators. So if they pick up general patterns and concepts in popular frameworks, and enough syntax of more niche frameworks, IME, they do a pretty great job of generating good niche framework code. Similar to how they can talk like a pirate about things things pirates never said in their training data.
- rmbyrro 2y ago
  In my experience, ChatGPT and Github Copilot are significantly worse at htmx compared to mainstream tech. They hallucinate A LOT more.
  - internetter 2y ago
    When this happens for me with Copilot, I increase the context by opening other files, and then it works fine.
    - azemetre 2y ago
      How does this work in copilot if you use the neovim plugin? Open lots of buffers?
- ffsm8 2y ago
  > they do a pretty great job of generating good niche framework code. If you mean plausible looking code, yes - totally! If you mean actually usable code: nope. Its always riddled with imaginary library calls that don't exist.
alexpetros 2y ago
> LLMs will further amplify the existing winner-take-all, first-mover nature of dev tools This will be true for people who rely on LLMs to code, which I strongly suggest is not a great long-term bet for a software engineering career.
- jimmaswell 2y ago
  I still think it's a perfectly fine productivity multiplier to use as long as you still understand what you're doing.
  - alexpetros 2y ago
    There's a very clear skill ceiling on the kind of code for which LLMs can serve as a productivity multiplier.
    - jimmaswell 2y ago
      I don't think so. No matter the code complexity, copilot gives me good enough suggestions that it saves me time. It often simply writes the exact line I had in mind.
- MetaWhirledPeas 2y ago
  Yes. And no matter how good LLMs get at coding there will always be a crowd intentionally doing it themselves, especially in the open source arena, if only just to keep the joy alive.
  - steve_adams_86 2y ago
    I fully expect to keep writing code for decades, just like I still use a hand plane and enjoy growing plants to eat. I know other methods are more efficient. But I’m here to experience life. I want to do the hard things. I want to be uncertain, confused, to make mistakes, and to learn. AI is a neat tool in limited scopes. Every time I use it though, I feel like I didn’t get the full experience. I’m acutely aware of all the rabbit holes I missed, or the tiny details I’d notice along the way and helpfully remember 10 years later. For everything it adds, it takes at least as much away from where I’m looking. And I like those parts.
SrslyJosh 2y ago
If your devs can't work without something writing their code for them, why are you hiring them?
- tom85 2y ago
  Maybe they are just faster with AI.
- cheema33 2y ago
  > If your devs can't work without something writing their code for them, why are you hiring them? I am currently in the process of hiring a backend engineer. Anybody who does not use AI to aid development work gets an automatic disqualification. In my experience, a good engineer using AI tools will run circles around a good engineer not using AI tools.
  - majewsky 2y ago
    > In my experience, a good engineer using AI tools will run circles around a good engineer not using AI tools. Even supposing that is true, what if you have a choice between a bad engineer who uses AI and a good engineer that won't?
  - bccdee 2y ago
    "A good cyclist using training wheels will go circles around a good cyclist who doesn't." No, training wheels only help bad cyclists. You can't generalize that and assume they will make a good cyclist even better. AIs generate deeply mediocre code. This is better than anything a person who can't code on their own would produce, but an experienced developer will have to spend all their time babysitting the AI to get it to behave properly.
    - smabie 2y ago
      Turns out mediocre code is fine most of the time though
      - 2y ago
wwarner 2y ago
respectfully disagree. i think that the value of llm suggestions are driving us toward a kind of standardization that is really good. we'll all be java programmers soon!
markmark 2y ago
The AI and Tooling support point is really just an extension of the Community and Ecosystem point. Even before LLMs React had an advantage in that every question you had was probably already on StackOverflow and there are mature React libraries for almost everything. Now some people might use an LLM to answer the question they previously would have gone to StackOverflow for but the outcome is the same: there are advantages to using what other people are using.
- mvkel 2y ago
  Exactly this. The language with the biggest community wins, even if it may be technically inferior
killerstorm 2y ago
In my experience LLMs are good at picking up syntax from the context. Recently I used Cursor with an in-house developed language and it was able to get almost perfect completions upon seeing ~100 lines of context. Not all LLMs are equally good at this: Gemini seems to be particularly resistant to adapting to a new syntax
bitbasher 2y ago
> LLMs will further amplify the existing winner-take-all, first-mover nature of dev tools Or, taken another way-- developers not using LLMs will be more likely to be first-mover winners.

by ketzo 2y ago

wodenokoto 2y ago

I saw a tool that had a page dedicated to AI to read. Basically you would point your llm to that page as the initial prompt and from there could start asking questions. I thought it was an interesting idea, but apparently not interesting enough to remember who did it or even check how that page looked.

q0uaur 2y ago
its actually a big thing i am waiting for - both websites and ai tools to agree on a way to facilitate this. i'm doing some game development in godot as a hobby, and the current llm's are really bad at it - very often i get code suggestions that use ancient versions of gdscript or the engine. I'd love to have a big enough context window and the tooling needed to go like "look at these godot docs for the current version: (insert link)" and then ask my questions, i think it would fix 99% of these issues. same with other less well-known tools and languages.

ryanchants 2y ago

I was considering this other day. AI tools are stuck at a particular point in time. And even training them on newer stuff, there's only so much information to train on. I've been exploring this being a _good_ thing. In software we spend so much time chasing the latest tooling, language features, frameworks, etc. Maybe it'll be a positive that it all stagnates a bit and we just use the tools we have to get work done instead of creating new hammers every 6 months.

treyd 2y ago
The problem is that for most domains our hammers are still pretty bad.
- mattgreenrocks 2y ago
  Agree. A lot of the most popular libraries/frameworks are not necessarily the best. Removing more fitness checks will only worsen this problem.
breuleux 2y ago
It would be nice if some AI tools could be developed to actually evaluate new libraries and frameworks. For instance, if there are already 10 libraries to do something and I develop a new one that's objectively faster than all of them (true story), could some AI do the work of installing and benchmarking it and incorporate the results in its knowledge base? And periodically update it? Is there any way to leverage AI for discovery of solid code? I suspect this is beyond current capabilities, but one can dream.

davidhaymond 2y ago

I had written a comment addressing this as well but you beat me to it. In a way it is similar to the effect StackOverflow had on popular libraries, but amplified. Even without StackOverflow, a library can do well if it has good documentation. I'm not sure if the same holds true with LLMs.

diggan 2y ago

My prediction is that it'll be like this for a while, but as soon as tooling becomes better and the context of current APIs + local files gets better taken into consideration, this "advantage" will disappear.

ClassAndBurn 2y ago

This will not be true for future frameworks, though it is likely true for current ones. Future frameworks will be designed for AI and enablement. There will be a reversal in convention-over-configuration. Explicit referencing and configuration allow models to make fewer assumptions with less training. All current models are trained on good and bad examples of existing frameworks. This is why asking an LLM to “code like John Carmack” produces better code.. Future frameworks can quickly build out example documentation and provide it within the framework for AI tools to reference directly.

riffraff 2y ago
I don't think convention over configuration causes LLMs any problems, GitHub copilot generates code matching rails conventions quite easily for example.
- freeone3000 2y ago
  Because there’s enough rails code in the training data to determine the proper conventions :) if you’re making something new without this glut of data, it’s going to be much more difficult for a coding assistant to match a convention it’s never seem before.
  - riffraff 2y ago
    true, but the conventions it has seen are the same across all similar domains not just same framework/language, copilot "picks up" the similarity. What I mean is: if you name your modules consistently, say Operation::Object::Verb or Action::ObjectVerb or ObjectManager.doSomething it's really easy for the LLM to guess the next one, just as it is easy for a human. Add a new file actions/users/update.rb and start typing "Act" and it may guess "class Actions::Users::Update, and start to fill in the code based on nearby modules, switch to the corresponding unit test and it'll fill it in too. Source: we have our own in-house conventions and it seems copilot gets them right most of the time, ymmv.
  - skydhash 2y ago
    The thing is, with some elbow grease, you can write a great plugin for your preferred editor. No need for dubious LLMs results, especially when the difficult part, code intellisense, is already solved with LSP. If you're a shop that has invested in a framework, it would be cheaper and more productive.

jimbokun 2y ago

And: - developers will be incapable of writing or debugging code for development stacks or project types without LLMs trained on lots of matching examples

spankalee 2y ago

I'm not too worried about this, and I think Gumroad's concern is likely overblown. I can't tell from their comment whether they actually experienced AI being bad at HTMX, or if they transitioned to talking about other resources. LLMs are often wildly good at being universal translators. So if they pick up general patterns and concepts in popular frameworks, and enough syntax of more niche frameworks, IME, they do a pretty great job of generating good niche framework code. Similar to how they can talk like a pirate about things things pirates never said in their training data.

rmbyrro 2y ago
In my experience, ChatGPT and Github Copilot are significantly worse at htmx compared to mainstream tech. They hallucinate A LOT more.
- internetter 2y ago
  When this happens for me with Copilot, I increase the context by opening other files, and then it works fine.
  - azemetre 2y ago
    How does this work in copilot if you use the neovim plugin? Open lots of buffers?
ffsm8 2y ago
> they do a pretty great job of generating good niche framework code. If you mean plausible looking code, yes - totally! If you mean actually usable code: nope. Its always riddled with imaginary library calls that don't exist.

alexpetros 2y ago

> LLMs will further amplify the existing winner-take-all, first-mover nature of dev tools This will be true for people who rely on LLMs to code, which I strongly suggest is not a great long-term bet for a software engineering career.

jimmaswell 2y ago
I still think it's a perfectly fine productivity multiplier to use as long as you still understand what you're doing.
- alexpetros 2y ago
  There's a very clear skill ceiling on the kind of code for which LLMs can serve as a productivity multiplier.
  - jimmaswell 2y ago
    I don't think so. No matter the code complexity, copilot gives me good enough suggestions that it saves me time. It often simply writes the exact line I had in mind.
MetaWhirledPeas 2y ago
Yes. And no matter how good LLMs get at coding there will always be a crowd intentionally doing it themselves, especially in the open source arena, if only just to keep the joy alive.
- steve_adams_86 2y ago
  I fully expect to keep writing code for decades, just like I still use a hand plane and enjoy growing plants to eat. I know other methods are more efficient. But I’m here to experience life. I want to do the hard things. I want to be uncertain, confused, to make mistakes, and to learn. AI is a neat tool in limited scopes. Every time I use it though, I feel like I didn’t get the full experience. I’m acutely aware of all the rabbit holes I missed, or the tiny details I’d notice along the way and helpfully remember 10 years later. For everything it adds, it takes at least as much away from where I’m looking. And I like those parts.

SrslyJosh 2y ago

If your devs can't work without something writing their code for them, why are you hiring them?

tom85 2y ago
Maybe they are just faster with AI.
cheema33 2y ago
> If your devs can't work without something writing their code for them, why are you hiring them? I am currently in the process of hiring a backend engineer. Anybody who does not use AI to aid development work gets an automatic disqualification. In my experience, a good engineer using AI tools will run circles around a good engineer not using AI tools.
- majewsky 2y ago
  > In my experience, a good engineer using AI tools will run circles around a good engineer not using AI tools. Even supposing that is true, what if you have a choice between a bad engineer who uses AI and a good engineer that won't?
- bccdee 2y ago
  "A good cyclist using training wheels will go circles around a good cyclist who doesn't." No, training wheels only help bad cyclists. You can't generalize that and assume they will make a good cyclist even better. AIs generate deeply mediocre code. This is better than anything a person who can't code on their own would produce, but an experienced developer will have to spend all their time babysitting the AI to get it to behave properly.
  - smabie 2y ago
    Turns out mediocre code is fine most of the time though
    - 2y ago

wwarner 2y ago

respectfully disagree. i think that the value of llm suggestions are driving us toward a kind of standardization that is really good. we'll all be java programmers soon!

markmark 2y ago

The AI and Tooling support point is really just an extension of the Community and Ecosystem point. Even before LLMs React had an advantage in that every question you had was probably already on StackOverflow and there are mature React libraries for almost everything. Now some people might use an LLM to answer the question they previously would have gone to StackOverflow for but the outcome is the same: there are advantages to using what other people are using.

mvkel 2y ago
Exactly this. The language with the biggest community wins, even if it may be technically inferior

killerstorm 2y ago

In my experience LLMs are good at picking up syntax from the context. Recently I used Cursor with an in-house developed language and it was able to get almost perfect completions upon seeing ~100 lines of context. Not all LLMs are equally good at this: Gemini seems to be particularly resistant to adapting to a new syntax

bitbasher 2y ago

> LLMs will further amplify the existing winner-take-all, first-mover nature of dev tools Or, taken another way-- developers not using LLMs will be more likely to be first-mover winners.