5 ms·

This post is absolutely devastating to me. Salvatore is surely one of the most capable software engineers working today. He can lucidly see that this supposed

This post is absolutely devastating to me. Salvatore is surely one of the most capable software engineers working today. He can lucidly see that this supposed tool is completely useless to him within his area of expertise. Then, rather than cast it off as the ill-fitting, bent screwdriver that it is, he accepts the boosters' premise that he must find some use for it. Just as any introductory macroeconomics class teaches, if one island has superior skill in producing widgets A, it doesn't matter how terrible the other island's skill at producing B is, we'll still see specialization where island A leverages island B. So of course antirez's relative ability in systems programming would relegate the LLM to other progamming tasks. However! We do not exist in isolation. There is a multitude of human beings around us, hungry for technical challenges and food. Many of them have or could obtain skills complimentary to our own. In working together, our cooperative efforts could be more than the sum of their parts. Perhaps the LLM is better at writing PyTorch code than antirez. Just because we have an old bent screwdriver in the garage doesn't mean we should try to use it. Perhaps we'd be better off heading to the hardware store today.

nuz 3y ago
The funny thing about LLMs is that there's no rush in adopting them. They're semi useful but not really right now, but you're not gonna be 'left behind' if you don't make use of them. Everyone involved is working their hardest to make them more capable so when that day comes, you'll just use it to prompt what you want. But there's no rush to try to squeeze out anything out of the current generation which mostly lowers productivity rather than increases it.
- fhd2 3y ago
  My thinking exactly! There's FOMO going on (and being fueled by people most of which seem to hope to somehow make money off it), but the barriers to entry for using LLMs are just not that high. When the tools are good enough, I'll happily use them. Today, for the work I do, I found that not to be the case yet. I wouldn't advocate for not trying them, but I see no reason to force yourself to use them.
  - unshavedyak 3y ago
    Yea. TBH the tooling is the bigger issue for me, personally. I come across a ton of times where an LLM might work, or could - potentially. Usually refactors. But it requires access to basically all my files, both for context and to find where references to that class/struct/etc are. Furthermore, i'd vastly prefer a workflow most of the time where i don't even have to ask. Or so i imagine. Ie i think i'd prefer a Clippy style "Want me to write a test for this? Want me to write some documentation for this?" etc helpers. I don't want to have to ask, i want it to intuit my needs - just like any programmer could if pair programming with you. And most of all i want it to have access to all files. To know everything about the code possible. Don't just look at the func name in isolation, attempt to understand how it's used in the project. If i have to baby sit an LLM for a simple function refactor to give it all files where the function is used or w/e, i'd rather do it myself with tools like AST Grep or even my LSP in many cases. I'm very interested in LLMs for simple tasks today, but the tooling feels like my primary blocker. Also possibly context length, but i think there's lots of ways around that.
    - thenevermind 3y ago
      Just to throw my 2c here, since I also want the models to access the whole codebase of (at least) the current project. I had great impression of sourcegraph’s cody. https://sourcegraph.com/cody https://sourcegraph.com/cody few months ago.At least with the enterprise version of the sourcegraph that had indexed most of the orgs private repos. The web ui (vscode extension was somehow worse, not sure why) was providing damn good responses, be it code generating or QA/explanation about code spanning through multiple repos. (e.g. terraform modules living in different repos) Afaiu it was using the sourcegraph index under the hood. But I never really deep dived into the cody’s design internals (not even sure if they are actually public) That being said, I’ve departed from the org months ago and haven’t used cody since then, so take this with a grain of salt, since the whole comment could be outdated a lot.
madeofpalk 3y ago
I think this is an extremely unfavorable interpretation of the article. I wonder if we even read the same thing? He sees a new tool that others have found interesting, and he identifies ways to use that tool that are useful for him, while also acknowledging where it's not useful. He backs it up with plenty of examples of where he found it not-useless. This is not a revolutionary insight, especially for a developer. We constantly use a variety of tools, such as programming languages, that have strengths and weaknesses. Why are LLMs so different? It seems foolish to claim they have zero strengths.
- HarHarVeryFunny 3y ago
  You might be surprised at the number of large companies who think that "GenAI" can be used to replace programmers due to non-technical executives having got the impression that a competency of LLMs is writing code ... Of course they do have uses, but more related to discovery of APIs and documentation than actually writing code (esp. where bugs matter) for the most part. I also have to wonder how long until open source code (e.g. GPL'd) regurgitated by LLMs and incorporated into corporate code bases becomes an issue. The C suite dudes seem concerned about employees using LLMs that may be publicly exposing their own company's code base, but illogically unworried about the reverse happening - maybe just due to not fully understanding the tech.
antirez 3y ago
If the LLM is better than me at writing Torch code, it is a great idea for me to use an LLM to write my model definition, since the exact syntax or the reshaping of the tensors is not so important to me. If I want to create a convnet and train it on my images, for my own usage, I don't need to bother some Torch expert to do it for me. I can do it myself, if I understand enough about convnets themselves and not enough about Torch syntax / methods. The alternative would be to study the details of Torch in a manual, and the end result would be the same: the important thing in this task is to govern the ML concepts, not the details of MLX, Keras or PyTorch.
- couchand 3y ago
  > I don't need to bother some Torch expert to do it for me. This is, indeed, the core of our disagreement. You seem confident it would be a bother to others to ask for help. I'm confident that there are many who would value the opportunity to collaborate with you. I feel sure that whatever analysis you're doing could benefit from the sounding board of a domain expert, and you'd both benefit from the exchange. EDIT: to clarify, being "famous" has nothing to do with it. Each of us has worth and we all would gain by working with others.
  - hobs 3y ago
    That's a pretty big exception to the general case though if your argument is "you are a famous person and people would love to collab" - at 2am your time, exactly when you are motivated? And what if you are not a world famous programmer, what then? To me its like saying you shouldn't play solitaire because you are a world class poker player, and there's plenty of people who want to game with you. They are orthogonal concepts - just communicating with people can be more work than just reading on your own.
  - deleted 3y ago
    [deleted]
  - kreetx 3y ago
    It's more of the scale you do your thing. For the quick and dirty thing, I'm going to write Torch model within the hours - where would I find a collaborator who is willing to start immediately within that time?
rolisz 3y ago
So you're suggesting that instead of asking an LLM, we should spend time on Fiverr/Upwork to find someone to do random coding tasks that might not fall under our expertise? Can you do that for less than 20$/month?
- couchand 3y ago
  I agree there are difficult coordination problems that our society has failed to grapple with, let alone solve.
esafak 3y ago
I don't understand which part was devastating.
- kvz 3y ago
  A respectable person offering nuance can be shattering if you identify as pro- or anti something. I wish we had an easier time talking about ideas with a little more detachment.
nkohari 3y ago
> He can lucidly see that this supposed tool is completely useless to him within his area of expertise. Did you read the article? Throughout the entire post he clearly says LLMs have a lot of value in his workflow.
- sevagh 3y ago
  Parent seems to think that Antirez has been begrudgingly pulled along by the tide of LLM hype, not as if Antirez is an accomplished developer whose judgement on tools can be trusted.
throwuwu 3y ago
Bad metaphor and worse that you use it to inform your conclusion. If you must have one then use training wheels, experts don’t need training wheels beginners do, simple as that. Though the utility of LLMs goes much further as the author points out, it can often make the boring or tedious parts easier so you can add assistive peddling to the metaphor. To carry it to the end, once you have four wheels and a motor it’s not long before someone invents the car.
Symmetry 3y ago
In a frictionless market it would make sense to do that. But as Coase pointed out nearly a century ago[1] forming and monitoring the relationships that allow specialization involves a certain amount of overhead. At a certain point going through the hiring or vetting process to utilize another person's skills makes sense, but it looks like the author is very far from that point. [1]https://en.wikipedia.org/wiki/The_Nature_of_the_Firm https://en.wikipedia.org/wiki/The_Nature_of_the_Firm
antonvs 3y ago
This seems like a very impractical perspective to me. If there really was a "hardware store" that we could head off to to get what we need, it might be different. But in general, that's not the case. There can also be significant overhead in looking elsewhere for a solution. That's a big part of why so many developers reinvent things. This is often dismissed as NIH syndrome, but there's more to it than that. You raised "introductory macroeconomics". The economic effect that will most strongly apply in the case of LLMs is that of the technology treadmill (Cochrane 1958): when there's a tool that can improve productivity, it will be used competitively so that those who don't use it effectively, to improve their productivity, will be outcompeted. This seems like an unavoidable result in typical capitalist economies. Your point about leveraging hungry humans would require strong incentives to overcome the treadmill effect. Most Western countries don't have many ways to implement anything like that. The closest thing might be unions, but of course most software development is not unionized.
bbor 3y ago
Like… hiring people? I think it’s a little rediculous to say “no don’t use that screwdriver, go hire a workman instead.” To say the least, there are some sizeable economic differences between those two options

nuz 3y ago

The funny thing about LLMs is that there's no rush in adopting them. They're semi useful but not really right now, but you're not gonna be 'left behind' if you don't make use of them. Everyone involved is working their hardest to make them more capable so when that day comes, you'll just use it to prompt what you want. But there's no rush to try to squeeze out anything out of the current generation which mostly lowers productivity rather than increases it.

fhd2 3y ago
My thinking exactly! There's FOMO going on (and being fueled by people most of which seem to hope to somehow make money off it), but the barriers to entry for using LLMs are just not that high. When the tools are good enough, I'll happily use them. Today, for the work I do, I found that not to be the case yet. I wouldn't advocate for not trying them, but I see no reason to force yourself to use them.
- unshavedyak 3y ago
  Yea. TBH the tooling is the bigger issue for me, personally. I come across a ton of times where an LLM might work, or could - potentially. Usually refactors. But it requires access to basically all my files, both for context and to find where references to that class/struct/etc are. Furthermore, i'd vastly prefer a workflow most of the time where i don't even have to ask. Or so i imagine. Ie i think i'd prefer a Clippy style "Want me to write a test for this? Want me to write some documentation for this?" etc helpers. I don't want to have to ask, i want it to intuit my needs - just like any programmer could if pair programming with you. And most of all i want it to have access to all files. To know everything about the code possible. Don't just look at the func name in isolation, attempt to understand how it's used in the project. If i have to baby sit an LLM for a simple function refactor to give it all files where the function is used or w/e, i'd rather do it myself with tools like AST Grep or even my LSP in many cases. I'm very interested in LLMs for simple tasks today, but the tooling feels like my primary blocker. Also possibly context length, but i think there's lots of ways around that.
  - thenevermind 3y ago
    Just to throw my 2c here, since I also want the models to access the whole codebase of (at least) the current project. I had great impression of sourcegraph’s cody. https://sourcegraph.com/cody https://sourcegraph.com/cody few months ago.At least with the enterprise version of the sourcegraph that had indexed most of the orgs private repos. The web ui (vscode extension was somehow worse, not sure why) was providing damn good responses, be it code generating or QA/explanation about code spanning through multiple repos. (e.g. terraform modules living in different repos) Afaiu it was using the sourcegraph index under the hood. But I never really deep dived into the cody’s design internals (not even sure if they are actually public) That being said, I’ve departed from the org months ago and haven’t used cody since then, so take this with a grain of salt, since the whole comment could be outdated a lot.

madeofpalk 3y ago

I think this is an extremely unfavorable interpretation of the article. I wonder if we even read the same thing? He sees a new tool that others have found interesting, and he identifies ways to use that tool that are useful for him, while also acknowledging where it's not useful. He backs it up with plenty of examples of where he found it not-useless. This is not a revolutionary insight, especially for a developer. We constantly use a variety of tools, such as programming languages, that have strengths and weaknesses. Why are LLMs so different? It seems foolish to claim they have zero strengths.

HarHarVeryFunny 3y ago
You might be surprised at the number of large companies who think that "GenAI" can be used to replace programmers due to non-technical executives having got the impression that a competency of LLMs is writing code ... Of course they do have uses, but more related to discovery of APIs and documentation than actually writing code (esp. where bugs matter) for the most part. I also have to wonder how long until open source code (e.g. GPL'd) regurgitated by LLMs and incorporated into corporate code bases becomes an issue. The C suite dudes seem concerned about employees using LLMs that may be publicly exposing their own company's code base, but illogically unworried about the reverse happening - maybe just due to not fully understanding the tech.

antirez 3y ago

If the LLM is better than me at writing Torch code, it is a great idea for me to use an LLM to write my model definition, since the exact syntax or the reshaping of the tensors is not so important to me. If I want to create a convnet and train it on my images, for my own usage, I don't need to bother some Torch expert to do it for me. I can do it myself, if I understand enough about convnets themselves and not enough about Torch syntax / methods. The alternative would be to study the details of Torch in a manual, and the end result would be the same: the important thing in this task is to govern the ML concepts, not the details of MLX, Keras or PyTorch.

couchand 3y ago
> I don't need to bother some Torch expert to do it for me. This is, indeed, the core of our disagreement. You seem confident it would be a bother to others to ask for help. I'm confident that there are many who would value the opportunity to collaborate with you. I feel sure that whatever analysis you're doing could benefit from the sounding board of a domain expert, and you'd both benefit from the exchange. EDIT: to clarify, being "famous" has nothing to do with it. Each of us has worth and we all would gain by working with others.
- hobs 3y ago
  That's a pretty big exception to the general case though if your argument is "you are a famous person and people would love to collab" - at 2am your time, exactly when you are motivated? And what if you are not a world famous programmer, what then? To me its like saying you shouldn't play solitaire because you are a world class poker player, and there's plenty of people who want to game with you. They are orthogonal concepts - just communicating with people can be more work than just reading on your own.
- deleted 3y ago
  [deleted]
- kreetx 3y ago
  It's more of the scale you do your thing. For the quick and dirty thing, I'm going to write Torch model within the hours - where would I find a collaborator who is willing to start immediately within that time?

rolisz 3y ago

So you're suggesting that instead of asking an LLM, we should spend time on Fiverr/Upwork to find someone to do random coding tasks that might not fall under our expertise? Can you do that for less than 20$/month?

couchand 3y ago
I agree there are difficult coordination problems that our society has failed to grapple with, let alone solve.

esafak 3y ago

I don't understand which part was devastating.

kvz 3y ago
A respectable person offering nuance can be shattering if you identify as pro- or anti something. I wish we had an easier time talking about ideas with a little more detachment.

nkohari 3y ago

> He can lucidly see that this supposed tool is completely useless to him within his area of expertise. Did you read the article? Throughout the entire post he clearly says LLMs have a lot of value in his workflow.

sevagh 3y ago
Parent seems to think that Antirez has been begrudgingly pulled along by the tide of LLM hype, not as if Antirez is an accomplished developer whose judgement on tools can be trusted.

throwuwu 3y ago

Bad metaphor and worse that you use it to inform your conclusion. If you must have one then use training wheels, experts don’t need training wheels beginners do, simple as that. Though the utility of LLMs goes much further as the author points out, it can often make the boring or tedious parts easier so you can add assistive peddling to the metaphor. To carry it to the end, once you have four wheels and a motor it’s not long before someone invents the car.

Symmetry 3y ago

In a frictionless market it would make sense to do that. But as Coase pointed out nearly a century ago[1] forming and monitoring the relationships that allow specialization involves a certain amount of overhead. At a certain point going through the hiring or vetting process to utilize another person's skills makes sense, but it looks like the author is very far from that point. [1]https://en.wikipedia.org/wiki/The_Nature_of_the_Firm https://en.wikipedia.org/wiki/The_Nature_of_the_Firm

antonvs 3y ago

This seems like a very impractical perspective to me. If there really was a "hardware store" that we could head off to to get what we need, it might be different. But in general, that's not the case. There can also be significant overhead in looking elsewhere for a solution. That's a big part of why so many developers reinvent things. This is often dismissed as NIH syndrome, but there's more to it than that. You raised "introductory macroeconomics". The economic effect that will most strongly apply in the case of LLMs is that of the technology treadmill (Cochrane 1958): when there's a tool that can improve productivity, it will be used competitively so that those who don't use it effectively, to improve their productivity, will be outcompeted. This seems like an unavoidable result in typical capitalist economies. Your point about leveraging hungry humans would require strong incentives to overcome the treadmill effect. Most Western countries don't have many ways to implement anything like that. The closest thing might be unions, but of course most software development is not unionized.

bbor 3y ago

Like… hiring people? I think it’s a little rediculous to say “no don’t use that screwdriver, go hire a workman instead.” To say the least, there are some sizeable economic differences between those two options