5 ms·

There's an unfortunately common take on AI that goes basically like this: "I tried it and it didn't do what I wanted, not impressed." My suggestion is to tune

by gdubs 4y ago

There's an unfortunately common take on AI that goes basically like this: "I tried it and it didn't do what I wanted, not impressed." My suggestion is to tune out the noise and really try experimenting with these tools – and know that they're rapidly improving. Even if ultimately you have criticisms or decide one way or another, at least really investigate them for your own use-cases rather than jumping on a bandwagon that's either "AI is bad" or the breathless hype-machine at the other end.

rootusrootus 4y ago
I agree it's a good idea to take a moderate approach. The hype that LLMs are going to replace SWEs is clearly just that, hype, if you've done any real work trying to get GPT4 to give you the code you want. But it's also clearly a very useful tool. I think it'll absolutely destroy Stack Overflow.
- lcnPylGDnU4H9OF 4y ago
  > destroy Stack Overflow It'll be interesting to see how future training data is sourced.
  - rootusrootus 4y ago
    Github would be my first guess.
    - lcnPylGDnU4H9OF 4y ago
      That does seem like a likely option. Discussions on issues alongside the actual working (and not working) code.
      - deleted 4y ago
        [deleted]
  - svachalek 4y ago
    You simply need the system to train itself on its own interactions, like how search engines improve results by counting clicks.
    - lcnPylGDnU4H9OF 4y ago
      I'm not wondering about how the system will determine what's most helpful but instead determining what's even "correct". A model will learn what's "correct" from Stack Overflow by finding accepted or highly-voted answers but when it can't find such content anymore (in this case because Stack Overflow is hypothetically gone) then what would even exist to generate these discussions to be used as training data? Github, per the sibling comment, is a good example because projects will have issues (tied to the individual repository of source code to be seen as a working implementation of the idea) which will be where such discussions happen.
      - LawTalkingGuy 4y ago
        Those topics that AI replaces the forums for won't need discussion. People won't be confused about that thing because the coding AI knows the details of it. Soon that'll be most syntax questions, soon simple to mid-level algorithms, etc. People will move on to higher-level questions.
      - ok_dad 4y ago
        When Google search became important, people structured their information so that Google could best index it. When AIs become important in the same way, people will start to structure their information so that a particular class of AI can best index it. If that involves API documentation, perhaps there will be a standard format that AIs understand the best.
        stygiansonic 4y ago
        The difference is that folks had an incentive to make their pages easily indexable: drive page views. With becoming training data the incentive to the creators is a little less clear
- z3c0 4y ago
  I am very critical of the LLM hype, but the threat to stackoverflow is evident. Like stackoverflow, I never write code verbatim that comes from even GPT4. I frequently find issues in the output, as the code I write is generally very context-specific. However, I find the back-and-forth with interesting tidbits of info dropped here-and-there amounts to something like rubber duck debugging on steroids.
- deleted 4y ago
  [deleted]
- drowsspa 4y ago
  Yet, the whole movement of getting blue collar workers to code seems to have lost its steam.
  - gumballindie 4y ago
    Probably because “graduating” bootcamps doesnt make one a swe and people figured out it’s a scam?
    - antifa 4y ago
      I'm a fullstack dev of 10+ years experience. We have people on our team who've done almost nothing more than a react bootcamp. They can't do what I do, even with chatGPT, but they can do a lot of things I don't want to do.
    - valray 4y ago
      Also it requires an inherent level of talent which gets turned into skill. Not everyone who takes piano and music harmony lessons can become a jazz composer, especially in 6 weeks or 6 months.
- tarruda 4y ago
  > The hype that LLMs are going to replace SWEs is clearly just that, hype LLMs cannot replace anyone, but it is clear that engineers which master LLMs usage might multiply their productivity by a lot. The question is: If one LLM assisted engineer can work 10x faster, will companies reduce their engineer staff by 90%?
  - majormajor 4y ago
    I've worked at far more companies with miles of product idea backlog we never get to than ones with nothing for engineering to do. Now product will be able to use an LLM to come up with feature proposals and design docs even faster! :o So: are you working at a company where engineering is a cost center or a revenue center? The latter wants to get more done at the same cost much more than it wants to just cut spend.
    - abhibeckert 4y ago
      I'm working at a company where we perpetually don't have enough engineers and we've learned (the hard way) that adding more tends to make the problem worse (too many cooks). Copilot helps a lot with that. I can write `// check if em` and it will finish the comment (email is valid) and write the actual check. It's a massive productivity boost. I'm spending less time looking stuff up (what was that function name again?) and less time fixing typos/basic syntax errors. Copilot doesn't always know the function name, but it does more often than me. And it makes errors too, but again less often than me (also, because I'm reading the code, not typing it, I tend to see the error immediately without needing to test it).
  - nuancebydefault 4y ago
    To answer your question with a question if I may -- when did productivity increase in software ever result in headcount reduction? The competition also will have similar productivity gain.
    - MacsHeadroom 4y ago
      >when did productivity increase in software ever result in headcount reduction? The competition also will have similar productivity gain. The average AI company has like 1 employee per $25M valuation. That's around 25x fewer employees than the typical tech company.
- peterashford 4y ago
  Of course, there's the issue that a lot of the info for useful LLMs probably comes from places like Stack Overflow
- chefandy 4y ago
  Nobody who professionally designs and writes software AND has used LLM code generation tools sees this as a drop-in replacement for developers, generally, anytime soon. That stance is for overeager, credulous enthusiasts and doomsdayers jumping to conclusions. Similarly, nobody who professionally designs and creates complex art products sees this as a drop-in replacement for commercial artists anytime soon. That stance is for people dazzled by their new image-generation superpower who don't know how little they know about professional creative work. I doubt the markets for utility-grade code work (e.g. customizing existing WordPress themes) or low-effort, high-volume creative assets (template-based logos, lightly customized game sprites) will survive. They're still real people with lives and families and medical bills and mortgages and we really ought to get serious about worker protections in this country. Seriously.
  - valray 4y ago
    In a few years, as mid-level cognitive tasks get automated by LLMs, resulting in elimination of some percent of well-paying white-collar jobs, there will be economic dislocation and social disruption. Oligarchic capitalist societies with a hypocritical philosophy of free market economics (such as the USA) will experience social unrest and civil strife. In the meantime, social-democratic societies that have effective governance and can grow their safety net with universal basic income will be advantaged in this new economic order. Thinking Scandinavian and some Asian economies. The geopolitical balance of power will shift toward stable societies that are able to make the conceptual leap to UBI. Others who follow the primitive fantasy of free market economics will crumble and get left behind. At least that is how it looks right now.
    - chefandy 4y ago
      Maybe. I don't see any reason to assume the US won't continue to successfully a) escalate police power to keep citizens under control, b) continue giving people just enough crumbs to keep them from getting violent, and c) propagandize the hell out of the idea that societal mismanagement is a matter of personal responsibility. China has done pretty well while clamping down on civil liberties, and the US has always been better at obfuscating and spinning similar tactics to be more palatable to its citizens. And you only need to look at the rust belt to see what happens when US business leadership decides to follow the cheaper option. Though, regardless of ensuing social unrest, the fact that the people involved are human beings should be enough to not treat them like used condoms when computers figure out how to do their jobs. Should be.
- lionkor 4y ago
  Sadly I dont think this can happen. There is a load of trash answers on SO, and you bet ChatGPT is trained on that. So you get not only the good of SO, you also get the worst of SO, and theres no way to tell. Just a downgrade for me, plus for most things I do you are better of reading the source code or the documentation (however lackluster) than fumbling with chatgpt and getting an answer that may or may not be right. I might as well ask someone else who doesnt know any programming to search for the answer for me - they wont be able to tell a trustworthy answer from another one. There are so many SO answers (esp. on C++) which look good, but one of the comments points out some edge case in which it breaks. Remember, not everyone does copy paste programming, some of us have to sit there and think of a solution and work it out over hours, because its not been done before publically
mise_en_place 4y ago
I was very impressed when it showed me the different techniques for deep reinforcement learning. However, where it struggles is when building an agent. Because you will need a high amount of tokens to template a prompt, in the case of langchain or AutoGPT.
spaceman_2020 4y ago
People also forget that the model is trained on older data. At first, it will default to referencing out of date frameworks and solutions, but if you tell it that its code isn't working, it will usually correct itself.
alexashka 4y ago
You may be underestimating how much meaning people derive from jumping on bandwagons and having a simple to understand group identity. Your suggestion would make many people unhappy. They can't win the competence game and hence 'really investigating' is a losing proposition for them. What they can do is jump on bandwagons very quickly, hoping to score a first mover advantage. How much of an advantage would one get from taking a couple of years to really investigate Bitcoin and the algorithms involved, vs buying some as early as possible and telling everyone else how great it is? :)
joseph_grobbles 4y ago
[dead]

by gdubs 4y ago

rootusrootus 4y ago

I agree it's a good idea to take a moderate approach. The hype that LLMs are going to replace SWEs is clearly just that, hype, if you've done any real work trying to get GPT4 to give you the code you want. But it's also clearly a very useful tool. I think it'll absolutely destroy Stack Overflow.

lcnPylGDnU4H9OF 4y ago
> destroy Stack Overflow It'll be interesting to see how future training data is sourced.
- rootusrootus 4y ago
  Github would be my first guess.
  - lcnPylGDnU4H9OF 4y ago
    That does seem like a likely option. Discussions on issues alongside the actual working (and not working) code.
    - deleted 4y ago
      [deleted]
- svachalek 4y ago
  You simply need the system to train itself on its own interactions, like how search engines improve results by counting clicks.
  - lcnPylGDnU4H9OF 4y ago
    I'm not wondering about how the system will determine what's most helpful but instead determining what's even "correct". A model will learn what's "correct" from Stack Overflow by finding accepted or highly-voted answers but when it can't find such content anymore (in this case because Stack Overflow is hypothetically gone) then what would even exist to generate these discussions to be used as training data? Github, per the sibling comment, is a good example because projects will have issues (tied to the individual repository of source code to be seen as a working implementation of the idea) which will be where such discussions happen.
    - LawTalkingGuy 4y ago
      Those topics that AI replaces the forums for won't need discussion. People won't be confused about that thing because the coding AI knows the details of it. Soon that'll be most syntax questions, soon simple to mid-level algorithms, etc. People will move on to higher-level questions.
    - ok_dad 4y ago
      When Google search became important, people structured their information so that Google could best index it. When AIs become important in the same way, people will start to structure their information so that a particular class of AI can best index it. If that involves API documentation, perhaps there will be a standard format that AIs understand the best.
      - stygiansonic 4y ago
        The difference is that folks had an incentive to make their pages easily indexable: drive page views. With becoming training data the incentive to the creators is a little less clear
z3c0 4y ago
I am very critical of the LLM hype, but the threat to stackoverflow is evident. Like stackoverflow, I never write code verbatim that comes from even GPT4. I frequently find issues in the output, as the code I write is generally very context-specific. However, I find the back-and-forth with interesting tidbits of info dropped here-and-there amounts to something like rubber duck debugging on steroids.
deleted 4y ago
[deleted]
drowsspa 4y ago
Yet, the whole movement of getting blue collar workers to code seems to have lost its steam.
- gumballindie 4y ago
  Probably because “graduating” bootcamps doesnt make one a swe and people figured out it’s a scam?
  - antifa 4y ago
    I'm a fullstack dev of 10+ years experience. We have people on our team who've done almost nothing more than a react bootcamp. They can't do what I do, even with chatGPT, but they can do a lot of things I don't want to do.
  - valray 4y ago
    Also it requires an inherent level of talent which gets turned into skill. Not everyone who takes piano and music harmony lessons can become a jazz composer, especially in 6 weeks or 6 months.
tarruda 4y ago
> The hype that LLMs are going to replace SWEs is clearly just that, hype LLMs cannot replace anyone, but it is clear that engineers which master LLMs usage might multiply their productivity by a lot. The question is: If one LLM assisted engineer can work 10x faster, will companies reduce their engineer staff by 90%?
- majormajor 4y ago
  I've worked at far more companies with miles of product idea backlog we never get to than ones with nothing for engineering to do. Now product will be able to use an LLM to come up with feature proposals and design docs even faster! :o So: are you working at a company where engineering is a cost center or a revenue center? The latter wants to get more done at the same cost much more than it wants to just cut spend.
  - abhibeckert 4y ago
    I'm working at a company where we perpetually don't have enough engineers and we've learned (the hard way) that adding more tends to make the problem worse (too many cooks). Copilot helps a lot with that. I can write `// check if em` and it will finish the comment (email is valid) and write the actual check. It's a massive productivity boost. I'm spending less time looking stuff up (what was that function name again?) and less time fixing typos/basic syntax errors. Copilot doesn't always know the function name, but it does more often than me. And it makes errors too, but again less often than me (also, because I'm reading the code, not typing it, I tend to see the error immediately without needing to test it).
- nuancebydefault 4y ago
  To answer your question with a question if I may -- when did productivity increase in software ever result in headcount reduction? The competition also will have similar productivity gain.
  - MacsHeadroom 4y ago
    >when did productivity increase in software ever result in headcount reduction? The competition also will have similar productivity gain. The average AI company has like 1 employee per $25M valuation. That's around 25x fewer employees than the typical tech company.
peterashford 4y ago
Of course, there's the issue that a lot of the info for useful LLMs probably comes from places like Stack Overflow
chefandy 4y ago
Nobody who professionally designs and writes software AND has used LLM code generation tools sees this as a drop-in replacement for developers, generally, anytime soon. That stance is for overeager, credulous enthusiasts and doomsdayers jumping to conclusions. Similarly, nobody who professionally designs and creates complex art products sees this as a drop-in replacement for commercial artists anytime soon. That stance is for people dazzled by their new image-generation superpower who don't know how little they know about professional creative work. I doubt the markets for utility-grade code work (e.g. customizing existing WordPress themes) or low-effort, high-volume creative assets (template-based logos, lightly customized game sprites) will survive. They're still real people with lives and families and medical bills and mortgages and we really ought to get serious about worker protections in this country. Seriously.
- valray 4y ago
  In a few years, as mid-level cognitive tasks get automated by LLMs, resulting in elimination of some percent of well-paying white-collar jobs, there will be economic dislocation and social disruption. Oligarchic capitalist societies with a hypocritical philosophy of free market economics (such as the USA) will experience social unrest and civil strife. In the meantime, social-democratic societies that have effective governance and can grow their safety net with universal basic income will be advantaged in this new economic order. Thinking Scandinavian and some Asian economies. The geopolitical balance of power will shift toward stable societies that are able to make the conceptual leap to UBI. Others who follow the primitive fantasy of free market economics will crumble and get left behind. At least that is how it looks right now.
  - chefandy 4y ago
    Maybe. I don't see any reason to assume the US won't continue to successfully a) escalate police power to keep citizens under control, b) continue giving people just enough crumbs to keep them from getting violent, and c) propagandize the hell out of the idea that societal mismanagement is a matter of personal responsibility. China has done pretty well while clamping down on civil liberties, and the US has always been better at obfuscating and spinning similar tactics to be more palatable to its citizens. And you only need to look at the rust belt to see what happens when US business leadership decides to follow the cheaper option. Though, regardless of ensuing social unrest, the fact that the people involved are human beings should be enough to not treat them like used condoms when computers figure out how to do their jobs. Should be.
lionkor 4y ago
Sadly I dont think this can happen. There is a load of trash answers on SO, and you bet ChatGPT is trained on that. So you get not only the good of SO, you also get the worst of SO, and theres no way to tell. Just a downgrade for me, plus for most things I do you are better of reading the source code or the documentation (however lackluster) than fumbling with chatgpt and getting an answer that may or may not be right. I might as well ask someone else who doesnt know any programming to search for the answer for me - they wont be able to tell a trustworthy answer from another one. There are so many SO answers (esp. on C++) which look good, but one of the comments points out some edge case in which it breaks. Remember, not everyone does copy paste programming, some of us have to sit there and think of a solution and work it out over hours, because its not been done before publically

mise_en_place 4y ago

I was very impressed when it showed me the different techniques for deep reinforcement learning. However, where it struggles is when building an agent. Because you will need a high amount of tokens to template a prompt, in the case of langchain or AutoGPT.

spaceman_2020 4y ago

People also forget that the model is trained on older data. At first, it will default to referencing out of date frameworks and solutions, but if you tell it that its code isn't working, it will usually correct itself.

alexashka 4y ago

You may be underestimating how much meaning people derive from jumping on bandwagons and having a simple to understand group identity. Your suggestion would make many people unhappy. They can't win the competence game and hence 'really investigating' is a losing proposition for them. What they can do is jump on bandwagons very quickly, hoping to score a first mover advantage. How much of an advantage would one get from taking a couple of years to really investigate Bitcoin and the algorithms involved, vs buying some as early as possible and telling everyone else how great it is? :)

joseph_grobbles 4y ago

[dead]