5 ms·

In my opinion, using AI tools for programming at the moment, unless in a sandboxed environment and on a toy project, is just ludicrous. The amount of shady thin

In my opinion, using AI tools for programming at the moment, unless in a sandboxed environment and on a toy project, is just ludicrous. The amount of shady things going on in this domain (AI trained on stolen content, no proper attribution, not proper way to audit what's going out to third party servers etc.) should be a huge red flag for any professional developer.

neuroelectron 1y ago
The icing on the shit cake is a text editor programmed in typeScript with an impossible to secure plugin architecture.
ThrowawayTestr 1y ago
Companies won't use open source software because of licencing concerns but if you launder it through an LLM it's hunky-dory.
- JTbane 1y ago
  This is kind of not true. Companies will gladly use MIT-style license open source software for on-premises proprietary products, and everything but AGPL-style software for cloud products.
AdieuToLogic 1y ago
> In my opinion, using AI tools for programming at the moment, unless in a sandboxed environment and on a toy project, is just ludicrous. Well put. The fundamental flaw is in trying to employ nondeterministic content generation based on statistical relevance defined by an unknown training data set, which is what commercial LLM offerings are, in an effort to repeatably produce content satisfying a strict mathematical model (program source code).
- mlyle 1y ago
  Nearly as bad: trying to use systems made out of meat, evolved from a unrelated background and trained on an undocumented and chaotic corpus of data, to try and produce content satisfying a strict mathematical model.
  - jdiff 1y ago
    Meaty feet can be held to a fire. To quote IBM, "A computer can never be held accountable."
    - MobiusHorizons 1y ago
      This is the question I keep asking leaders (I literally asked a VP this question once in an all hands). How do we approach the risk associated mistakes made by AI?(process, legal, security, insurance etc) We have process and legal agreements in place to deal with humans that work for a business making mistakes. We need analogs for AI if we want to use it in similar ways.
      - jpcosta 1y ago
        What was the answer? Asking for a vp friend
        AdieuToLogic 1y ago
        >>> Meaty feet can be held to a fire. To quote IBM, "A computer can never be held accountable." >> This is the question I keep asking leaders (I literally asked a VP this question once in an all hands). How do we approach the risk associated mistakes made by AI? > What was the answer? Asking for a vp friend This is a difficult issue to tackle, no doubt. What follows drifts into the philosophical realm by necessity. Software exists to provide value to people. Malicious software qualifies as such due to the desires of the actors which produce same, but will no longer be considered here as this is not germane. AI is an umbrella term for numerous algorithms having wide ranging problem domain applicability and often can approximate near-optimal solutions using significantly less resources than other approaches. But they are still algorithms, capable of only one thing - execute their defined logic. Sometimes this logic can produce results similar to be what a person would in a similar situation. Sometimes the logic will produce wildly different results. Often there is significant value when the logic is used appropriately. In all cases AI algorithms do not possess the concept of understanding. This includes derivatives of understanding such as: - empathy - integrity - morals - right - wrong Which brings us back to part of the first quoted post: To quote IBM, "A computer can never be held accountable." Accountability requires justification of actions taken or lack thereof, which demands the ability to explain why said actions were undertaken relative to other options, and implies a potential consequence be imposed by an authority. Algorithms can partially "justify their output" via strategic logging, but that's about it. Which is why "a computer can never be held accountable." Because it is a machine, executing the instructions ultimately initiated by one or more persons whom can be held accountable.
- ozim 1y ago
  Sounds like we are in the end game for „move fast and break things”. Doesn’t feel like we can invent something that moves even faster and breaks more.
- Finbel 1y ago
  Have you not used commercial LLMs to generate program source code? You describe it as if it's an almost unsolvable problem which might have been reasonable 2 years ago but I just used gpt-5 to generate a complete NextJS application for flashcards. I've literally been employing nondeterministic content generation based on statistical relevance defined by an unknown training data, to repeatably produce content satisfying a strict mathematical model for months now.
  - anticodon 1y ago
    I'm pretty sure there're thousands of blog posts and books describing creation of a complete flashcards application in all popular programming languages and on all popular frameworks.
    - Finbel 1y ago
      There're thousands of blog posts and books describing the vast majority of code most software developers write on a day to day basis. 99.99% of the code in that B2B SaaS for finding the cheapest industrial shipping option isn't novel.
      - AdieuToLogic 1y ago
        > 99.99% of the code in that B2B SaaS for finding the cheapest industrial shipping option isn't novel. That's like saying 99.99% of the food people eat consists of protein, carbohydrates, fats, and/or vegetables and therefore isn't novel. The implication being a McDonald's Big Mac and fries is the same as a spinach salad. The only way someone could believe all food is the same as a Big Mac and fries is if this is all they ate and knew nothing else. Hyperbole never ends well and neither does assuming novelty requires rarity or uniqueness, as distinct combinations of programmatic operations which deliver value in a problem domain is the very definition "new in an interesting way." Just like how Thai noodles have proteins, carbohydrates, fats, and/or vegetables, yet are nothing like a Big Mac and fries.
        Finbel 1y ago
- pmxi 1y ago
  Humans are far less deterministic than LLMs, yet presumably are acceptable for writing program source code?
cut3 1y ago
[flagged]
Sparkyte 1y ago
[flagged]
matt3210 1y ago
Use AI to audit what’s produced by AI. Problem solved! /sarcasm
- bratbag 1y ago
  If this reduces error rates to below those of a human, then that's an acceptable approach. Unless you think humans code reviewing humans is pointless because errors sometimes still slip through?
devjab 1y ago
Unless you turn telemetry off (and believe they respect it) your entire filestructure, error and metadata will be shipped to Microsoft with no audit log available, simply by using VSCode. Which is frankly what copilot is doing here, except it's doing it on your 365 documents. I'm personally less concerned about Microsoft's impact on safety in terms of software development than I am with how all my data is handled by the public sector in Denmark. At this point they shouldn't be allowed to use Windows.
- TheRoque 1y ago
  Sending some telemetry metadata on private servers is vastly different from sending code chunks and in some cases private environnement variables. On top of that, there are already many exploits and failures related to these tools which one again don't compare to simple telemetry. And I'm not even talking about the ethics of reproduced code without proper attribution, which is a different subject.
t1E9mE7JTRjf 1y ago
I imagine when people started using the typewriter, some people writing on paper said similar things. Ultimately 'shady' things are irrelevant. Shady is subjective, and people don't use a technology for lack of related things they might not agree with, let alone what others dislike. They want to get things done, said technology gets thing done.
IlikeKitties 1y ago
And any company that uses it will undercut you on price until you too stop worrying and start using the AI slop machine.
winternewt 1y ago
> The amount of shady things going on in this domain (AI trained on stolen content, no proper attribution, not proper way to audit what's going out to third party servers etc.) should be a huge red flag for any professional developer. If you already have your entire information infrastructure in Office 365 (including all email, Excel sheets with material non-public information etc) I think this point is moot. Why would MS abuse information only from Copilot and not the rest of its products when the legal agreements permit them to do neither?
beefnugs 1y ago
Actually all the illegal and immoral shit is absolutely standard for all the rich bastards of the world... the weird thing is that this is somehow in reach of the plebs now? They somehow don't understand how they are breaking their own business models. We can only assume its a quick spin up cash grab before they jack up prices to unbelievable corp only levels

neuroelectron 1y ago

The icing on the shit cake is a text editor programmed in typeScript with an impossible to secure plugin architecture.

ThrowawayTestr 1y ago

Companies won't use open source software because of licencing concerns but if you launder it through an LLM it's hunky-dory.

JTbane 1y ago
This is kind of not true. Companies will gladly use MIT-style license open source software for on-premises proprietary products, and everything but AGPL-style software for cloud products.

AdieuToLogic 1y ago

> In my opinion, using AI tools for programming at the moment, unless in a sandboxed environment and on a toy project, is just ludicrous. Well put. The fundamental flaw is in trying to employ nondeterministic content generation based on statistical relevance defined by an unknown training data set, which is what commercial LLM offerings are, in an effort to repeatably produce content satisfying a strict mathematical model (program source code).

mlyle 1y ago
Nearly as bad: trying to use systems made out of meat, evolved from a unrelated background and trained on an undocumented and chaotic corpus of data, to try and produce content satisfying a strict mathematical model.
- jdiff 1y ago
  Meaty feet can be held to a fire. To quote IBM, "A computer can never be held accountable."
  - MobiusHorizons 1y ago
    This is the question I keep asking leaders (I literally asked a VP this question once in an all hands). How do we approach the risk associated mistakes made by AI?(process, legal, security, insurance etc) We have process and legal agreements in place to deal with humans that work for a business making mistakes. We need analogs for AI if we want to use it in similar ways.
    - jpcosta 1y ago
      What was the answer? Asking for a vp friend
      - AdieuToLogic 1y ago
        >>> Meaty feet can be held to a fire. To quote IBM, "A computer can never be held accountable." >> This is the question I keep asking leaders (I literally asked a VP this question once in an all hands). How do we approach the risk associated mistakes made by AI? > What was the answer? Asking for a vp friend This is a difficult issue to tackle, no doubt. What follows drifts into the philosophical realm by necessity. Software exists to provide value to people. Malicious software qualifies as such due to the desires of the actors which produce same, but will no longer be considered here as this is not germane. AI is an umbrella term for numerous algorithms having wide ranging problem domain applicability and often can approximate near-optimal solutions using significantly less resources than other approaches. But they are still algorithms, capable of only one thing - execute their defined logic. Sometimes this logic can produce results similar to be what a person would in a similar situation. Sometimes the logic will produce wildly different results. Often there is significant value when the logic is used appropriately. In all cases AI algorithms do not possess the concept of understanding. This includes derivatives of understanding such as: - empathy - integrity - morals - right - wrong Which brings us back to part of the first quoted post: To quote IBM, "A computer can never be held accountable." Accountability requires justification of actions taken or lack thereof, which demands the ability to explain why said actions were undertaken relative to other options, and implies a potential consequence be imposed by an authority. Algorithms can partially "justify their output" via strategic logging, but that's about it. Which is why "a computer can never be held accountable." Because it is a machine, executing the instructions ultimately initiated by one or more persons whom can be held accountable.
ozim 1y ago
Sounds like we are in the end game for „move fast and break things”. Doesn’t feel like we can invent something that moves even faster and breaks more.
Finbel 1y ago
Have you not used commercial LLMs to generate program source code? You describe it as if it's an almost unsolvable problem which might have been reasonable 2 years ago but I just used gpt-5 to generate a complete NextJS application for flashcards. I've literally been employing nondeterministic content generation based on statistical relevance defined by an unknown training data, to repeatably produce content satisfying a strict mathematical model for months now.
- anticodon 1y ago
  I'm pretty sure there're thousands of blog posts and books describing creation of a complete flashcards application in all popular programming languages and on all popular frameworks.
  - Finbel 1y ago
    There're thousands of blog posts and books describing the vast majority of code most software developers write on a day to day basis. 99.99% of the code in that B2B SaaS for finding the cheapest industrial shipping option isn't novel.
    - AdieuToLogic 1y ago
      > 99.99% of the code in that B2B SaaS for finding the cheapest industrial shipping option isn't novel. That's like saying 99.99% of the food people eat consists of protein, carbohydrates, fats, and/or vegetables and therefore isn't novel. The implication being a McDonald's Big Mac and fries is the same as a spinach salad. The only way someone could believe all food is the same as a Big Mac and fries is if this is all they ate and knew nothing else. Hyperbole never ends well and neither does assuming novelty requires rarity or uniqueness, as distinct combinations of programmatic operations which deliver value in a problem domain is the very definition "new in an interesting way." Just like how Thai noodles have proteins, carbohydrates, fats, and/or vegetables, yet are nothing like a Big Mac and fries.
      - Finbel 1y ago
pmxi 1y ago
Humans are far less deterministic than LLMs, yet presumably are acceptable for writing program source code?

cut3 1y ago

[flagged]

Sparkyte 1y ago

[flagged]

matt3210 1y ago

Use AI to audit what’s produced by AI. Problem solved! /sarcasm

bratbag 1y ago
If this reduces error rates to below those of a human, then that's an acceptable approach. Unless you think humans code reviewing humans is pointless because errors sometimes still slip through?

devjab 1y ago

Unless you turn telemetry off (and believe they respect it) your entire filestructure, error and metadata will be shipped to Microsoft with no audit log available, simply by using VSCode. Which is frankly what copilot is doing here, except it's doing it on your 365 documents. I'm personally less concerned about Microsoft's impact on safety in terms of software development than I am with how all my data is handled by the public sector in Denmark. At this point they shouldn't be allowed to use Windows.

TheRoque 1y ago
Sending some telemetry metadata on private servers is vastly different from sending code chunks and in some cases private environnement variables. On top of that, there are already many exploits and failures related to these tools which one again don't compare to simple telemetry. And I'm not even talking about the ethics of reproduced code without proper attribution, which is a different subject.

t1E9mE7JTRjf 1y ago

I imagine when people started using the typewriter, some people writing on paper said similar things. Ultimately 'shady' things are irrelevant. Shady is subjective, and people don't use a technology for lack of related things they might not agree with, let alone what others dislike. They want to get things done, said technology gets thing done.

IlikeKitties 1y ago

And any company that uses it will undercut you on price until you too stop worrying and start using the AI slop machine.

winternewt 1y ago

> The amount of shady things going on in this domain (AI trained on stolen content, no proper attribution, not proper way to audit what's going out to third party servers etc.) should be a huge red flag for any professional developer. If you already have your entire information infrastructure in Office 365 (including all email, Excel sheets with material non-public information etc) I think this point is moot. Why would MS abuse information only from Copilot and not the rest of its products when the legal agreements permit them to do neither?

beefnugs 1y ago

Actually all the illegal and immoral shit is absolutely standard for all the rich bastards of the world... the weird thing is that this is somehow in reach of the plebs now? They somehow don't understand how they are breaking their own business models. We can only assume its a quick spin up cash grab before they jack up prices to unbelievable corp only levels