4 ms·

Interesting read -- and a correct take, given the software development perspective. In that context, LLM-based AI is faulty, unpredictable, and unmanageable, an

Interesting read -- and a correct take, given the software development perspective. In that context, LLM-based AI is faulty, unpredictable, and unmanageable, and not ready for mission-critical applications. If you want to argue otherwise, do a quick thought experiment first: would you let an LLM manage your financial affairs (entirely, unsupervised)? Would you let it perform your job while you receive the rewards and consequences? Would you be comfortable to give it full control of your smart home? There are different sets of expectations put on human actors vs autonomous systems. We expect people to be fallible and wrong some of the time, even if the individuals in question can't/won't admit it. With a software-based system, the expectations are that it will be robust, tested, and performing correctly 100% of the time, and when a fault occurs, it will be clear, marked with yellow tape and flashing lights. LLM-based AIs are sort of insidious in that they straddle this expectation gap: the emergent behaviour is erratic, projecting confident omniscience, while often hallucinating and plain wrong. However vague, the catch-all term "AI" still implies "computer system" and by extension "engineered and tested".

cesaref 2y ago
It's a bad example. Lots of finance firms use AI to manage their financial affairs - go and investigate what is currently considered state of the art for trading algorithms. Now if you substituted something safety critical instead, say, running a nuclear power station, or my favourite currently in use example, self driving cars, then yes, you should be scared.
- galleywest200 2y ago
  > go and investigate what is currently considered state of the art for trading algorithms. These are not LLMs but algorithms written and designed by human minds. It is unfortunate that AI has become a catch-all word for any kind of machine learning.
  - varelse 2y ago
    [dead]
  - chuckadams 2y ago
    LLMs are algorithms written by humans. "AI" is supposed to be a vague term, and not synonymous with one particular implementation.
    - teucris 2y ago
      LLMs are architectures written by humans. What an LLM creates is not algorithmic.
      - FrustratedMonky 2y ago
        "What an LLM creates is not algorithmic." Not strictly true. There are patterns in the weights that could be steps in an algorithm.
        seadan83 2y ago
        LLMs create models, not algorithms. An algorithm is a rote sequence if steps to accomplish a task. The following is an algorithm: - plug in input to model - say yes if result is positive, else say no LLMs use models, the model is not an algorithm. > There are patterns in the weights that could be steps in an algorithm. Sure, but yeah... no.. "Could be steps in an algorithm" does not constitute an algorithm. Weights are inputs, they are not themselves parts of an algorithm. The algorithm might still try to come up with weights. Still, don't confuse procedure from data.
        FrustratedMonky 2y ago
        Don't want to get to pedantic on that response. The model can contain complex information. There is already evidence it can form a model of the world. So why not something like steps to get from A to B. And, it is clear that LLMs can follow steps. One didn't place in the Math Olympiad without some ability to follow steps. https://research.google/blog/teaching-language-models-to-reason-algorithmically/ https://research.google/blog/teaching-language-models-to-rea... And, Anyway, when I asked it, it said it could "Yes, an LLM model can contain the steps of an algorithm, especially when prompted to "think step-by-step" or use a "chain-of-thought" approach, which allows it to break down a complex problem into smaller, more manageable steps and generate a solution by outlining each stage of the process in a logical sequence; essentially mimicking how a human would approach an algorithm. "
- wavemode 2y ago
  He said LLM, not just any AI
  - aleph_minus_one 2y ago
    I would bet that there do exist some finance firms that do use LLM as AIs for the purposes that cesaref sketched.
    - wavemode 2y ago
      Then read btbuildem's comment more carefully. Because you are still misunderstanding it, or ignoring parts of it, if you are willing to bet this.
- ryukoposting 2y ago
  Makes me wonder how they detect market manipulation and fraud. Trivial activities, like marking the close, probably aren't hard to detect, but I imagine that some kind of ML thingy is involved in flagging accounts for manual inspection.
msabalau 2y ago
Pragmatically, "AI" will mean (and for, many people already does mean) stochastic and fallible. If your users are likely to be AI illiterate and mistakenly feel that an AI app is reliable and suitable for mission critical applications when it isn't, that is a risk you mitigate. But it seems deeply unserious of the author to just assert that mission-critical software is the only "serious context" and the only thing matters, and therefore AI is dead end. "Serious, mission critical" apps are just going to be a niche in the future.
- timeon 2y ago
  > "Serious, mission critical" apps are just going to be a niche in the future. High quality only for niche "serious, mission critical" everywhere else, enshittification already started, LLMs will just accelerate it.
JumpCrisscross 2y ago
> LLM-based AI is faulty, unpredictable, and unmanageable Is there a fundamental (à La Gödel) reason why we can’t predict or manage LLMs? > would you let an LLM manage your financial affairs (entirely, unsupervised)? No. But history is littered with eccentric geniuses who couldn’t be trusted on their own, but who nevertheless were relied on by decision makers. Maybe there is an Erdős principle at play: AI can be involved in questions of arbitrary complexity, but can only advise on important decisions.
- rob_c 2y ago
  [flagged]
  - bee_rider 2y ago
    >> Is there a fundamental (à La Gödel) reason why we can’t predict or manage LLMs? > We're building statistical models to make statistical predictions after training with a sufficiently large and statistically diverse dataset. Aka it's random. This seems like a sort of hand wavy answer to what seems like a pretty deep and technical question. And to be fair, this isn’t the environment to ask that question, it seems like something a bunch of researchers would work out. Of course we build statistical models all the time and then use them to make pretty good predictions. Is there something actually fundamental about these LL modes that makes them… unmanageable? Well, we’ll have to define manageable first… etc etc.
    - rob_c 2y ago
      So the question is so abstract as to not have a handle on the answer... Great, What is M theory? This also is missing a structure to allow us to unify existing fundamental theories. The only way to assess this is to stop treating the models as statistical beasts which simply only work with "enough" statistics and start talking about them from the direction of information theory. The answer is glib because the problem is the field requires a mix between stats, computing, and mathematics. 2 of these fields have their own languages for the problem (which differ, but exist) and computing has come along and made another set of names for the same complex things making the whole thing (for now) a mess... Especially with the main practicioners stuck in the view that big llm are simply money printers.
manquer 2y ago
> would you let an LLM manage your financial affairs (entirely, unsupervised)? It will likely be better[2] not because AI is good at this . It would be because study after study[1] has shown that active management performs poorer than passive funds, less intervention gives better result over longer timeframe . [1] the famous warren buffet bet comes to mind . There are more formal ones validating this . [2] if configured to do minimal changes
- seadan83 2y ago
  What if financial affairs were broadened to be everything, not just portfolio management? Eg: paying bills, credit cards, cash balance in check vs savings vs brokerage.
  - manquer 2y ago
    Good financial management(portfolio and personal) is a matter of disciplined routine, performed consistently over long timeframe, combined with impulse control. It is not complicated at all, any program (LLM or just a rules engine) will always do far better than we can because it will not suffer either problem(sticking to the routine or impulse). Most humans make very bad decisions around personal finance, whether it is big things like gambling or impulse buys with expensive credit, to smaller items like tracking subscriptions or keeping not needed money in checking account etc. This is irrespective of financial literacy, education, wealth or professions like say working in finance/ personal wealth management even. Entire industries like lottery, gambling, luxury goods, gaming, credit card APRs, Buy Now Pay Later, Consumer SaaS, Banking overdraft fees are all built around our inability to control our impulses or follow disciplined routines. This is why trust funds with wealth management professionals are the only way to generational wealth. You need the ability to control any benefactor (the next generations) from excising their impulses on amounts beyond their annual draw. Plus the disciplined routine of a professional team who are paid to do only this with multiple layers that vet the impulses of individual managers and conservative mandate to keep them risk averse and therefore less impulsive. If an program can do it for me (provided of course I irrevocably give away my control to override or alter its decisions) then normal people can also benefit without the high net worth required for wealth management.
clint 2y ago
The primary fallacy in your argument is that you seem to think that humans produce much better products on some kind of metric. My lived experience the software industry at almost all levels over the last 25 years leads me to believe that the vast majority of humans and teams of humans produce atrocious code that only wastes time, money, and people's patience. Often because it is humans producing the code, other humans are not willing to fully engage, criticize and improve that code, deferring to just passing it on to the next person, team, generation, whatever. Yes, this perhaps happens better in some (very large and very small) organizations, but most often it only happens with the inclusions of horrendous layers of protocol, bureaucracy, more time, more emotional exhaustion, etc. In other words a very costly process to produce excellent code, both in real capital and human capital. It literally burns through actual humans and results in very bad health outcomes for most people in the industry, ranging from minor stuff to really major things. The reality is that probably 80% of people working in the tech industry can be outperformed by an AI and at a fraction of the cost. AIs can be tuned, guided, and steered to produce code that I would call exception compared even to most developers who have been in the field for 5years or more. You probably come to this fallacy because you have worked in one of these very small or very large companies that takes producing code seriously and believe that your experience represents the vast majority of the industry, but in fact the middle area is where most code is being "produced" and if you've never been fully engaged in those situations, you may literally have no idea of the crap that's being produced and shipped on a daily basis. These companies have no incentive to change, they make lots of money doing this, and fresh meat (humans) is relatively easy to come by. Most of these AI benchmarks are trying to get these LLMs to produce outputs at the scale and quantity of one of these exceptional organizations when in fact, the real benefits will come in the bulk of organizations that cannot do this stuff and AI will produce as good or better code than a team of mediocre developers slogging away in a mediocre, but profitable, company. Yes there are higher levels of abstraction around code, and getting it deployed, comprehensive testing, triaging issues, QA blah blah, that humans are going to be better at for now, but I see many of those issues being addressed by some kind of LLM system sooner or later. Finally, I think most of the friction people are seeing right now in their organization is because of the wildly ad hoc way people and organizations are using AI, not so much about the technological abilities of the models themselves.
superjan 2y ago
With respect to hallucinating, I never read about training LLM’s to say: “I don’t know” when they don’t know. Is that even researched?
- Sohcahtoa82 2y ago
  ChatGPT seems to be good about this. If you invent something and ask about it, like "What was the No More Clowning Act of 2025?", it will say it can't find any information on it. The older or smaller models, like anything you can run locally, are probably far more likely to just invent some bullshit. That said, I've certainly asked ChatGPT about things that definitely have a correct answer and had it give me incorrect information. When talking about hallucinating, I do think we need to differentiate between "what you asked about exists and has a correct answer, but the AI got it wrong" and "What you're asking for does not exist or does not have an answer, but the AI just generated some bullshit".
- sroussey 2y ago
  Not sure why you are downvoted. It’s a difficult problem, but lots of angles on how to deal with it. For example: https://arxiv.org/abs/2412.15176 https://arxiv.org/abs/2412.15176
rsanek 2y ago
> Would you let it perform your job while you receive the rewards and consequences? isn't this what being a human manager is? not sure why you're saying it must be entirely + unsupervised. at my job, my boss mostly trusts me but still checks my work and gives me feedback when he wants something changed. he's ultimately responsible for what I do.
- deleted 2y ago
  [deleted]
- bredren 2y ago
  Indeed, even if you have a professional accountant do your taxes, you must still sign off on their work. Detecting omissions or errors on prepared tax forms often requires knowledge of context missed by or not provided to the accountant.
PaulDavisThe1st 2y ago
I believe you're asking the wrong question, or at least you're asking it in the wrong way. From my POV, it comes in two parts: 1. Do you believe that LLMs operate in a similar way to the important parts of human cognition? 2. If not, do you believe that they operate in a way that makes them useful for tasks other than responding to text prompts, and if so, what are those tasks? If you believe that the answer to Q1 is substantively "yes" - that is, humans and LLM are engaged in the same sort of computational behavior when we engage in speech generation - then there's presumably no particular impediment to using an LLM where you might otherwise use a human (and with the same caveats). My own answer is that while some human speech behavior is possibly generated by systems that function in a semantically equivalent way to current LLMs, human cognition is capable of tasks that LLMs cannot perform de novo even if they can give the illusion of doing so (primarily causal chain reasoning). Consequently, LLMs are not in any real sense equivalent to a human being, and using them as such is a mistake.
- User23 2y ago
  I think C.S. Peirce's distinction between corollarial reasoning and theorematic reasoning[1][2] is helpful here. In short, the former is the grindy rule following sort of reasoning, and the latter is the kind of reasoning that's associated with new insights that are not determined by the premises alone. As an aside, Students of Peirce over the years have quite the pedigree in data science too, including the genius Edgar F. Codd, who invented the relational database largely inspired by Peirce's approach to relations. Anyhow, computers are already quite good at corollarial reasoning and have been for some time, even before LLMs. On the other hand, they struggle with theorematic reasoning. Last I knew, the absolute state of the art performs about as well as a smart high school student. And even there, the tests are synthetic, so how theorematic they truly are is questionable. I wouldn't rule out the possibility of some automaton proposing a better explanation for gravitational anomalies than dark matter for example, but so far as I know nothing like that is being done yet. There's also the interesting question of whether or not an LLM that produces a sequence of tokens that induces a genuine insight in the human reader actually means the LLM itself had said insight. [1] https://www.cspeirce.com/menu/library/bycsp/l75/ver1/l75v1-02.htm https://www.cspeirce.com/menu/library/bycsp/l75/ver1/l75v1-0... [2] https://groups.google.com/g/cybcom/c/Es8Bh0U2Vcg https://groups.google.com/g/cybcom/c/Es8Bh0U2Vcg
qaq 2y ago
Would you let an LLM manage your financial affairs (entirely, unsupervised)? Hmm I would not let other human manage my financial affairs entirely unsupervised.
tossandthrow 2y ago
> would you let an LLM manage your financial affairs (entirely, unsupervised)? No, but I also would not let another person do that. It is telling that you needed to interject "entirely, unsupervised". Most people will let an llm do it partially, and probably already do.
- BhavdeepSethi 2y ago
  People pay to use a financial advisor. Isn't that another person?
  - tossandthrow 2y ago
    The key is: entirely and unsupervised. Mostly your financial advisor writes your return you sign off on or manages your portfolio. But the advisor usually solicits and interacts with you to know what your financial goals are and ensure you are on board with the consequences of their advice. I do not dismiss that some people are completely hands off at great risk IMHO. But these are not me - as was my initial proposition.
wyager 2y ago
> would you let an LLM manage your financial affairs (entirely, unsupervised)? I wouldn't let another human do this.
Earw0rm 2y ago
A more informative question is: _Who_ would you let manage your financial affairs, and under what circumstances? To which my answer would be something like: a qualified financial adviser with a good track record, who can be trusted to do the job to, if not the best of their abilities, at least an acceptable level of professional competence. A related question: who would you let give you a lift someplace in a car? And here's where things get interesting. Because on the one hand there's a LOT more at stake (literally, your life), and yet various social norms, conventions , economic pressures and so on mean that in practice we quite often entrust that responsibility to people who are very, very far from performing at their best. So while a financial adviser AI is useless unless it can perform at the level of a trained professional doing their job (or unless it can perform at maybe 95% of that level at much lower cost), a self-driving car is at least _potentially_ useful if it's only somewhat better than people at or close to their worst. As a high proportion of road traffic collisions are caused by people who are drunk, tired, emotionally unstable or otherwise very very far from the peak performance of a human being operating a car. (We can argue that a system which routinely requires people to carry out life-or-death, mission-critical tasks while significantly impaired is dangerously flawed and needs a major overhaul, but that's a slightly different debate).

cesaref 2y ago

It's a bad example. Lots of finance firms use AI to manage their financial affairs - go and investigate what is currently considered state of the art for trading algorithms. Now if you substituted something safety critical instead, say, running a nuclear power station, or my favourite currently in use example, self driving cars, then yes, you should be scared.

galleywest200 2y ago
> go and investigate what is currently considered state of the art for trading algorithms. These are not LLMs but algorithms written and designed by human minds. It is unfortunate that AI has become a catch-all word for any kind of machine learning.
- varelse 2y ago
  [dead]
- chuckadams 2y ago
  LLMs are algorithms written by humans. "AI" is supposed to be a vague term, and not synonymous with one particular implementation.
  - teucris 2y ago
    LLMs are architectures written by humans. What an LLM creates is not algorithmic.
    - FrustratedMonky 2y ago
      "What an LLM creates is not algorithmic." Not strictly true. There are patterns in the weights that could be steps in an algorithm.
      - seadan83 2y ago
        LLMs create models, not algorithms. An algorithm is a rote sequence if steps to accomplish a task. The following is an algorithm: - plug in input to model - say yes if result is positive, else say no LLMs use models, the model is not an algorithm. > There are patterns in the weights that could be steps in an algorithm. Sure, but yeah... no.. "Could be steps in an algorithm" does not constitute an algorithm. Weights are inputs, they are not themselves parts of an algorithm. The algorithm might still try to come up with weights. Still, don't confuse procedure from data.
        FrustratedMonky 2y ago
        Don't want to get to pedantic on that response. The model can contain complex information. There is already evidence it can form a model of the world. So why not something like steps to get from A to B. And, it is clear that LLMs can follow steps. One didn't place in the Math Olympiad without some ability to follow steps. https://research.google/blog/teaching-language-models-to-reason-algorithmically/ https://research.google/blog/teaching-language-models-to-rea... And, Anyway, when I asked it, it said it could "Yes, an LLM model can contain the steps of an algorithm, especially when prompted to "think step-by-step" or use a "chain-of-thought" approach, which allows it to break down a complex problem into smaller, more manageable steps and generate a solution by outlining each stage of the process in a logical sequence; essentially mimicking how a human would approach an algorithm. "
wavemode 2y ago
He said LLM, not just any AI
- aleph_minus_one 2y ago
  I would bet that there do exist some finance firms that do use LLM as AIs for the purposes that cesaref sketched.
  - wavemode 2y ago
    Then read btbuildem's comment more carefully. Because you are still misunderstanding it, or ignoring parts of it, if you are willing to bet this.
ryukoposting 2y ago
Makes me wonder how they detect market manipulation and fraud. Trivial activities, like marking the close, probably aren't hard to detect, but I imagine that some kind of ML thingy is involved in flagging accounts for manual inspection.

msabalau 2y ago

Pragmatically, "AI" will mean (and for, many people already does mean) stochastic and fallible. If your users are likely to be AI illiterate and mistakenly feel that an AI app is reliable and suitable for mission critical applications when it isn't, that is a risk you mitigate. But it seems deeply unserious of the author to just assert that mission-critical software is the only "serious context" and the only thing matters, and therefore AI is dead end. "Serious, mission critical" apps are just going to be a niche in the future.

timeon 2y ago
> "Serious, mission critical" apps are just going to be a niche in the future. High quality only for niche "serious, mission critical" everywhere else, enshittification already started, LLMs will just accelerate it.

JumpCrisscross 2y ago

> LLM-based AI is faulty, unpredictable, and unmanageable Is there a fundamental (à La Gödel) reason why we can’t predict or manage LLMs? > would you let an LLM manage your financial affairs (entirely, unsupervised)? No. But history is littered with eccentric geniuses who couldn’t be trusted on their own, but who nevertheless were relied on by decision makers. Maybe there is an Erdős principle at play: AI can be involved in questions of arbitrary complexity, but can only advise on important decisions.

rob_c 2y ago
[flagged]
- bee_rider 2y ago
  >> Is there a fundamental (à La Gödel) reason why we can’t predict or manage LLMs? > We're building statistical models to make statistical predictions after training with a sufficiently large and statistically diverse dataset. Aka it's random. This seems like a sort of hand wavy answer to what seems like a pretty deep and technical question. And to be fair, this isn’t the environment to ask that question, it seems like something a bunch of researchers would work out. Of course we build statistical models all the time and then use them to make pretty good predictions. Is there something actually fundamental about these LL modes that makes them… unmanageable? Well, we’ll have to define manageable first… etc etc.
  - rob_c 2y ago
    So the question is so abstract as to not have a handle on the answer... Great, What is M theory? This also is missing a structure to allow us to unify existing fundamental theories. The only way to assess this is to stop treating the models as statistical beasts which simply only work with "enough" statistics and start talking about them from the direction of information theory. The answer is glib because the problem is the field requires a mix between stats, computing, and mathematics. 2 of these fields have their own languages for the problem (which differ, but exist) and computing has come along and made another set of names for the same complex things making the whole thing (for now) a mess... Especially with the main practicioners stuck in the view that big llm are simply money printers.

manquer 2y ago

> would you let an LLM manage your financial affairs (entirely, unsupervised)? It will likely be better[2] not because AI is good at this . It would be because study after study[1] has shown that active management performs poorer than passive funds, less intervention gives better result over longer timeframe . [1] the famous warren buffet bet comes to mind . There are more formal ones validating this . [2] if configured to do minimal changes

seadan83 2y ago
What if financial affairs were broadened to be everything, not just portfolio management? Eg: paying bills, credit cards, cash balance in check vs savings vs brokerage.
- manquer 2y ago
  Good financial management(portfolio and personal) is a matter of disciplined routine, performed consistently over long timeframe, combined with impulse control. It is not complicated at all, any program (LLM or just a rules engine) will always do far better than we can because it will not suffer either problem(sticking to the routine or impulse). Most humans make very bad decisions around personal finance, whether it is big things like gambling or impulse buys with expensive credit, to smaller items like tracking subscriptions or keeping not needed money in checking account etc. This is irrespective of financial literacy, education, wealth or professions like say working in finance/ personal wealth management even. Entire industries like lottery, gambling, luxury goods, gaming, credit card APRs, Buy Now Pay Later, Consumer SaaS, Banking overdraft fees are all built around our inability to control our impulses or follow disciplined routines. This is why trust funds with wealth management professionals are the only way to generational wealth. You need the ability to control any benefactor (the next generations) from excising their impulses on amounts beyond their annual draw. Plus the disciplined routine of a professional team who are paid to do only this with multiple layers that vet the impulses of individual managers and conservative mandate to keep them risk averse and therefore less impulsive. If an program can do it for me (provided of course I irrevocably give away my control to override or alter its decisions) then normal people can also benefit without the high net worth required for wealth management.

clint 2y ago

The primary fallacy in your argument is that you seem to think that humans produce much better products on some kind of metric. My lived experience the software industry at almost all levels over the last 25 years leads me to believe that the vast majority of humans and teams of humans produce atrocious code that only wastes time, money, and people's patience. Often because it is humans producing the code, other humans are not willing to fully engage, criticize and improve that code, deferring to just passing it on to the next person, team, generation, whatever. Yes, this perhaps happens better in some (very large and very small) organizations, but most often it only happens with the inclusions of horrendous layers of protocol, bureaucracy, more time, more emotional exhaustion, etc. In other words a very costly process to produce excellent code, both in real capital and human capital. It literally burns through actual humans and results in very bad health outcomes for most people in the industry, ranging from minor stuff to really major things. The reality is that probably 80% of people working in the tech industry can be outperformed by an AI and at a fraction of the cost. AIs can be tuned, guided, and steered to produce code that I would call exception compared even to most developers who have been in the field for 5years or more. You probably come to this fallacy because you have worked in one of these very small or very large companies that takes producing code seriously and believe that your experience represents the vast majority of the industry, but in fact the middle area is where most code is being "produced" and if you've never been fully engaged in those situations, you may literally have no idea of the crap that's being produced and shipped on a daily basis. These companies have no incentive to change, they make lots of money doing this, and fresh meat (humans) is relatively easy to come by. Most of these AI benchmarks are trying to get these LLMs to produce outputs at the scale and quantity of one of these exceptional organizations when in fact, the real benefits will come in the bulk of organizations that cannot do this stuff and AI will produce as good or better code than a team of mediocre developers slogging away in a mediocre, but profitable, company. Yes there are higher levels of abstraction around code, and getting it deployed, comprehensive testing, triaging issues, QA blah blah, that humans are going to be better at for now, but I see many of those issues being addressed by some kind of LLM system sooner or later. Finally, I think most of the friction people are seeing right now in their organization is because of the wildly ad hoc way people and organizations are using AI, not so much about the technological abilities of the models themselves.

superjan 2y ago

With respect to hallucinating, I never read about training LLM’s to say: “I don’t know” when they don’t know. Is that even researched?

Sohcahtoa82 2y ago
ChatGPT seems to be good about this. If you invent something and ask about it, like "What was the No More Clowning Act of 2025?", it will say it can't find any information on it. The older or smaller models, like anything you can run locally, are probably far more likely to just invent some bullshit. That said, I've certainly asked ChatGPT about things that definitely have a correct answer and had it give me incorrect information. When talking about hallucinating, I do think we need to differentiate between "what you asked about exists and has a correct answer, but the AI got it wrong" and "What you're asking for does not exist or does not have an answer, but the AI just generated some bullshit".
sroussey 2y ago
Not sure why you are downvoted. It’s a difficult problem, but lots of angles on how to deal with it. For example: https://arxiv.org/abs/2412.15176 https://arxiv.org/abs/2412.15176

rsanek 2y ago

> Would you let it perform your job while you receive the rewards and consequences? isn't this what being a human manager is? not sure why you're saying it must be entirely + unsupervised. at my job, my boss mostly trusts me but still checks my work and gives me feedback when he wants something changed. he's ultimately responsible for what I do.

deleted 2y ago
[deleted]
bredren 2y ago
Indeed, even if you have a professional accountant do your taxes, you must still sign off on their work. Detecting omissions or errors on prepared tax forms often requires knowledge of context missed by or not provided to the accountant.

PaulDavisThe1st 2y ago

I believe you're asking the wrong question, or at least you're asking it in the wrong way. From my POV, it comes in two parts: 1. Do you believe that LLMs operate in a similar way to the important parts of human cognition? 2. If not, do you believe that they operate in a way that makes them useful for tasks other than responding to text prompts, and if so, what are those tasks? If you believe that the answer to Q1 is substantively "yes" - that is, humans and LLM are engaged in the same sort of computational behavior when we engage in speech generation - then there's presumably no particular impediment to using an LLM where you might otherwise use a human (and with the same caveats). My own answer is that while some human speech behavior is possibly generated by systems that function in a semantically equivalent way to current LLMs, human cognition is capable of tasks that LLMs cannot perform de novo even if they can give the illusion of doing so (primarily causal chain reasoning). Consequently, LLMs are not in any real sense equivalent to a human being, and using them as such is a mistake.

User23 2y ago
I think C.S. Peirce's distinction between corollarial reasoning and theorematic reasoning[1][2] is helpful here. In short, the former is the grindy rule following sort of reasoning, and the latter is the kind of reasoning that's associated with new insights that are not determined by the premises alone. As an aside, Students of Peirce over the years have quite the pedigree in data science too, including the genius Edgar F. Codd, who invented the relational database largely inspired by Peirce's approach to relations. Anyhow, computers are already quite good at corollarial reasoning and have been for some time, even before LLMs. On the other hand, they struggle with theorematic reasoning. Last I knew, the absolute state of the art performs about as well as a smart high school student. And even there, the tests are synthetic, so how theorematic they truly are is questionable. I wouldn't rule out the possibility of some automaton proposing a better explanation for gravitational anomalies than dark matter for example, but so far as I know nothing like that is being done yet. There's also the interesting question of whether or not an LLM that produces a sequence of tokens that induces a genuine insight in the human reader actually means the LLM itself had said insight. [1] https://www.cspeirce.com/menu/library/bycsp/l75/ver1/l75v1-02.htm https://www.cspeirce.com/menu/library/bycsp/l75/ver1/l75v1-0... [2] https://groups.google.com/g/cybcom/c/Es8Bh0U2Vcg https://groups.google.com/g/cybcom/c/Es8Bh0U2Vcg

qaq 2y ago

Would you let an LLM manage your financial affairs (entirely, unsupervised)? Hmm I would not let other human manage my financial affairs entirely unsupervised.

tossandthrow 2y ago

> would you let an LLM manage your financial affairs (entirely, unsupervised)? No, but I also would not let another person do that. It is telling that you needed to interject "entirely, unsupervised". Most people will let an llm do it partially, and probably already do.

BhavdeepSethi 2y ago
People pay to use a financial advisor. Isn't that another person?
- tossandthrow 2y ago
  The key is: entirely and unsupervised. Mostly your financial advisor writes your return you sign off on or manages your portfolio. But the advisor usually solicits and interacts with you to know what your financial goals are and ensure you are on board with the consequences of their advice. I do not dismiss that some people are completely hands off at great risk IMHO. But these are not me - as was my initial proposition.

wyager 2y ago

> would you let an LLM manage your financial affairs (entirely, unsupervised)? I wouldn't let another human do this.

Earw0rm 2y ago

A more informative question is: _Who_ would you let manage your financial affairs, and under what circumstances? To which my answer would be something like: a qualified financial adviser with a good track record, who can be trusted to do the job to, if not the best of their abilities, at least an acceptable level of professional competence. A related question: who would you let give you a lift someplace in a car? And here's where things get interesting. Because on the one hand there's a LOT more at stake (literally, your life), and yet various social norms, conventions , economic pressures and so on mean that in practice we quite often entrust that responsibility to people who are very, very far from performing at their best. So while a financial adviser AI is useless unless it can perform at the level of a trained professional doing their job (or unless it can perform at maybe 95% of that level at much lower cost), a self-driving car is at least _potentially_ useful if it's only somewhat better than people at or close to their worst. As a high proportion of road traffic collisions are caused by people who are drunk, tired, emotionally unstable or otherwise very very far from the peak performance of a human being operating a car. (We can argue that a system which routinely requires people to carry out life-or-death, mission-critical tasks while significantly impaired is dangerously flawed and needs a major overhaul, but that's a slightly different debate).