6 ms·

I hereby dub this "Baron-von-Munchausen-as-a-Service." Now you can pay real money for a chatbot to make stuff up about your company and its products. While it

I hereby dub this "Baron-von-Munchausen-as-a-Service." Now you can pay real money for a chatbot to make stuff up about your company and its products. While it is pretty incredible stuff, until or unless they have a veracity bit — some sort of "please don't lie" flag in it, I'd be wary of what it produces. Will ChatGPT offer off-the-cuff pricing, discounts, rebates and refunds that are in line with your actual business model? Will it make written offers you will be legally-obligated to adhere to? Will it lie about your features and capabilities, or that of your competitors? Will it invent whole-cloth SLAs and KPIs you'll never be able to adhere to? Will it make up arrival and departure times? Medical conditions and prescribed treatments/therapies? Will it glibly give instructions that can get consumers/users who read these statements fired from their jobs, fined, imprisoned or killed? There are places where you only want to give a generative AI so much room to color outside the lines. And after that, you are going to want traditional procedural logic to take over. Decision-trees from manually-maintained systems and even multi-decadal-honed expert systems. There are some things where letting a computer make up a plausible response is ... okay-ish. And there are times where you can cause immense amounts of damage by letting things get outside of your control and human review. This should come with a consumer warning label that would put your typical pharmaceutical safety disclosure to shame. Now, with all that said, I am amazed at many of the results these LLM-based systems can generate. My main concern is that they are getting so good — or at least so plausible — that they can even fool domain experts unless they read really, really closely. And the experts won't be able to do that at scales of 10s or 100s of thousands of transactions per second.

lijok 3y ago
If you’re not reviewing ChatGPT output before utilizing it, you’re doing it wrong and will end up in hot water
- PeterCorless 3y ago
  100%. But people will do it wrong. A lot.
  - lijok 3y ago
    Should get better over time as the tool gets better incorporated into the industry. People always misuse new tools
    - biophysboy 3y ago
      But why will it get better at accuracy if it isnt trained to be accurate?
      - lijok 3y ago
        What I mean is, the situation of people misusing it should get better over time
        PeterCorless 3y ago
        Fair. Yes, people will eventually develop best practices through trial & error, and through better features/capabilities of the product/service itself. I just remain extremely cautious and pragmatic during this part of the hype cycle and zone of inflated expectations.
    - PeterCorless 3y ago
      In such cases, I believe there are (and should be) guardrails. It already has some guardrails internally set: ChatGPT already knows how to say: "I am not a lawyer, but I can provide you with some general information on this topic." The same thing for medicine: "I'm not a medical professional, but I can provide some general information..." But, for instance, ask it to design a canard for a 4th generation supersonic fighter, and suddenly it spits out pages of output. [Though it finally corrects itself and balks to answer when you ask for angle of forward sweep and a formula for the curvature of the surface.] There needs to be a way to "sniff out" if there's topics that people are getting too close to danger zones for it to answer. Ways for organizations themselves to set those 'bounding boxes.'
      - earthboundkid 3y ago
tsunamifury 3y ago
This sounds a bit like word salad, and your argument is worse than an LLM output. You even contradict yourself at the end. Also I think you need to better understand how much hallucination has been driven down recently and its clear path going forward.
- PeterCorless 3y ago
  “This isn’t fixable,” said Emily Bender, a linguistics professor and director of the University of Washington’s Computational Linguistics Laboratory. “It’s inherent in the mismatch between the technology and the proposed use cases.” https://apnews.com/article/artificial-intelligence-hallucination-chatbots-chatgpt-falsehoods-ac4672c5b06e6f91050aa46ee731bcf4 https://apnews.com/article/artificial-intelligence-hallucina...
  - dhoe 3y ago
    Bender also recently said in some podcast that there's basically no useful case for LLMs. I think she's a good candidate for some soundbite to add to "a global market of maybe five computers", "nobody needs more than 640k", etc (but she is probably not notable enough even for that).
- PeterCorless 3y ago
  I just asked ChatGPT to give me directions to my favorite shop in Manhattan from Penn Station. It gave me wrong public transit directions about which subway to take. It also gave me wrong walking directions, putting me blocks off course and as to which side of the street I'd find the shop on. The only thing it got right was: "Please note that subway schedules and routes can vary, so it's a good idea to use a navigation app like Google Maps or the official MTA website for real-time information and step-by-step directions."
  - redhale 3y ago
    Well I guess that ends the debate then. ChatGPT: not even as good as MapQuest. Checkmate, OpenAI.
PeterCorless 3y ago
By the way, I asked ChatGPT itself about how it would respond to the message above. Its answer: The statement you provided raises valid concerns about the capabilities and potential risks of advanced language models like ChatGPT Enterprise. Here's a response that addresses these concerns: "Thank you for sharing your thoughts on ChatGPT Enterprise. It's important to recognize that advanced language models like ChatGPT do indeed have a remarkable capacity to generate human-like responses. However, they lack inherent understanding, critical thinking, and ethical judgment. OpenAI acknowledges the concerns you've highlighted about accuracy, veracity, and potential consequences of generated content. It's crucial to understand that ChatGPT does not have an inherent "veracity bit" or a built-in mechanism to discern truth from falsehood. As such, it's important for users to exercise caution and discretion when relying on the output generated by the model. In situations where accuracy and compliance are paramount, it's advisable to rely on established procedural logic, expert systems, and human review. ChatGPT is not a substitute for human oversight, particularly in sensitive areas such as legal, medical, or business transactions. OpenAI is actively working to improve the control and guidance mechanisms for ChatGPT, and they acknowledge the importance of striking the right balance between automation and human review. As you pointed out, there are instances where generative AI is appropriate and others where traditional procedural logic and human intervention are necessary. While ChatGPT can be a powerful tool for generating content and ideas, it is incumbent upon users to apply their expertise and judgment to evaluate the output, particularly in contexts where accuracy and adherence to policies are critical. Your concerns align with ongoing discussions in the AI community about the ethical use and potential limitations of these technologies. As AI systems continue to advance, responsible development, transparency, and user education will remain important factors in their deployment." It's important to remember that while AI models like ChatGPT have the potential to assist and enhance various tasks, they do not replace the need for human expertise, ethical considerations, and careful oversight in complex and critical scenarios.
andrewjl 3y ago
Companies already hire (and pay them real money too) people to make stuff up about their products.
redhale 3y ago
I think this is a trap a lot of people fall into: assuming that in ChatGPT solutions that have business value, ChatGPT output is delivered directly to a customer. While that may be true in some cases (e.g. companies that produce cheap blog marketing content should be running scared), it ignores the iceberg of use cases where a ChatGPT tool can simply be one in a set of tools, like Excel is. Like think of a technical B2B product's technical support engineer who now has a personal assistant trained on the company's full catalog of technical documentation, who can provide instant answers to customer questions that the rep can validate before passing along to the customer on the phone. Or an overworked public defender who now has an instant paralegal to proof documents or search for and summarize relevant case law that the human lawyer then reviews. And so on.

lijok 3y ago

If you’re not reviewing ChatGPT output before utilizing it, you’re doing it wrong and will end up in hot water

PeterCorless 3y ago
100%. But people will do it wrong. A lot.
- lijok 3y ago
  Should get better over time as the tool gets better incorporated into the industry. People always misuse new tools
  - biophysboy 3y ago
    But why will it get better at accuracy if it isnt trained to be accurate?
    - lijok 3y ago
      What I mean is, the situation of people misusing it should get better over time
      - PeterCorless 3y ago
        Fair. Yes, people will eventually develop best practices through trial & error, and through better features/capabilities of the product/service itself. I just remain extremely cautious and pragmatic during this part of the hype cycle and zone of inflated expectations.
  - PeterCorless 3y ago
    In such cases, I believe there are (and should be) guardrails. It already has some guardrails internally set: ChatGPT already knows how to say: "I am not a lawyer, but I can provide you with some general information on this topic." The same thing for medicine: "I'm not a medical professional, but I can provide some general information..." But, for instance, ask it to design a canard for a 4th generation supersonic fighter, and suddenly it spits out pages of output. [Though it finally corrects itself and balks to answer when you ask for angle of forward sweep and a formula for the curvature of the surface.] There needs to be a way to "sniff out" if there's topics that people are getting too close to danger zones for it to answer. Ways for organizations themselves to set those 'bounding boxes.'
    - earthboundkid 3y ago

tsunamifury 3y ago

This sounds a bit like word salad, and your argument is worse than an LLM output. You even contradict yourself at the end. Also I think you need to better understand how much hallucination has been driven down recently and its clear path going forward.

PeterCorless 3y ago
“This isn’t fixable,” said Emily Bender, a linguistics professor and director of the University of Washington’s Computational Linguistics Laboratory. “It’s inherent in the mismatch between the technology and the proposed use cases.” https://apnews.com/article/artificial-intelligence-hallucination-chatbots-chatgpt-falsehoods-ac4672c5b06e6f91050aa46ee731bcf4 https://apnews.com/article/artificial-intelligence-hallucina...
- dhoe 3y ago
  Bender also recently said in some podcast that there's basically no useful case for LLMs. I think she's a good candidate for some soundbite to add to "a global market of maybe five computers", "nobody needs more than 640k", etc (but she is probably not notable enough even for that).
PeterCorless 3y ago
I just asked ChatGPT to give me directions to my favorite shop in Manhattan from Penn Station. It gave me wrong public transit directions about which subway to take. It also gave me wrong walking directions, putting me blocks off course and as to which side of the street I'd find the shop on. The only thing it got right was: "Please note that subway schedules and routes can vary, so it's a good idea to use a navigation app like Google Maps or the official MTA website for real-time information and step-by-step directions."
- redhale 3y ago
  Well I guess that ends the debate then. ChatGPT: not even as good as MapQuest. Checkmate, OpenAI.

PeterCorless 3y ago

By the way, I asked ChatGPT itself about how it would respond to the message above. Its answer: The statement you provided raises valid concerns about the capabilities and potential risks of advanced language models like ChatGPT Enterprise. Here's a response that addresses these concerns: "Thank you for sharing your thoughts on ChatGPT Enterprise. It's important to recognize that advanced language models like ChatGPT do indeed have a remarkable capacity to generate human-like responses. However, they lack inherent understanding, critical thinking, and ethical judgment. OpenAI acknowledges the concerns you've highlighted about accuracy, veracity, and potential consequences of generated content. It's crucial to understand that ChatGPT does not have an inherent "veracity bit" or a built-in mechanism to discern truth from falsehood. As such, it's important for users to exercise caution and discretion when relying on the output generated by the model. In situations where accuracy and compliance are paramount, it's advisable to rely on established procedural logic, expert systems, and human review. ChatGPT is not a substitute for human oversight, particularly in sensitive areas such as legal, medical, or business transactions. OpenAI is actively working to improve the control and guidance mechanisms for ChatGPT, and they acknowledge the importance of striking the right balance between automation and human review. As you pointed out, there are instances where generative AI is appropriate and others where traditional procedural logic and human intervention are necessary. While ChatGPT can be a powerful tool for generating content and ideas, it is incumbent upon users to apply their expertise and judgment to evaluate the output, particularly in contexts where accuracy and adherence to policies are critical. Your concerns align with ongoing discussions in the AI community about the ethical use and potential limitations of these technologies. As AI systems continue to advance, responsible development, transparency, and user education will remain important factors in their deployment." It's important to remember that while AI models like ChatGPT have the potential to assist and enhance various tasks, they do not replace the need for human expertise, ethical considerations, and careful oversight in complex and critical scenarios.

andrewjl 3y ago

Companies already hire (and pay them real money too) people to make stuff up about their products.

redhale 3y ago

I think this is a trap a lot of people fall into: assuming that in ChatGPT solutions that have business value, ChatGPT output is delivered directly to a customer. While that may be true in some cases (e.g. companies that produce cheap blog marketing content should be running scared), it ignores the iceberg of use cases where a ChatGPT tool can simply be one in a set of tools, like Excel is. Like think of a technical B2B product's technical support engineer who now has a personal assistant trained on the company's full catalog of technical documentation, who can provide instant answers to customer questions that the rep can validate before passing along to the customer on the phone. Or an overworked public defender who now has an instant paralegal to proof documents or search for and summarize relevant case law that the human lawyer then reviews. And so on.