6 ms·

Enterprises are getting stuck in AI pilot hell, say Chatterbox Labs execs

84 points by dijksterhuis 1y ago

matt3210 1y ago
Who in their right mind would intentionally deploy non-deterministic, unreviewable and unprovable software to critical systems?
- gamblor956 1y ago
  DOGE would and did. Results were as expected... complete failure.
- lo0dot0 1y ago
  The answers can be recorded and reviewed. The other points are true, or is there a way to make outcomes deterministic, when compared to previous versions while allowing to add more knowledge in newer versions?
  - vintermann 1y ago
    It's possible to make any model deterministic. Used to be just to save the seed, but I'm not sure it still is now that everything is distributed. Maybe a little more effort.
    - lo0dot0 1y ago
      A part of my question that you didn't go into was, can new knowledge be added in a new version without making the answers with knowledge learned in previous versions non-deterministic?
      - dijksterhuis 1y ago
        that’s not really how training works. changing the input (data) means you get a different output (model). source data has nothing to do with model determinism. as an end-user of AI products, your perspective might be that the models are non-deterministic, but really it’s just different models returning different results … because they are different models. “end-user non-determinism” is only really solved by repeatedly using the same version of a trained model (like a normal software dependency), potentially needing a bunch of work to upgrade the (model) dependency version later on.
    - dustingetz 1y ago
      determinism isn’t really enough, we want “predictable”. Most of these AI wavefunctions are “chaotic” - tiny changes in state can cause wildly divergent outcomes
    - Yoric 1y ago
      But that won't survive an upgrade, will it?
    - kevingadd 1y ago
      This requires an exact lock-down of things like the hardware and driver version, doesn't it? Is that sustainable?
      - vintermann 1y ago
        It shouldn't. It didn't used to, at least.
- smodo 1y ago
  My colleagues at the head of a company. I’m one of four bosses. One of us is pushing for AI every single meeting. The other is ignoring her. The last one is starting to ‘see her point.’ I’m considering quitting if this goes to far but unwilling to make that threat yet, as it’s a bridge I can only cross once. Anyway. To me it just speaks to the disdain for semi-intellectual work. People seem to think producing text has some value of its own. They think they can shortcircuit the basic assumption that behind every text is an intention that can be relied upon. They think that if they substitute this intention with a prompt, they can create the same value. I expect there to be some kind of bureaucratic collapse because of this, with parties unable to figure out responsibility around these zombie-texts. After that begins cleaning up, legislating and capturing in policy what the status of a given text is etc. Altman &co will have cashed out by then.
  - dustingetz 1y ago
    the essence of man is blind spots and hubris
  - mirekrusin 1y ago
    It's interesting to still hear this kind of sentiment. > People seem to think producing text has some value of its own. Reading this sentence makes me think that the author actually never seen agentic work in action? Producing value out of text does work and one of good examples is putting it in a loop with some form of verification output. It's easy to do with programming - type checker, tests, linter etc. – so it can chat by itself with it's own results until the problem is solved. I also find it personally strange that so often discussions require reminder that rate of change in capabilities is also big part of "the thing" (as opposed to pure capabilities today). It changes on weekly/monthly basis and it changes in one direction only.
    - dijksterhuis 1y ago
      i think you might have misunderstood the meaning of “producing text” in the parent comment. the kind of people the parent comment was talking about tend to believe they can send three emails and make millions of pounds suddenly appear in business value (i’m being hyperbolic and grossly unfair but the premise is there). they think the idea is far more valuable than the implementation - the idea is their bit (or the bit they’ve decided is their bit) and everyone else is there to make their fantastic idea magically appear out of thin air. they aren’t looking at tests and don’t have a clue what a linter is (they probably think it’s some fancy device to keep lint off their expensive suits).
- nikanj 1y ago
  Someone who was ordered by their boss to deploy it, and made sure to get the instructions in writing - with their protests also in writing.
- moron4hire 1y ago
  Someone who is really pissed off at how much they have to rely on software developers to run their business. They should not have so much power and direction in the company. I mean, they don't even have memberships at the country club!
- christophilus 1y ago
  Anyone who isn’t a software engineer. There is so much hype that non-technical people have bought into. Their tech teams should know better, but it’s hard to say “no”, when it feels like your salary depends on you saying “yes”.
  - mathgeek 1y ago
    > Their tech teams should know better, but it’s hard to say “no”, when it feels like your salary depends on you saying “yes”. There's some truth to the difference between "short term profits" and "my salary depends on this" being whether you're the boss or the employee.
- red75prime 1y ago
  Anyone who doesn't fully understand current differences between existing non-deterministic, unreviewable and unprovable agents (humans) and the artificial ones.
- rurban 1y ago
  If you train it the right data, there is no security risk. It cannot know what it doesn't see. However, if you train it on internal secrets, they will leak, simple as that. Filtering will not help. But this interview is only fear-mongering to sell expensive models. Ditching the industry leaders.
neepi 1y ago
One of my former contract outfits is there right now. Two failed projects so far, one of which impacted customers so badly that they ended up in trade press. The other one wrote off 5% of revenue with nothing to show. No you can't solve everything with a chatbot because your CEO needs an AI proposition or he's going to look silly down the golf course with all the other CEOs that aren't talking about how theirs are failing...
- tough 1y ago
  this does make sense, but there's infinite n ways to use AI on the workplace, i gotta wonder how much bad consultants trying to just sell services are to blame here. at least as much as the CEO's trying to shoehorn products nobody asked for i guess
  - neepi 1y ago
    Yes. They don't know what AI actually is and what the capabilities are and the companies selling integrations are running on hope rather than technical competence and suitability. So it gets applied to unsuitable problem domains and fails.
    - matt3210 1y ago
      They vibe coded everything so it basically a second year cs student level of work and security.
    - tough 1y ago
      I hate consultants, their incentives are all whack from the beginning. Hopefully more companies will encourage their own employees to explore how AI can fit on their current workflows or improve them and not try to hope for some magical thinking to solve their problems
      - SirBomalot 1y ago
        I currently have to deal with such consultants. They want to sell their magical AI black box. Speaking with the consultants let's me assume that they too get the pressure from the top to do ai stuff maybe because they fear that else they will be replaced by ai or so. It seems really somewhat desperate.
Garlef 1y ago
Doesn't this mean: There's room for disruption/land grab? If the big corporations can't move fast enough and 100 startups gamble on getting there, eventually one of them will be successful.
- Pmop 1y ago
  And a lot of them cannot get up to speed, even when they want to. Many big corporations struggle with evolution and innovation due to crippling bureaucracy, created and supported by risk averse leadership. This is usually worse for publicly traded companies. Unless it is something like Meta, then they have a Zuck, someone smart, with enough oversight and power, who can drain the swamp and make the whole machine move.
  - owebmaster 1y ago
    Zuckerberg made a genius move from the web 2.0 to the current smartphone era we still live in. But I would not be on his capability to do it again, he failed badly with metaverse and so far is failing with AI
    - deleted 1y ago
      [deleted]
  - cowboylowrez 1y ago
    hehe "drain the swamp" this guy knows how to "trump" the naysayers!
- nikanj 1y ago
  A hundred startups also gamble on perpetual motion, and their arguments always come from a place of "perpetual motion would revolutionize markets and there is strong demand", never from a place of "we have figured out how to alter laws of physics and make it possible"
- deleted 1y ago
  [deleted]
calebkaiser 1y ago
Before getting too invested in any conclusions drawn from this piece, it's important to recognize this is mostly PR from Chatterbox. From the Chatterbox site: > Our patented AIMI platform independently validates your AI models & data, generating quantitative AI risk metrics at scale. The article's subtitle: > Security, not model performance, is what's stalling adoption
gsky 1y ago
I have been using AI models to build 2 projects atm. Yes it's not perfect (30% wrong) but it solves problems so quickly and cheaply so I continue to use it going forward. As a software engineer I want everything to be perfect but not as an entrepreneur.
- add-sub-mul-div 1y ago
  Temu also solves a problem quickly and cheaply, but I wouldn't make it a big wardrobe strategy unless I was too poor to solve the problem a better way.
bsenftner 1y ago
Chatterbox's PR money is being well spent. This article squarely places them in the center of that trillion dollar revenue stream. Marketing dollars very well spent.
- stopthe 1y ago
  https://www.chatterbox.io/ https://www.chatterbox.io/ "Corporate language training powered by marginalised talent" - is that satire? Did I found the wrong Chatterbox?
  - simonw 1y ago
    That's the wrong one. https://chatterbox.co/ https://chatterbox.co/
  - deleted 1y ago
    [deleted]
sbarre 1y ago
AI and vibe coding lets you get that rough prototype up and running so much faster than before, and so creates that illusion of momentum and completeness more than ever. How many people here have been subjected to that "looks good, put it in production!" directive after showing off a quick POC for something? And then you have to explain how far away from being production-ready things are, etc... There's a reason wireframing tools intentionally use messy lines, and why most UX people know better than to put brand colours in wireframes.
- bowsamic 1y ago
  Prototypes are very dangerous. Our team made the mistake of having our demo look very nice even though there is still a lot of unseen work to do. Now upper management of course think “this is ready, just send it out”. Prototypes live forever, no upper manager will want you to spend time on the real thing. It is unsafe for the project to come across well
- pragmatic 1y ago
  Head of an engineering program told us to always make sure the prototype has at least one glaring bug/flaw. His background was electrical engineering but it applies doubly in software.
lofaszvanitt 1y ago
LLMs should be trained on CEOs and middle management and of course politicians. Society would be very grateful.
deleted 1y ago
[deleted]
kevin_thibedeau 1y ago
They've got to finish their blockchain deployment first. Then It'll all go smoothly.
bowsamic 1y ago
Slavoj Zizek says that the true terrifying situation is when the leaders act and know they no longer need to justify their actions. I am currently in this fight with our upper management. I ask why this push for AI, what it will do for our product, why are we making huge cuts on the scope of the project to rebrand as an AI project? All I receive is a bad confused response. Of course it’s just none of my business, they are the leaders
nyarlathotep_ 1y ago
Was on a few of these as a consultant, all major F500 companies. Most recent was a few months ago. Every instance was some variation of RAG chat/langgraph thing. On multiple occasions, I heard "I don't see what value this has over ChatGPT", except they now had 5-6 figure cloud bills with it. Technical users really weren't thrilled with it (i.e they wanted usable insights from their data (something best served by a db query), but ended up with LLM copypasta of internal docs) and seemed to expect significant functionality and utility on top of "regular" LLM use. Stakeholders constantly complained (rightfully so) about issues with inaccuracy in responses, or "why is this presented in this fashion", resulting in hours of the data team folks coming up with new prompts and crossing fingers.
- pragmatic 1y ago
  So right back to basic data engineering/analytics? “Why is this dashboard showing this number?” That’s my concern with any data “insight” magic. How do you debug what it’s telling the users?
  - asudhakar11 1y ago
    It should be able to tell you what assumptions it made. “Sales is $X because I assumed ARR and calendar year”. You’re then able to say great that’s what I wanted or “no I want bookings and fiscal year”.
- asudhakar11 1y ago
  Why weren’t you able to show usable insights from data?

84 points by dijksterhuis 1y ago

Who in their right mind would intentionally deploy non-deterministic, unreviewable and unprovable software to critical systems?

gamblor956 1y ago
DOGE would and did. Results were as expected... complete failure.
lo0dot0 1y ago
The answers can be recorded and reviewed. The other points are true, or is there a way to make outcomes deterministic, when compared to previous versions while allowing to add more knowledge in newer versions?
- vintermann 1y ago
  It's possible to make any model deterministic. Used to be just to save the seed, but I'm not sure it still is now that everything is distributed. Maybe a little more effort.
  - lo0dot0 1y ago
    A part of my question that you didn't go into was, can new knowledge be added in a new version without making the answers with knowledge learned in previous versions non-deterministic?
    - dijksterhuis 1y ago
      that’s not really how training works. changing the input (data) means you get a different output (model). source data has nothing to do with model determinism. as an end-user of AI products, your perspective might be that the models are non-deterministic, but really it’s just different models returning different results … because they are different models. “end-user non-determinism” is only really solved by repeatedly using the same version of a trained model (like a normal software dependency), potentially needing a bunch of work to upgrade the (model) dependency version later on.
  - dustingetz 1y ago
    determinism isn’t really enough, we want “predictable”. Most of these AI wavefunctions are “chaotic” - tiny changes in state can cause wildly divergent outcomes
  - Yoric 1y ago
    But that won't survive an upgrade, will it?
  - kevingadd 1y ago
    This requires an exact lock-down of things like the hardware and driver version, doesn't it? Is that sustainable?
    - vintermann 1y ago
      It shouldn't. It didn't used to, at least.
smodo 1y ago
My colleagues at the head of a company. I’m one of four bosses. One of us is pushing for AI every single meeting. The other is ignoring her. The last one is starting to ‘see her point.’ I’m considering quitting if this goes to far but unwilling to make that threat yet, as it’s a bridge I can only cross once. Anyway. To me it just speaks to the disdain for semi-intellectual work. People seem to think producing text has some value of its own. They think they can shortcircuit the basic assumption that behind every text is an intention that can be relied upon. They think that if they substitute this intention with a prompt, they can create the same value. I expect there to be some kind of bureaucratic collapse because of this, with parties unable to figure out responsibility around these zombie-texts. After that begins cleaning up, legislating and capturing in policy what the status of a given text is etc. Altman &co will have cashed out by then.
- dustingetz 1y ago
  the essence of man is blind spots and hubris
- mirekrusin 1y ago
  It's interesting to still hear this kind of sentiment. > People seem to think producing text has some value of its own. Reading this sentence makes me think that the author actually never seen agentic work in action? Producing value out of text does work and one of good examples is putting it in a loop with some form of verification output. It's easy to do with programming - type checker, tests, linter etc. – so it can chat by itself with it's own results until the problem is solved. I also find it personally strange that so often discussions require reminder that rate of change in capabilities is also big part of "the thing" (as opposed to pure capabilities today). It changes on weekly/monthly basis and it changes in one direction only.
  - dijksterhuis 1y ago
    i think you might have misunderstood the meaning of “producing text” in the parent comment. the kind of people the parent comment was talking about tend to believe they can send three emails and make millions of pounds suddenly appear in business value (i’m being hyperbolic and grossly unfair but the premise is there). they think the idea is far more valuable than the implementation - the idea is their bit (or the bit they’ve decided is their bit) and everyone else is there to make their fantastic idea magically appear out of thin air. they aren’t looking at tests and don’t have a clue what a linter is (they probably think it’s some fancy device to keep lint off their expensive suits).
nikanj 1y ago
Someone who was ordered by their boss to deploy it, and made sure to get the instructions in writing - with their protests also in writing.
moron4hire 1y ago
Someone who is really pissed off at how much they have to rely on software developers to run their business. They should not have so much power and direction in the company. I mean, they don't even have memberships at the country club!
christophilus 1y ago
Anyone who isn’t a software engineer. There is so much hype that non-technical people have bought into. Their tech teams should know better, but it’s hard to say “no”, when it feels like your salary depends on you saying “yes”.
- mathgeek 1y ago
  > Their tech teams should know better, but it’s hard to say “no”, when it feels like your salary depends on you saying “yes”. There's some truth to the difference between "short term profits" and "my salary depends on this" being whether you're the boss or the employee.
red75prime 1y ago
Anyone who doesn't fully understand current differences between existing non-deterministic, unreviewable and unprovable agents (humans) and the artificial ones.
rurban 1y ago
If you train it the right data, there is no security risk. It cannot know what it doesn't see. However, if you train it on internal secrets, they will leak, simple as that. Filtering will not help. But this interview is only fear-mongering to sell expensive models. Ditching the industry leaders.

neepi 1y ago

One of my former contract outfits is there right now. Two failed projects so far, one of which impacted customers so badly that they ended up in trade press. The other one wrote off 5% of revenue with nothing to show. No you can't solve everything with a chatbot because your CEO needs an AI proposition or he's going to look silly down the golf course with all the other CEOs that aren't talking about how theirs are failing...

tough 1y ago
this does make sense, but there's infinite n ways to use AI on the workplace, i gotta wonder how much bad consultants trying to just sell services are to blame here. at least as much as the CEO's trying to shoehorn products nobody asked for i guess
- neepi 1y ago
  Yes. They don't know what AI actually is and what the capabilities are and the companies selling integrations are running on hope rather than technical competence and suitability. So it gets applied to unsuitable problem domains and fails.
  - matt3210 1y ago
    They vibe coded everything so it basically a second year cs student level of work and security.
  - tough 1y ago
    I hate consultants, their incentives are all whack from the beginning. Hopefully more companies will encourage their own employees to explore how AI can fit on their current workflows or improve them and not try to hope for some magical thinking to solve their problems
    - SirBomalot 1y ago
      I currently have to deal with such consultants. They want to sell their magical AI black box. Speaking with the consultants let's me assume that they too get the pressure from the top to do ai stuff maybe because they fear that else they will be replaced by ai or so. It seems really somewhat desperate.

Garlef 1y ago

Doesn't this mean: There's room for disruption/land grab? If the big corporations can't move fast enough and 100 startups gamble on getting there, eventually one of them will be successful.

Pmop 1y ago
And a lot of them cannot get up to speed, even when they want to. Many big corporations struggle with evolution and innovation due to crippling bureaucracy, created and supported by risk averse leadership. This is usually worse for publicly traded companies. Unless it is something like Meta, then they have a Zuck, someone smart, with enough oversight and power, who can drain the swamp and make the whole machine move.
- owebmaster 1y ago
  Zuckerberg made a genius move from the web 2.0 to the current smartphone era we still live in. But I would not be on his capability to do it again, he failed badly with metaverse and so far is failing with AI
  - deleted 1y ago
    [deleted]
- cowboylowrez 1y ago
  hehe "drain the swamp" this guy knows how to "trump" the naysayers!
nikanj 1y ago
A hundred startups also gamble on perpetual motion, and their arguments always come from a place of "perpetual motion would revolutionize markets and there is strong demand", never from a place of "we have figured out how to alter laws of physics and make it possible"
deleted 1y ago
[deleted]

calebkaiser 1y ago

Before getting too invested in any conclusions drawn from this piece, it's important to recognize this is mostly PR from Chatterbox. From the Chatterbox site: > Our patented AIMI platform independently validates your AI models & data, generating quantitative AI risk metrics at scale. The article's subtitle: > Security, not model performance, is what's stalling adoption

gsky 1y ago

I have been using AI models to build 2 projects atm. Yes it's not perfect (30% wrong) but it solves problems so quickly and cheaply so I continue to use it going forward. As a software engineer I want everything to be perfect but not as an entrepreneur.

add-sub-mul-div 1y ago
Temu also solves a problem quickly and cheaply, but I wouldn't make it a big wardrobe strategy unless I was too poor to solve the problem a better way.

bsenftner 1y ago

Chatterbox's PR money is being well spent. This article squarely places them in the center of that trillion dollar revenue stream. Marketing dollars very well spent.

stopthe 1y ago
https://www.chatterbox.io/ https://www.chatterbox.io/ "Corporate language training powered by marginalised talent" - is that satire? Did I found the wrong Chatterbox?
- simonw 1y ago
  That's the wrong one. https://chatterbox.co/ https://chatterbox.co/
- deleted 1y ago
  [deleted]

sbarre 1y ago

AI and vibe coding lets you get that rough prototype up and running so much faster than before, and so creates that illusion of momentum and completeness more than ever. How many people here have been subjected to that "looks good, put it in production!" directive after showing off a quick POC for something? And then you have to explain how far away from being production-ready things are, etc... There's a reason wireframing tools intentionally use messy lines, and why most UX people know better than to put brand colours in wireframes.

bowsamic 1y ago
Prototypes are very dangerous. Our team made the mistake of having our demo look very nice even though there is still a lot of unseen work to do. Now upper management of course think “this is ready, just send it out”. Prototypes live forever, no upper manager will want you to spend time on the real thing. It is unsafe for the project to come across well
pragmatic 1y ago
Head of an engineering program told us to always make sure the prototype has at least one glaring bug/flaw. His background was electrical engineering but it applies doubly in software.

lofaszvanitt 1y ago

LLMs should be trained on CEOs and middle management and of course politicians. Society would be very grateful.

deleted 1y ago

[deleted]

kevin_thibedeau 1y ago

They've got to finish their blockchain deployment first. Then It'll all go smoothly.

bowsamic 1y ago

Slavoj Zizek says that the true terrifying situation is when the leaders act and know they no longer need to justify their actions. I am currently in this fight with our upper management. I ask why this push for AI, what it will do for our product, why are we making huge cuts on the scope of the project to rebrand as an AI project? All I receive is a bad confused response. Of course it’s just none of my business, they are the leaders

nyarlathotep_ 1y ago

Was on a few of these as a consultant, all major F500 companies. Most recent was a few months ago. Every instance was some variation of RAG chat/langgraph thing. On multiple occasions, I heard "I don't see what value this has over ChatGPT", except they now had 5-6 figure cloud bills with it. Technical users really weren't thrilled with it (i.e they wanted usable insights from their data (something best served by a db query), but ended up with LLM copypasta of internal docs) and seemed to expect significant functionality and utility on top of "regular" LLM use. Stakeholders constantly complained (rightfully so) about issues with inaccuracy in responses, or "why is this presented in this fashion", resulting in hours of the data team folks coming up with new prompts and crossing fingers.

pragmatic 1y ago
So right back to basic data engineering/analytics? “Why is this dashboard showing this number?” That’s my concern with any data “insight” magic. How do you debug what it’s telling the users?
- asudhakar11 1y ago
  It should be able to tell you what assumptions it made. “Sales is $X because I assumed ARR and calendar year”. You’re then able to say great that’s what I wanted or “no I want bookings and fiscal year”.
asudhakar11 1y ago
Why weren’t you able to show usable insights from data?