4 ms·

> Although I was on an AI team, I often pushed back against applying AI unless we had a really compelling reason. What is the user problem we are trying to solv

by mtrovo 2y ago

> Although I was on an AI team, I often pushed back against applying AI unless we had a really compelling reason. What is the user problem we are trying to solve? Do we really need an LLM or could a few if statements suffice? Are we sure that natural language is the appropriate interface for this? This practical approach to AI feels refreshing in a field drowning in buzzwords. I’ve built tools where simple regression models outperformed neural networks, and convincing teams was an uphill battle. It's hard to not get pushback from teams for not going all-in on AI when it seems decisions and budgets are hype-driven.

d_sem 2y ago
This is a perfect lesson in why strong communication skills are important in engineering organizations. It's leaderships responsibility to learn from engineering what is technically feasible but its also engineering's responsibility to communicate well enough to convince their organization on the right path forward.
spacecadet 2y ago
For years I worked at a company doing manufacturing automation with a generative CAD component. People would constantly throw out, "use ML" "use AI", everything was just regressions and open cascade... and those people never understood that it worked just fine without buzzwords.
- BOOSTERHIDROGEN 2y ago
  Could you offer more insight into how regression analysis and open cascade are utilized in manufacturing processes?
cm2187 2y ago
CV driven development!
extr 2y ago
I have pushed back in a similar way many times (wrt LLMs), the response I typically get is some combination of: - A custom/heuristic driven approach would perform better but will take much longer to build so would be lower ROI. - There is a strategic angle to using AI here (building competency). We aren't sure what new use cases will open up in the medium term and we need to be fluent with building AI products. - There is a perceptual/marketing angle to using AI here. We need to convince the market/investors we are on the bleeding edge (hype). 3 is widely mocked but is a completely rational allocation of resources when you need to compete in a market for funding.
- _glass 2y ago
  Prolog is AI, so whenever I see such a problem, I use miniKanren, implementing Relational/Logic programming in a light-weight way. Bleeding edge AI it is.
- physicsguy 2y ago
  > will take much longer to build so would be lower ROI This one is funny because my experience has been that ekeing out the issues in this sort of thing is enormously complicated and unreliable and takes an inordinate amount of time. Often the 'bugs' aren't trivially fixable. One we had was the LLM formatting URIs given in the prompt wrongly meaning they're no longer valid. Most of the time it works fine, but sometimes it doesn't, and it's not reproducible easily.
  - extr 2y ago
    It's true, it can be maddening (impossible?) to chase down all the edge-case failures LLMs produce. But outside of life/death applications with extreme accuracy requirements (eg: medical diagnostics) the attitude I've seen is: who cares? A lot of users "get" AI now and don't really expect it to be 100% reliable. They're satisfied with a 95% solution, especially if it was deployed quickly and produces something they can iterate on for the last 5%.
- jorblumesea 2y ago
  for 1/2, surprised to hear this because debugging models is usually a total black box and practically impossible. for 2, it's a similar problem where getting performance and accuracy using the same model over and over again on different problem sets can be challenging. not an AI expert or anything this has been my experience on the product side.
  - extr 2y ago
    Responded to the same sentiment elsewhere but my general sense is that for many use cases users simply do not care about high 9s accuracy/consistency. A 95% solution using AI is "good enough" if you can ship it quickly and give them the tools to iterate on that last 5%.
j45 2y ago
It's important to consider if existing tech can do something as well if not better. LLMs can have have great application where existing tech can't reach. Too often, seeing LLMs doing something that's done better already by an existing tech or something it's not designed for seems to miss the impact being sought.
IanCal 2y ago
I've found having a cascade of things helps. Trying to split things into "decision at this later, or pass on to the next" with increasingly complicated models/approaches. Start with ifs, then svms then something else for example. This has some technical benefits, like speed, and gives you a place to put important hard coded fixes for where a better model makes a small but key mistake. But the bigger benefit imo is getting something to solve the bulk of the problem quicker, and a organisationally it means not saying no to an approach - just where it fits and at what level of improvement it's worth it.
- IanCal 2y ago
  Too late to edit - "decision at this layer"
frereubu 2y ago
I have a friend who was tasked by a think tank to build an AI model for which they'd managed to get an enormous amount of funding. My friend pointed out that it would be much more effective if they used a regression model that could be built in a couple of days by a junior developer. Eventually he was fired from the project for sticking to his guns and refusing to pad his time while building something worse just so they could say they'd used "AI".

by mtrovo 2y ago

d_sem 2y ago

This is a perfect lesson in why strong communication skills are important in engineering organizations. It's leaderships responsibility to learn from engineering what is technically feasible but its also engineering's responsibility to communicate well enough to convince their organization on the right path forward.

spacecadet 2y ago

For years I worked at a company doing manufacturing automation with a generative CAD component. People would constantly throw out, "use ML" "use AI", everything was just regressions and open cascade... and those people never understood that it worked just fine without buzzwords.

BOOSTERHIDROGEN 2y ago
Could you offer more insight into how regression analysis and open cascade are utilized in manufacturing processes?

cm2187 2y ago

CV driven development!

extr 2y ago

I have pushed back in a similar way many times (wrt LLMs), the response I typically get is some combination of: - A custom/heuristic driven approach would perform better but will take much longer to build so would be lower ROI. - There is a strategic angle to using AI here (building competency). We aren't sure what new use cases will open up in the medium term and we need to be fluent with building AI products. - There is a perceptual/marketing angle to using AI here. We need to convince the market/investors we are on the bleeding edge (hype). 3 is widely mocked but is a completely rational allocation of resources when you need to compete in a market for funding.

_glass 2y ago
Prolog is AI, so whenever I see such a problem, I use miniKanren, implementing Relational/Logic programming in a light-weight way. Bleeding edge AI it is.
physicsguy 2y ago
> will take much longer to build so would be lower ROI This one is funny because my experience has been that ekeing out the issues in this sort of thing is enormously complicated and unreliable and takes an inordinate amount of time. Often the 'bugs' aren't trivially fixable. One we had was the LLM formatting URIs given in the prompt wrongly meaning they're no longer valid. Most of the time it works fine, but sometimes it doesn't, and it's not reproducible easily.
- extr 2y ago
  It's true, it can be maddening (impossible?) to chase down all the edge-case failures LLMs produce. But outside of life/death applications with extreme accuracy requirements (eg: medical diagnostics) the attitude I've seen is: who cares? A lot of users "get" AI now and don't really expect it to be 100% reliable. They're satisfied with a 95% solution, especially if it was deployed quickly and produces something they can iterate on for the last 5%.
jorblumesea 2y ago
for 1/2, surprised to hear this because debugging models is usually a total black box and practically impossible. for 2, it's a similar problem where getting performance and accuracy using the same model over and over again on different problem sets can be challenging. not an AI expert or anything this has been my experience on the product side.
- extr 2y ago
  Responded to the same sentiment elsewhere but my general sense is that for many use cases users simply do not care about high 9s accuracy/consistency. A 95% solution using AI is "good enough" if you can ship it quickly and give them the tools to iterate on that last 5%.

j45 2y ago

It's important to consider if existing tech can do something as well if not better. LLMs can have have great application where existing tech can't reach. Too often, seeing LLMs doing something that's done better already by an existing tech or something it's not designed for seems to miss the impact being sought.

IanCal 2y ago

I've found having a cascade of things helps. Trying to split things into "decision at this later, or pass on to the next" with increasingly complicated models/approaches. Start with ifs, then svms then something else for example. This has some technical benefits, like speed, and gives you a place to put important hard coded fixes for where a better model makes a small but key mistake. But the bigger benefit imo is getting something to solve the bulk of the problem quicker, and a organisationally it means not saying no to an approach - just where it fits and at what level of improvement it's worth it.

IanCal 2y ago
Too late to edit - "decision at this layer"

frereubu 2y ago

I have a friend who was tasked by a think tank to build an AI model for which they'd managed to get an enormous amount of funding. My friend pointed out that it would be much more effective if they used a regression model that could be built in a couple of days by a junior developer. Eventually he was fired from the project for sticking to his guns and refusing to pad his time while building something worse just so they could say they'd used "AI".