6 ms·

Super cool, thanks for sharing! This is one of the reasons I am so skeptical of the current AI hype cycle. There are boring, well-behaved classical solutions f

Super cool, thanks for sharing! This is one of the reasons I am so skeptical of the current AI hype cycle. There are boring, well-behaved classical solutions for many of the use-cases where fancy ML is pushed today. You'd think that rational businesses would take the low-risk snooze-fest high-margin option any day instead of unintelligible and unreliable options that demand a lot of resources, and yet...

menaerus 11mo ago
> There are boring, well-behaved classical solutions for many of the use-cases where fancy ML is pushed today. I know some examples but not too many. Care to share more examples?
- auxiliarymoose 11mo ago
  Some off the top of my head... - Instead of trying to get LLMs to answer user questions, write better FAQs informed by reviewing tickets submitted by customers - Instead of RAG for anything involving business data, have some DBA write a bunch of reports that answer specific business questions - Instead of putting some copilot chat into tools and telling users to ask it to e.g. "explain recent sales trends", make task-focused wizards and visualizations so users can answer these with hard numbers - Instead of generating code with LLMs, write more expressive frameworks and libraries that don't require so much plumbing and boilerplate Of course, maybe there is something I am missing, but these are just my personal observations!
  - menaerus 11mo ago
    With all due respect, all of those examples are the examples of "yesterday" ... that's how we have been bringing money to businesses for decades, no? Today we have AI models that can already do as good, almost as good, or even better than the average human in many many tasks, including the ones you mentioned. Businesses are incentivized to be more productive and cost-effective since they are solely profit-driven so they naturally see this as an opportunity to make more money by hiring less people while keeping the amount of work done roughly the same or even more. So "classical" approach to many of the problems is I think the thing of a past already.
    - auxiliarymoose 11mo ago
      > Today we have AI models that can already do as good, almost as good, or even better than the average human in many many tasks, including the ones you mentioned. We really don't. There are demos that look cool onstage, but there is a big difference between "in store good" and "at home good" in the sense that products aren't living up to their marketing during actual use. IMO there is a lot of room to grow within the traditional approaches of "yesterday" - The problem is that large orgs get bogged down in legacy + bureaucracy, and most startups don't understand the business problems well enough to make a better solution. And I don't think that there is any technical silver bullet that can solve either of these problems (AI or otherwise)
      - menaerus 11mo ago
        I am wondering how often do you use AI models? Because I do it on a daily basis, and as much as they have limitations, I find them to be performing incredibly well. It's far very far from being a demo - last time it was a demo that looked "cool" was around 2020/21 when they were cool for spitting out the haiku poetry, and perhaps 2022 when capabilities were not as good. But today? Completely mind-blowing. If you're not convinced, I suggest you to search for the law firms, hospitals, and laboratories ... all of which are using AI models as of today to do both the research and boiler-plate work. Creative industries are being literally erased by the generative AI as we are speaking. What will happen with the Photoshop and other similar tools when I can create whatever I want using the free AI model in literally 2 seconds without prior knowledge? What will happen with majority of movie effect makers when single guy will be able to do the work of 5 people at the same time? Or interior designers? The heck, what will happen with the Google search - I anticipate nobody will be using it in a year or two. I already don't because it's a massive sink of time compared to what I can do with perplexity for example. There's many many examples. You just need to have your mind open to see it.
        eitally 11mo ago
        You're making a ridiculously overconfident statement. * Show me a discrete manufacturing company using AI models for statistical process control or quality reporting * Show me a pharmaceutical company using AI models for safety data analysis * Show me an engineering company using AI models for structural design The list goes on and on. There are precious few industries or companies that have replaced traditional analysis & prediction with AI. Why? Because one of two things are true: 1) their data is already in highly structured relational stores that have long legacies of SQL-based extraction and analysis, 2) they're in regulated industries and have to have audit-proof, explainable reporting, or 3) they need evidence-based design and analysis that has a key component coming from real people observing real processes in action. For all the hyped "AI Automation" you read about, there are 100 other things that aren't, or where firms don't believe they can be, or where they'll struggle to for [reasons].
  - fishmicrowaver 11mo ago
    I agree, however, I've seen first hand how the AI fever and mandate from the top has finally busted enough information silos that previously 'have some DBA write a bunch of reports that answer specific business questions' just wasn't feasible in the first place, and now is.
    - auxiliarymoose 11mo ago
      Yeah, I will say it provides a good excuse to leadership & external stakeholders for fixing technical debt and working cross-functionally! Making infrastructure "AI-ready" and all that. I imagine more of the benefit of AI will come from companies prioritizing infrastructure and effective organization than from the technology itself.
- srean 11mo ago
  In the realm of data science, Linear models and SAT solvers used cleverly will get you a surprisingly long way.
  - menaerus 11mo ago
    I thought the OCR was one of the obvious examples where we have a classical technology that is already working very well but in the long-run I don't see it surviving. _Generic_ AI models already can do the OCR kinda good but they are not even trained for that purpose, it's almost incidental - they've never been trained to extract the, let's say name/surname from some sort of a document with a completely unfamiliar structure, but the crazy thing is that it does work somehow! I think that once somebody finetunes the AI model only for this purpose I think there's a good chance it will outperform classical approach in terms of precision and scalability.
    - srean 11mo ago
      In general I agree. For OCR I agree vehemently. Part of the reason is the structure of the solution (convolutions) match the space so well. The failure cases are those where AI solutions have to stay in a continuous debug, train, update mode. Then you have to think about the resources you need, both in terms of people as well as compute to maintain such a solution. Because of the way the world works, it's endemic nonstationarity, the debug-retrain-update is a common state of affairs even in traditional stats and ML.
      - menaerus 11mo ago
        I see. Let's take another example here, I hope I understood you - imagine you have an AI model which is connected to all of your company's in-house data generation sources such as wiki, chat, jira, emails, merge requests, excel sheets, etc. Basically everything that can be deemed useful to query or to create business inteligence on top of. These data sources are continously generating more and more data every day, and given their nature they are more or less unstructured. Yet, we have such systems in place where we don't have to retrain the model with ever-growing data. This is one example I could think of but it kinda suggests that models, at least for some purposes, don't have to be retrained continuously to keep them running well. I also use a technique of explaining something to the AI model he has not seen before (according to the wrong answer I got from it previously), and it manages to evolve the steps, whatever they are, so that it gives me the correct answer in the end. This also suggests that capacity of the models is larger than what they have been trained on.
- AlotOfReading 11mo ago
  In my domain, I see lots of people reaching immediately for "AI" techniques to solve sensor fusion and state estimation problems where a traditional Kalman filter type solution would be faster and much more interpretable.
  - menaerus 11mo ago
    Incidentally, I worked on the exact same thing - Kalman filtering for tracking objects in hard real-time systems. And it is not quite as simple as one would think - developing mathematical models for all kinds of different objects that one might wanna track is far from trivial, and it was difficult to model the real-world with more or less simplistic discrete equations. And it didn't work completely reliably so we needed an extra layer of confidence - I don't remember what we used back then but it was yet another algorithm with yet another source of data.
    - srean 11mo ago
      Sigma point filters ? The escalation ladder is usually KF, EKF, unscented KF, sigma point ...
      - menaerus 11mo ago
        I believe it was random forest that we combined with EKF.
    - AlotOfReading 11mo ago
      There's a lot of situations where you genuinely need more sophisticated algorithms. I'm talking about people reaching for those kinds of things before they determine the traditional algorithms won't work.
srean 11mo ago
It does not work that way in the short term. Say you have bet billions as a CEO, CTO, CFO. The decision has already been made. Such a steep price had to come at the cost of many groups and teams and projects in the company. Now is not a time to water plants that offer alternatives. You will have a smoother ride choosing tools that justifies that billion dollar bet.
- auxiliarymoose 11mo ago
  Decision-making in organizations is definitely a hard problem. I think an uncomfortable reality is that a lot of decisions (technology, strategy, etc.) are not optimal or even rational, but more just an outcome of personal preferences. Even data-driven approaches aren't immune since they depend on the analysis and interpretation of the data (which is subjective).
  - srean 11mo ago
    Data informed is good. Purely data driven is a bad idea. After all even in Physics big advances came from thought experiments. Data is one way to reason about a decision, logic and knowledgebase is another way. Both can be very powerful if one retains the humility of fallibility. In organizations one common failure mode is that the organisational level at which decisions are made are not the same levels where the decisions are going to have their effects felt. It's a really difficult problem to solve. Too much decentralisation is also a bad idea. You get the mess of unplanned congested cities.
    - cb321 11mo ago
      For a while now, I've been summarizing the ease with which everything turns into a "Humanity Complete" problem via: "Delegation affords so much, but trust sure is tricky." This has been observed forever in various forms/contexts. Planning & policy people call them "Wicked Problems" (https://en.wikipedia.org/wiki/Wicked_problem https://en.wikipedia.org/wiki/Wicked_problem). The Philosophy of Science one goes by the Demarcation Problem (https://en.wikipedia.org/wiki/Demarcation_problem https://en.wikipedia.org/wiki/Demarcation_problem) { roughly, in the sense that the really hard nugget connects to "trust" }. At least one aspect of all of it is that trust is a little like money/capital and "faking it" is a bit like "stealing". The game theory of it is that since faking is virtually always vastly cheaper there are (eventually) huge incentives to do so, at some point by someone(s). So, almost any kind of trust/delegation structure has a strong pull toward "decay", from knock-off brands to whatever. It just takes a sadly small fraction of Prisoner's Dilemma defectors to ruin things/systems thereof. 2nd law of thermo makes order cost energy and this decay feels like almost an isomorphic (maybe even the same..?) thing. It's not just product/tech enshittification, but that might be yet another special case/example. Anyway, I have no great answers or as some responder to me a while back said, if I did, I'd "have a Nobel and possibly be the first president of the united planet".
DavidPeiffer 11mo ago
>This is one of the reasons I am so skeptical of the current AI hype cycle. There are boring, well-behaved classical solutions for many of the use-cases where fancy ML is pushed today. In 2013 my statistics professor warned that once we are in the real world, "people will come up to you trying to sell fancy machine learning models for big money, though the simple truth is that many problems can be solved better by applying straightforward statistical methods". There has always been the ML hype, but the last couple years are a whole different level.
fuzzfactor 11mo ago
>unintelligible and unreliable options that demand a lot of resources Some options have more persuasive salesmen than others.

menaerus 11mo ago

> There are boring, well-behaved classical solutions for many of the use-cases where fancy ML is pushed today. I know some examples but not too many. Care to share more examples?

auxiliarymoose 11mo ago
Some off the top of my head... - Instead of trying to get LLMs to answer user questions, write better FAQs informed by reviewing tickets submitted by customers - Instead of RAG for anything involving business data, have some DBA write a bunch of reports that answer specific business questions - Instead of putting some copilot chat into tools and telling users to ask it to e.g. "explain recent sales trends", make task-focused wizards and visualizations so users can answer these with hard numbers - Instead of generating code with LLMs, write more expressive frameworks and libraries that don't require so much plumbing and boilerplate Of course, maybe there is something I am missing, but these are just my personal observations!
- menaerus 11mo ago
  With all due respect, all of those examples are the examples of "yesterday" ... that's how we have been bringing money to businesses for decades, no? Today we have AI models that can already do as good, almost as good, or even better than the average human in many many tasks, including the ones you mentioned. Businesses are incentivized to be more productive and cost-effective since they are solely profit-driven so they naturally see this as an opportunity to make more money by hiring less people while keeping the amount of work done roughly the same or even more. So "classical" approach to many of the problems is I think the thing of a past already.
  - auxiliarymoose 11mo ago
    > Today we have AI models that can already do as good, almost as good, or even better than the average human in many many tasks, including the ones you mentioned. We really don't. There are demos that look cool onstage, but there is a big difference between "in store good" and "at home good" in the sense that products aren't living up to their marketing during actual use. IMO there is a lot of room to grow within the traditional approaches of "yesterday" - The problem is that large orgs get bogged down in legacy + bureaucracy, and most startups don't understand the business problems well enough to make a better solution. And I don't think that there is any technical silver bullet that can solve either of these problems (AI or otherwise)
    - menaerus 11mo ago
      I am wondering how often do you use AI models? Because I do it on a daily basis, and as much as they have limitations, I find them to be performing incredibly well. It's far very far from being a demo - last time it was a demo that looked "cool" was around 2020/21 when they were cool for spitting out the haiku poetry, and perhaps 2022 when capabilities were not as good. But today? Completely mind-blowing. If you're not convinced, I suggest you to search for the law firms, hospitals, and laboratories ... all of which are using AI models as of today to do both the research and boiler-plate work. Creative industries are being literally erased by the generative AI as we are speaking. What will happen with the Photoshop and other similar tools when I can create whatever I want using the free AI model in literally 2 seconds without prior knowledge? What will happen with majority of movie effect makers when single guy will be able to do the work of 5 people at the same time? Or interior designers? The heck, what will happen with the Google search - I anticipate nobody will be using it in a year or two. I already don't because it's a massive sink of time compared to what I can do with perplexity for example. There's many many examples. You just need to have your mind open to see it.
      - eitally 11mo ago
        You're making a ridiculously overconfident statement. * Show me a discrete manufacturing company using AI models for statistical process control or quality reporting * Show me a pharmaceutical company using AI models for safety data analysis * Show me an engineering company using AI models for structural design The list goes on and on. There are precious few industries or companies that have replaced traditional analysis & prediction with AI. Why? Because one of two things are true: 1) their data is already in highly structured relational stores that have long legacies of SQL-based extraction and analysis, 2) they're in regulated industries and have to have audit-proof, explainable reporting, or 3) they need evidence-based design and analysis that has a key component coming from real people observing real processes in action. For all the hyped "AI Automation" you read about, there are 100 other things that aren't, or where firms don't believe they can be, or where they'll struggle to for [reasons].
- fishmicrowaver 11mo ago
  I agree, however, I've seen first hand how the AI fever and mandate from the top has finally busted enough information silos that previously 'have some DBA write a bunch of reports that answer specific business questions' just wasn't feasible in the first place, and now is.
  - auxiliarymoose 11mo ago
    Yeah, I will say it provides a good excuse to leadership & external stakeholders for fixing technical debt and working cross-functionally! Making infrastructure "AI-ready" and all that. I imagine more of the benefit of AI will come from companies prioritizing infrastructure and effective organization than from the technology itself.
srean 11mo ago
In the realm of data science, Linear models and SAT solvers used cleverly will get you a surprisingly long way.
- menaerus 11mo ago
  I thought the OCR was one of the obvious examples where we have a classical technology that is already working very well but in the long-run I don't see it surviving. _Generic_ AI models already can do the OCR kinda good but they are not even trained for that purpose, it's almost incidental - they've never been trained to extract the, let's say name/surname from some sort of a document with a completely unfamiliar structure, but the crazy thing is that it does work somehow! I think that once somebody finetunes the AI model only for this purpose I think there's a good chance it will outperform classical approach in terms of precision and scalability.
  - srean 11mo ago
    In general I agree. For OCR I agree vehemently. Part of the reason is the structure of the solution (convolutions) match the space so well. The failure cases are those where AI solutions have to stay in a continuous debug, train, update mode. Then you have to think about the resources you need, both in terms of people as well as compute to maintain such a solution. Because of the way the world works, it's endemic nonstationarity, the debug-retrain-update is a common state of affairs even in traditional stats and ML.
    - menaerus 11mo ago
      I see. Let's take another example here, I hope I understood you - imagine you have an AI model which is connected to all of your company's in-house data generation sources such as wiki, chat, jira, emails, merge requests, excel sheets, etc. Basically everything that can be deemed useful to query or to create business inteligence on top of. These data sources are continously generating more and more data every day, and given their nature they are more or less unstructured. Yet, we have such systems in place where we don't have to retrain the model with ever-growing data. This is one example I could think of but it kinda suggests that models, at least for some purposes, don't have to be retrained continuously to keep them running well. I also use a technique of explaining something to the AI model he has not seen before (according to the wrong answer I got from it previously), and it manages to evolve the steps, whatever they are, so that it gives me the correct answer in the end. This also suggests that capacity of the models is larger than what they have been trained on.
AlotOfReading 11mo ago
In my domain, I see lots of people reaching immediately for "AI" techniques to solve sensor fusion and state estimation problems where a traditional Kalman filter type solution would be faster and much more interpretable.
- menaerus 11mo ago
  Incidentally, I worked on the exact same thing - Kalman filtering for tracking objects in hard real-time systems. And it is not quite as simple as one would think - developing mathematical models for all kinds of different objects that one might wanna track is far from trivial, and it was difficult to model the real-world with more or less simplistic discrete equations. And it didn't work completely reliably so we needed an extra layer of confidence - I don't remember what we used back then but it was yet another algorithm with yet another source of data.
  - srean 11mo ago
    Sigma point filters ? The escalation ladder is usually KF, EKF, unscented KF, sigma point ...
    - menaerus 11mo ago
      I believe it was random forest that we combined with EKF.
  - AlotOfReading 11mo ago
    There's a lot of situations where you genuinely need more sophisticated algorithms. I'm talking about people reaching for those kinds of things before they determine the traditional algorithms won't work.

srean 11mo ago

It does not work that way in the short term. Say you have bet billions as a CEO, CTO, CFO. The decision has already been made. Such a steep price had to come at the cost of many groups and teams and projects in the company. Now is not a time to water plants that offer alternatives. You will have a smoother ride choosing tools that justifies that billion dollar bet.

auxiliarymoose 11mo ago
Decision-making in organizations is definitely a hard problem. I think an uncomfortable reality is that a lot of decisions (technology, strategy, etc.) are not optimal or even rational, but more just an outcome of personal preferences. Even data-driven approaches aren't immune since they depend on the analysis and interpretation of the data (which is subjective).
- srean 11mo ago
  Data informed is good. Purely data driven is a bad idea. After all even in Physics big advances came from thought experiments. Data is one way to reason about a decision, logic and knowledgebase is another way. Both can be very powerful if one retains the humility of fallibility. In organizations one common failure mode is that the organisational level at which decisions are made are not the same levels where the decisions are going to have their effects felt. It's a really difficult problem to solve. Too much decentralisation is also a bad idea. You get the mess of unplanned congested cities.
  - cb321 11mo ago
    For a while now, I've been summarizing the ease with which everything turns into a "Humanity Complete" problem via: "Delegation affords so much, but trust sure is tricky." This has been observed forever in various forms/contexts. Planning & policy people call them "Wicked Problems" (https://en.wikipedia.org/wiki/Wicked_problem https://en.wikipedia.org/wiki/Wicked_problem). The Philosophy of Science one goes by the Demarcation Problem (https://en.wikipedia.org/wiki/Demarcation_problem https://en.wikipedia.org/wiki/Demarcation_problem) { roughly, in the sense that the really hard nugget connects to "trust" }. At least one aspect of all of it is that trust is a little like money/capital and "faking it" is a bit like "stealing". The game theory of it is that since faking is virtually always vastly cheaper there are (eventually) huge incentives to do so, at some point by someone(s). So, almost any kind of trust/delegation structure has a strong pull toward "decay", from knock-off brands to whatever. It just takes a sadly small fraction of Prisoner's Dilemma defectors to ruin things/systems thereof. 2nd law of thermo makes order cost energy and this decay feels like almost an isomorphic (maybe even the same..?) thing. It's not just product/tech enshittification, but that might be yet another special case/example. Anyway, I have no great answers or as some responder to me a while back said, if I did, I'd "have a Nobel and possibly be the first president of the united planet".

DavidPeiffer 11mo ago

>This is one of the reasons I am so skeptical of the current AI hype cycle. There are boring, well-behaved classical solutions for many of the use-cases where fancy ML is pushed today. In 2013 my statistics professor warned that once we are in the real world, "people will come up to you trying to sell fancy machine learning models for big money, though the simple truth is that many problems can be solved better by applying straightforward statistical methods". There has always been the ML hype, but the last couple years are a whole different level.

fuzzfactor 11mo ago

>unintelligible and unreliable options that demand a lot of resources Some options have more persuasive salesmen than others.