7 ms·

Good old-fashioned AI remains viable in spite of the rise of LLMs

82 points by webmaven 3y ago

electroly 3y ago
That is not what people typically mean by "GOFAI," which is a term more commonly understood to refer to classical symbolic AI as practiced by the MIT AI lab starting in the late 60s. From the title I thought they were claiming that the OLD old fashioned AI was still viable--they are not. They're still talking about trained neural network models; they're distinguishing between foundational models and single-task models.
- randcraw 3y ago
  This. Symbolic AI (AKA GOFAI) died about 1985. But task-oriented AI will never die, because the need for intelligent agents with cutting-edge experise in niche skill areas costs too much to embed all the available info on Earth into every foundational AI model, much less, keep it up-to-date. As the old saw says, the only perfect model of the universe is the universe itself.
  - cjauvin 3y ago
    It didn't die per se I would say, it mainly got subsumed into more general programming and problem solving patterns (search, heuristics, A*, that sort of thing). What probably "died" is the dream that those would be sufficient to achieve general intelligence.
  - PartiallyTyped 3y ago
    "Symbolic" AI is very much used wherever you have theorem provers e.g. z3, no?
    - tensor 3y ago
      Yes, and closely related are various forms of constraint solvers, things that LLMS are still fantastically poor at.
      - PartiallyTyped 3y ago
        SMT Solvers are used extensively in certain parts of the industry, especially when working in security-mindful fields.
  - horsh1 3y ago
    Nothing existing only before 1985, that is, before the invention of the open source, was alive in the first place. The code for the classic programs is nowhere to be found. And if found, it is written in some interlisp dead language.
- deleted 3y ago
  [deleted]
- cscurmudgeon 3y ago
  > OLD old fashioned AI was still viable--they are not False. Theorem provers exist and are widely used, sometimes even in deep learning. https://arxiv.org/pdf/2304.10558.pdf https://arxiv.org/pdf/2304.10558.pdf
  - FreakLegion 3y ago
    You misread the statement. > I thought they were claiming that the OLD old fashioned AI was still viable--they are not. Electroly is saying something about the authors' claim, not about the viability of GOFAI.
    - cscurmudgeon 3y ago
      You are right. My bad. Unfortunately, can't edit the comment now :(
- tensor 3y ago
  Good old linear classifiers still solve many tasks close to optimally. Probabilistic AI is not just neural networks.
minimaxir 3y ago
As with most software development, modern AI work is all about knowing your tools and when it's appropriate to use them. If you have tabular data, even good LLMs will have trouble beating a gradient-boost tree algorithm, but if you're working with anything involving text data, you'll save yourself a lot of hassle and likely get better results if you go directly to a pretrained-LLM-generated text embeddings.
- cuuupid 3y ago
  I've found the best LLMs still fall a bit short when it comes to being truly best in class, SOTA in most useful tasks is still dominated by smaller models with focused approaches and innovative loss mechanisms.
  - dartos 3y ago
    Yeah, but you can talk to an LLM. I think the most compelling use for LLMs is “tool use.” Imagine if everyone could write simple automation scripts in literal English. We’re there.
- tensor 3y ago
  Using fasttext as a first pass is probably the best idea for production work.
  - minimaxir 3y ago
    fasttext/word2vec/training-your-own-word-embeddings-from-scratch requires very optimistic assumptions about your data that rarely hold well in practice, such as not having idiosyncratic syntax from user input. I've learnt this the hard way. Transformers-based LLMs with a BPE tokenizer compensate for this well.
- miven 3y ago
  Well, LLMs don't consistently beat purpose-trained BERT-type models just yet, for example in [1] the authors show RoBERTa (fine-tuned for each individual task separately of course) going basically toe-to-toe with GPT-4 on quite a few somewhat conventional NLP tasks, while open-source LLaMA 2 models are getting severely outclassed. [1] https://arxiv.org/abs/2308.10092 https://arxiv.org/abs/2308.10092
synthc 3y ago
There are still tons of domains were GOFAI is very useful, like planning and scheduling, and also in cases where you don't have access to lots of training data or pretrained models.
bugglebeetle 3y ago
If anything, LLMs just heighten their abilities. I can generate synthetic datasets of size and quality for training BERT models and the like that would’ve taken ages before. Similarly for knowledge distillation from large models to small. It’s never been more exciting times for ML.
mejutoco 3y ago
Also. Good oldfashioned statistics remains viable in spite of AI.
deleted 3y ago
[deleted]
yeldarb 3y ago
We’ve been seeing this in vision too. I’m really excited about multimodal LLMs but it seems like they’re going to be most useful for solving new types of (qualitative) problems vs taking over (quantitative) problems where small, fast CNNs have excelled. Though they can also play well together[1] given _creating_ quantitative models is a qualitative problem. [1] https://github.com/autodistill/autodistill https://github.com/autodistill/autodistill
- godelski 3y ago
  > We’ve been seeing this in vision too. It's always amazing to me that __researchers__ still ask if there are uses for GANs, thinking diffusion killed them. I have a hard time taking anyone's claim that they are an expert in synthetic image synthesis but it seems top companies hire them. I see the same thing with ResNets and I just don't get it. There's a tendency to not just railroad, but to actively build it.
phillipcarter 3y ago
One of the most common design patterns for using LLMs, Retrieval Augmented Generation (RAG), is precisely a "good old fashioned AI" problem to solve. The quality of your application depends on the quality of your data, how you organize it, how you understand results you get from real-world usage, which model is used to compute semantic embeddings, how you handle tricky retrieval problems (e.g., "show me bananas" vs. "show me not bananas"), and so on. All this means to me is that the possibilities around what can be built is increasing, as are the needs for people who understand both the "old fashioned" AI world and the new one that we're stepping into.
gremgoth 3y ago
Generative symbolic AI occurs all the time in data integration, when you have to generate new identifiers (e.g. create new targets for foreign keys that don't exist in any source), recursively merge identified entities together along their attributes, and so on. But now that this process is mathematically well-understood it is no longer called "AI" but rather "logic programming".
davesque 3y ago
LLMs themselves are made up of "good old fashioned" components. The public conversation about generative AI seems to assume that it's doing something fundamentally different than what came before. Of course, LLMs are still just deep NNs trained through back propagation to "classify" strings of text into one of many categories corresponding to each possible token. The only thing they do differently is to compute auto-correlation coefficients in each layer and zero some of them out. And of course they take advantage of massive amounts of training data by self-supervising. LLMs are only a slight variation on the good old fashioned models that already existed at the time they showed up.
- cjauvin 3y ago
  What they bring in my mind that is deeply surprising is the realization that from such a simple objective function (predicting the next word) could emerge such a wide array of general capabilities. Before LLMs, neural networks accomplishing "narrow" tasks were understood to be powerful, but not in such a general way.
  - rdedev 3y ago
    My understanding is that at some point the model needs to gain an understanding of how the world works if it needs to know the right set of words that come next to keep reducing perplexity. But I don't think this can scale to AGI but I have nothing to back it up
  - famouswaffles 3y ago
    Predicting the next word as an objective is straightforward, not simple. Think about what it would mean to predict internet scale text accurately. To for instance, know the results of scientific papers before they are uttered. There's a difference between an objective and what you need to do to fulfill it.
- snarkconjecture 3y ago
  "Good old-fashioned AI" (GOFAI) is a term of art that refers to symbolic AI, in contrast to deep NNs. Maybe we need to rename it if deep learning is already "old fashioned"! (EDIT: to be clear, I have no idea what TechCrunch thinks GOFAI means.)
gdiamos 3y ago
LLMs popularized zero-shot learning, or “prompt engineering” which is drastically easier to use and more effective than labeling data. You can also retrofit “prompt engineering” onto good old fashion ML like text classifiers. I wrote a library to do just that here: https://github.com/lamini-ai/llm-classifier https://github.com/lamini-ai/llm-classifier IMO, it’s a short matter of time before this takes over all of what used to be called “deep learning”. Supervised learned, which required Herculean efforts to label big datasets was important to prove that deep learning worked, but we have been engineering it to be easier and more effective ever since and there is no going back.
- malshe 3y ago
  Thanks for sharing. This looks interesting.
tedivm 3y ago
One of my former coworkers at Rad AI always used to joke that most problems people try to solve with AI could be solved better with simple linear regression, and I really do think that input was one of the reasons why that company is doing so well right now. When you're trying to build a product you need to look at the best tools to solve the problems you're working on. Sometimes that truly is an LLM, and other times it isn't. Throwing a technology into things just because it's the cool thing to do at the moment is almost always a mistake though.
- blowski 3y ago
  Following fashions can bring you a bit of short term cash, but it won’t build a long-term sustainable business.
- varelse 3y ago
  [dead]
- chaostheory 3y ago
  That makes sense, but we often ignore and overlook the marketing side for both attracting investors and customers. If you need runway, it’s “AI”
- seeknotfind 3y ago
  Ah yes. Customer service. Who needs LLM? Just use a simple linear regression.
  - tedivm 3y ago
    That's just about the silliest interpretation of my comment that could have been made.
adastra22 3y ago
Article title is misleading. It is not about "Good old-fashioned AI" (GOFAI) at all. That's a term of art referring to symbolic AI like planning and explicit inference. All the stuff covered in Norvig's classic "AI: A modern approach."
- deleted 3y ago
  [deleted]

82 points by webmaven 3y ago

That is not what people typically mean by "GOFAI," which is a term more commonly understood to refer to classical symbolic AI as practiced by the MIT AI lab starting in the late 60s. From the title I thought they were claiming that the OLD old fashioned AI was still viable--they are not. They're still talking about trained neural network models; they're distinguishing between foundational models and single-task models.

randcraw 3y ago
This. Symbolic AI (AKA GOFAI) died about 1985. But task-oriented AI will never die, because the need for intelligent agents with cutting-edge experise in niche skill areas costs too much to embed all the available info on Earth into every foundational AI model, much less, keep it up-to-date. As the old saw says, the only perfect model of the universe is the universe itself.
- cjauvin 3y ago
  It didn't die per se I would say, it mainly got subsumed into more general programming and problem solving patterns (search, heuristics, A*, that sort of thing). What probably "died" is the dream that those would be sufficient to achieve general intelligence.
- PartiallyTyped 3y ago
  "Symbolic" AI is very much used wherever you have theorem provers e.g. z3, no?
  - tensor 3y ago
    Yes, and closely related are various forms of constraint solvers, things that LLMS are still fantastically poor at.
    - PartiallyTyped 3y ago
      SMT Solvers are used extensively in certain parts of the industry, especially when working in security-mindful fields.
- horsh1 3y ago
  Nothing existing only before 1985, that is, before the invention of the open source, was alive in the first place. The code for the classic programs is nowhere to be found. And if found, it is written in some interlisp dead language.
deleted 3y ago
[deleted]
cscurmudgeon 3y ago
> OLD old fashioned AI was still viable--they are not False. Theorem provers exist and are widely used, sometimes even in deep learning. https://arxiv.org/pdf/2304.10558.pdf https://arxiv.org/pdf/2304.10558.pdf
- FreakLegion 3y ago
  You misread the statement. > I thought they were claiming that the OLD old fashioned AI was still viable--they are not. Electroly is saying something about the authors' claim, not about the viability of GOFAI.
  - cscurmudgeon 3y ago
    You are right. My bad. Unfortunately, can't edit the comment now :(
tensor 3y ago
Good old linear classifiers still solve many tasks close to optimally. Probabilistic AI is not just neural networks.

minimaxir 3y ago

As with most software development, modern AI work is all about knowing your tools and when it's appropriate to use them. If you have tabular data, even good LLMs will have trouble beating a gradient-boost tree algorithm, but if you're working with anything involving text data, you'll save yourself a lot of hassle and likely get better results if you go directly to a pretrained-LLM-generated text embeddings.

cuuupid 3y ago
I've found the best LLMs still fall a bit short when it comes to being truly best in class, SOTA in most useful tasks is still dominated by smaller models with focused approaches and innovative loss mechanisms.
- dartos 3y ago
  Yeah, but you can talk to an LLM. I think the most compelling use for LLMs is “tool use.” Imagine if everyone could write simple automation scripts in literal English. We’re there.
tensor 3y ago
Using fasttext as a first pass is probably the best idea for production work.
- minimaxir 3y ago
  fasttext/word2vec/training-your-own-word-embeddings-from-scratch requires very optimistic assumptions about your data that rarely hold well in practice, such as not having idiosyncratic syntax from user input. I've learnt this the hard way. Transformers-based LLMs with a BPE tokenizer compensate for this well.
miven 3y ago
Well, LLMs don't consistently beat purpose-trained BERT-type models just yet, for example in [1] the authors show RoBERTa (fine-tuned for each individual task separately of course) going basically toe-to-toe with GPT-4 on quite a few somewhat conventional NLP tasks, while open-source LLaMA 2 models are getting severely outclassed. [1] https://arxiv.org/abs/2308.10092 https://arxiv.org/abs/2308.10092

synthc 3y ago

There are still tons of domains were GOFAI is very useful, like planning and scheduling, and also in cases where you don't have access to lots of training data or pretrained models.

bugglebeetle 3y ago

If anything, LLMs just heighten their abilities. I can generate synthetic datasets of size and quality for training BERT models and the like that would’ve taken ages before. Similarly for knowledge distillation from large models to small. It’s never been more exciting times for ML.

mejutoco 3y ago

Also. Good oldfashioned statistics remains viable in spite of AI.

deleted 3y ago

[deleted]

yeldarb 3y ago

We’ve been seeing this in vision too. I’m really excited about multimodal LLMs but it seems like they’re going to be most useful for solving new types of (qualitative) problems vs taking over (quantitative) problems where small, fast CNNs have excelled. Though they can also play well together[1] given _creating_ quantitative models is a qualitative problem. [1] https://github.com/autodistill/autodistill https://github.com/autodistill/autodistill

godelski 3y ago
> We’ve been seeing this in vision too. It's always amazing to me that __researchers__ still ask if there are uses for GANs, thinking diffusion killed them. I have a hard time taking anyone's claim that they are an expert in synthetic image synthesis but it seems top companies hire them. I see the same thing with ResNets and I just don't get it. There's a tendency to not just railroad, but to actively build it.

phillipcarter 3y ago

One of the most common design patterns for using LLMs, Retrieval Augmented Generation (RAG), is precisely a "good old fashioned AI" problem to solve. The quality of your application depends on the quality of your data, how you organize it, how you understand results you get from real-world usage, which model is used to compute semantic embeddings, how you handle tricky retrieval problems (e.g., "show me bananas" vs. "show me not bananas"), and so on. All this means to me is that the possibilities around what can be built is increasing, as are the needs for people who understand both the "old fashioned" AI world and the new one that we're stepping into.

gremgoth 3y ago

Generative symbolic AI occurs all the time in data integration, when you have to generate new identifiers (e.g. create new targets for foreign keys that don't exist in any source), recursively merge identified entities together along their attributes, and so on. But now that this process is mathematically well-understood it is no longer called "AI" but rather "logic programming".

davesque 3y ago

LLMs themselves are made up of "good old fashioned" components. The public conversation about generative AI seems to assume that it's doing something fundamentally different than what came before. Of course, LLMs are still just deep NNs trained through back propagation to "classify" strings of text into one of many categories corresponding to each possible token. The only thing they do differently is to compute auto-correlation coefficients in each layer and zero some of them out. And of course they take advantage of massive amounts of training data by self-supervising. LLMs are only a slight variation on the good old fashioned models that already existed at the time they showed up.

cjauvin 3y ago
What they bring in my mind that is deeply surprising is the realization that from such a simple objective function (predicting the next word) could emerge such a wide array of general capabilities. Before LLMs, neural networks accomplishing "narrow" tasks were understood to be powerful, but not in such a general way.
- rdedev 3y ago
  My understanding is that at some point the model needs to gain an understanding of how the world works if it needs to know the right set of words that come next to keep reducing perplexity. But I don't think this can scale to AGI but I have nothing to back it up
- famouswaffles 3y ago
  Predicting the next word as an objective is straightforward, not simple. Think about what it would mean to predict internet scale text accurately. To for instance, know the results of scientific papers before they are uttered. There's a difference between an objective and what you need to do to fulfill it.
snarkconjecture 3y ago
"Good old-fashioned AI" (GOFAI) is a term of art that refers to symbolic AI, in contrast to deep NNs. Maybe we need to rename it if deep learning is already "old fashioned"! (EDIT: to be clear, I have no idea what TechCrunch thinks GOFAI means.)

gdiamos 3y ago

LLMs popularized zero-shot learning, or “prompt engineering” which is drastically easier to use and more effective than labeling data. You can also retrofit “prompt engineering” onto good old fashion ML like text classifiers. I wrote a library to do just that here: https://github.com/lamini-ai/llm-classifier https://github.com/lamini-ai/llm-classifier IMO, it’s a short matter of time before this takes over all of what used to be called “deep learning”. Supervised learned, which required Herculean efforts to label big datasets was important to prove that deep learning worked, but we have been engineering it to be easier and more effective ever since and there is no going back.

malshe 3y ago
Thanks for sharing. This looks interesting.

tedivm 3y ago

One of my former coworkers at Rad AI always used to joke that most problems people try to solve with AI could be solved better with simple linear regression, and I really do think that input was one of the reasons why that company is doing so well right now. When you're trying to build a product you need to look at the best tools to solve the problems you're working on. Sometimes that truly is an LLM, and other times it isn't. Throwing a technology into things just because it's the cool thing to do at the moment is almost always a mistake though.

blowski 3y ago
Following fashions can bring you a bit of short term cash, but it won’t build a long-term sustainable business.
varelse 3y ago
[dead]
chaostheory 3y ago
That makes sense, but we often ignore and overlook the marketing side for both attracting investors and customers. If you need runway, it’s “AI”
seeknotfind 3y ago
Ah yes. Customer service. Who needs LLM? Just use a simple linear regression.
- tedivm 3y ago
  That's just about the silliest interpretation of my comment that could have been made.

adastra22 3y ago

Article title is misleading. It is not about "Good old-fashioned AI" (GOFAI) at all. That's a term of art referring to symbolic AI like planning and explicit inference. All the stuff covered in Norvig's classic "AI: A modern approach."

deleted 3y ago
[deleted]