15 ms·

Generative AI is overrated, long live old-school AI

227 points by Buhljingo 4y ago

sposeray 4y ago
[dead]
goldenkey 4y ago
When the generative model is autoregressive (autocomplete), it can easily be used as a predictor. All of the state of the art language models are tested against multiple choice exams and other types of prediction tasks. In fact, it's how they are trained...masking - https://www.microsoft.com/en-us/research/blog/mpnet-combines-strengths-of-masked-and-permuted-language-modeling-for-language-understanding https://www.microsoft.com/en-us/research/blog/mpnet-combines... For example: "Multiple-choice questions in 57 subjects (professional & academic)" - https://openai.com/research/gpt-4 https://openai.com/research/gpt-4
- k8si 4y ago
  For GPT4: "Pricing is $0.03 per 1,000 “prompt” tokens (about 750 words) and $0.06 per 1,000 “completion” tokens (again, about 750 words)." Meanwhile, there are off-shelf models that you can train very efficiently, on relevant data, privately, and you can run these on your own infrastructure. Yes, GPT4 is probably great at all the benchmark tasks, but models have been great at all the open benchmark tasks for a long time. That's why they have to keep making harder tasks. Depending on what you actually want to do with LMs, GPT4 might lose to a BERTish model in a cost-benefit analysis--especially given that (in my experience), the hard part of ML is still getting data/QA/infrastructure aligned with whatever it is you want to do with the ML. (At least at larger companies, maybe it's different at startups.)
  - potatoman22 4y ago
    This is true, but the humans to develop the non-LLM solution are expensive and the OpenAI API is easy.
- croes 4y ago
  Being good at standardized tests isn't really a good measure. What happens with completely new questions from totally different subject. The generative model will produce nonsense.
  - deleted 4y ago
    [deleted]
all2 4y ago
From TFA: TLDR; Don't be dazzled by generative AI's creative charm! Predictive AI, though less flashy, remains crucial for solving real-world challenges and unleashing AI's true potential. By merging the powers of both AI types and closing the prototype-to-production gap, we'll accelerate the AI revolution and transform our world. Keep an eye on both these AI stars to witness the future unfold.
jasfi 4y ago
[flagged]
- ulrikhansen54 4y ago
  There's about ~10% point improvement left (i.e, from 80% to 90%) before it starts to stagnate. We've seen the same with predictive models benchmarked on ImageNet et. al.
  - whiplash451 4y ago
    By stagnate, you mean beating humankind at the task, right? :)
    - dimitrios1 4y ago
      It's funny to me we look at GPT4 scoring high on all these tests and think it's worth anything when educators and a lot of us here have been lamenting the standardized tests since Bush made it a preeminent feature of our country's education system. They are not a good measure of intelligence. They measure how well you can take a test.
      - dahdum 4y ago
        > They are not a good measure of intelligence. They measure how well you can take a test. The tests aren't trying to measure intelligence, but rather whether you've learned the material.
        dimitrios1 4y ago
        Again, they are horrible at that.
      - kenjackson 4y ago
        Funny -- I literally had someone tell me this same thing this morning... but the exact same guy last week was arguing with me against the reduced importance of these same tests for college admissions. Last week he was arguing how critical these tests were for the college admissions process, but this morning the same tests are basically worthless. Not saying you hold the same opinions -- but I wouldn't be surprised if people's take on these tests is more about what is convenient for their psyche than any actual principled position.
draxil 4y ago
We are all struck with the novelty of generative AI, it needs time to settle. People will throw the universe at the wall and see what really sticks. To my mind generative AI is great at finding needles in the haystack of stuff we already know. Of course it just as often gives you a fake needle right now, just to see if you notice. On the other hand "traditional"/predictive AI is often better at the things we don't already know or understand.
version_five 4y ago
Seems like the person who wrote the blog works in "classical" deep learning. So do I, so here's the fairest take I can come up with: "AI" has for recent memory been a marketing term anyway. Deep learning and variations have had a good run at being what people mean when they refer to AI, probably overweighting towards big convolution based computer vision models. Now, "AI" in people's minds means generative models. That's it, it doesn't mean generative models are replacing CNNs, just like CNNs don't replace SVMs or regression or whatever. It's just that pop culture has fallen in love with something else.
- JohnFen 4y ago
  Spot on. I work with deep learning systems in industrial control, and generative models are simply ill-suited for this sort of work. Wrong tool for the job. But neither the traditional nor generative models are "AI" in the sense that normal people think when they hear "AI".
  - fakedang 4y ago
    I'm curious about your work, because I worked on something similar during my grad school. What kind of applications in industry do you use deep learning systems for? Process control?
    - JohnFen 4y ago
      Yes, process control. It's used in coordination with vision systems to analyze work pieces, determine the best way of processing them, and direct other machinery how to do that processing.
      - fakedang 4y ago
        That's cool. If you don't mind me asking, would you have any shallow level stuff that I could read on about this? Even a website or a blog post would be great. In my grad school, we were working on something similar - using computer vision to analyze reactor flows to then change process variables. The results would be fed back into the system for RL. Too bad the project sorta froze after I graduated.
DeathArrow 4y ago
>investors have become only interested in companies building generative AI, relegating those working on predictive models to “old school” AI. If that is the definition of old school AI, I wonder how symbolic AI should be named.
- snapcaster 4y ago
  how about "useless with no successes of note" AI?
  - deleted 4y ago
    [deleted]
  - TuringTest 4y ago
    I hope you've never used the power grid or parcel shipping, as those are heavily optimized using symbolic AI.
  - qorrect 4y ago
    What ? We all use it everyday, it's just that as soon as the problem was solved with 'old AI', everyone forgot it was an AI problem.
- TuringTest 4y ago
  > If that is the definition of old school AI It is not. Symbolic, deductive reasoning engines have the same claim to being old-school AI as predictive statistic models.
pretendscholar 4y ago
I’m not sure I understand a definition of AI that doesn’t include the ability to generate things.
- croes 4y ago
  The point is that AI is more than just generating more of the same data it was trained on.
- WoodenChair 4y ago
  > I’m not sure I understand a definition of AI that doesn’t include the ability to generate things. It depends how you define "generate." For example, is software that controls a robot arm generating anything? I guess it's generating the movements of the arm. But when people use the term "generative" with regards to machine learning models right now, they generally mean content—e.g. text or images for consumption.
  - yunwal 4y ago
    Generative has a more technical meaning than that. Generative AI is essentially the opposite of a classifier. You give it a prompt that could mean many different things, and it gives you one of those things. A robotic arm could use generative AI, because there are many different sets of electrical signals that would result in success for, say, catching a ball. Classification is an example of a non-generative AI in that there is only 1 correct answer, but it still requires machine learning to acquire the classification function.
- TuringTest 4y ago
  You can use AI to validate things, i.e. to check that they conform to some specification. You may twist the language to say that they are generating a list of validations and errors, but even then it's definitely a different use case than merely creating new items.
whiplash451 4y ago
The author might be missing the fact that generative models can be used for "old-school" prediction tasks, with quite outstanding results. Their power does not only lie in their ability to _generate_ new data, but to _model_ existing data.
- jasonjmcghee 4y ago
  The biggest issue with using them in this way is how alien the failure modes are. Interpretable models with transparent loss functions are easy to grok. How LLMs might fail on a classic task is (afaict right now) difficult to predict.
  - whiplash451 4y ago
    What is not transparent in the cross-entropy loss used in a large number of deep nets?
    - jasonjmcghee 4y ago
      I think there was a breakdown in communication here. If I train a classic deep net as a classifier and there are 5 possible classes, it will only ever output those 5 classes (unless there's a bug). With ChatGPT, for example, it could theoretically decide to introduce a 6th class - what I would call an alien failure mode, even if you explicitly told it not to. I think formally / provably constraining the output of LLM APIs will help mitigate these issues, rather than needing to use an embedding API / use the LLM as a featurizer and train another model on top of it.
      - calf 4y ago
        Formal proof is problematic because English has no formal specification. Some people are working on this, it's a nascent area bringing formal methods (model checking) to neural network models of computation. But it's an interesting fundamental issue that arises there, if you can't even specify the design intentions then how do you prove anything about it.
- deleted 4y ago
peter_retief 4y ago
"So has generative AI been overhyped? Not exactly. Having generative models capable of delivering value is an exciting development. For the first time, people can interact with AI systems that don’t just automate but create an activity of which only humans were previously capable." Good answer but I feel that most users/people do not understand the difference between generative and predictive machine learning and that will probably cause unpredictable failures and false flags. So yes it has been overhyped in my opinion
- Xelynega 4y ago
  I think the issue is more with people marketing/talking about them as "AI". When I think AI I think of something like Skynet. I would assume something like Skynet would be good at chess, able to generate new text, and synthesize new images. I think when shown novel algorithms that can do those things and told by the people selling the algorithms that they are "AI", it's hard to disagree since they quack like an AI so it's easy to accept that these are the same "artificial intelligence" concept in our brains which we previously only had examples of from fiction. Basically I think it's overhyped by the use of the term "AI" and how easy we are to accept it generally. Some aspect of them being generative models could have been the term used to market/describe them, but instead a much broader term is used.
- kenjackson 4y ago
  IMO, it has been underhyped. We're seeing things with LLMs that a decade ago I'd say was multiple decades out, if not more. We're just years into generative approaches. And I think we'll more combinations of methods used in the future. The goal of AI has never been to build an all knowing perfect system. It has also never been to replicate the way the human brain works. But its been to build an artificial system that can learn -- and AGI specifically to be able to give the appearance of human learning. I feel like we've turned this corner where the question now is, "Can we build something that knows everything that has been documented and can also synthesize and infer all of that data at a level of a very smart human". The fact that this has become the new bar is IMO one of the biggest tech changes in history. Not the biggest, but up there.
seydor 4y ago
I m not sure it's overrated, but the concerns are very real. We love the model because it speaks our language as if it's "one of us", but this may be deceiving, and the complete lack of model for truth is disturbing. Making silly poems is fun but the real uses are in medicine and biology, fields that are so complex that they are probably impenetrable to the human mind. Can Reinforcement learning alone create a model for the truth? The Transformer does not seem to have one, it only works with syntax and referencing. How much % of truthfulness can we achieve, and is it good enough for scientific applications? If a blocker is found in the interface between the model and reality, it will be a huge disappointment
- aaroninsf 4y ago
  I am not so sure, there seems to be accumulating evidence that "finding the optimal solutions" means (requires) building a world model. Whether it's consistent with ground truth probably depends on what you mean by ground truth. Given the hypothesis that the optimal solution for deep learning presented with a given training set, is to represent (simulate) the formal systemic relationships that generated that set, by "modeling" such relationships (or discovering non-lossy optimized simplifications), I believe an implicit corollary, that the fidelity of simulation is only bounded by the information in the original data. Prediction: a big enough network, well enough trained, is capable of simulating with arbitrary fidelity, an arbitrarily complex system, to the point that lack of fidelity hits a noise floor. The testable bit of interest being whether such simulations predict novel states and outcomes (real world behavior) well enough. I don't see why they shouldn't, but the X-factor would seem to be the resolution and comprehensiveness of our training data. I can imagine toy domains like SHRDLU which are simple enough that we should be able to build large models well enough already to "model" them and tease this sort of speculation experimentally. I hope (assume) this is already being done...
  - JohnFen 4y ago
    > there seems to be accumulating evidence that "finding the optimal solutions" means (requires) building a world model. Was this ever in doubt? This has been the case forever (even before "AI"), and I thought it was well-established. The fidelity of the model is the core problem. What "AI" is really providing is a shortcut that allows the creation of better models. But no model can ever be perfect, because the value of them is that they're an abstraction. As the old truism goes, a perfect map of a terrain would necessarily be indistinguishable from the actual terrain.
kyleyeats 4y ago
I'm working on an old-school AI personal project right now. I don't know how long that lasts. The generative stuff is more and more tempting. It rewards the horrible micromanager in me like nothing else.
nathias 4y ago
after the era of low hanging fruits of generative AI will be over I'm sure there will be a return to other approaches
GuB-42 4y ago
Is there a fundamental difference? I mean, the only thing GPT does is predict the next word, which makes it not so different from a compression algorithm. And diffusion models (the image generating stuff) are essentially fancy denoisers. Depending on how you assemble the big building blocks, you get generation or you get prediction.
- deleted 4y ago
  [deleted]
- sweezyjeezy 4y ago
  Depends how far you take the word 'fundamental', on the one hand yeah most DL systems are trying to predict something, and they generally have some concept of compression built in. But in terms of the steps to curate a dataset, train, test, iterate and actually use the model for a given end goal - they are pretty fundamentally different.
  - sharemywin 4y ago
    I think the thing is though in Large multi models you give it all the data and test it against everything. And it generally does better across most of the benchmarks.
    - sweezyjeezy 4y ago
      That depends entirely on the use-case - for example if you wanted to build an AI to operate a self-driving car, just training on unlabelled data scraped from the internet is only going to get you so far. It doesn't learn how to do EVERYTHING (not yet at least).
- baq 4y ago
  GPT-3.5 is not a Markov chain, this is trivially true. While ‘predicts the next word’ is true, the mechanism of it is of interest and that is most certainly not trivial.
EGreg 4y ago
Yes! Just like HN is anti blockchain but super pro AI. It seems most applications of generative AI at scale will havd a huge negative for society, far worse than anything blockchain could have brought about.
- deleted 4y ago
  [deleted]
uoaei 4y ago
Generative methods per se are pretty sick and dope, and are still useful for many things beyond art generation.
kmeisthax 4y ago
People calling neural-net classifiers "old-school" AI confused me. For a second I thought they were talking about the really old "expert systems" with everything being a pile of hard-coded rules.
- 01100011 4y ago
  It still feels like there's a place for these rule based systems(Prolog?) to at least place some constraints on the output of non-deterministic, generative AI. If nothing else, have a generative AI generate the ruleset so you have some explicit rules you can audit from time to time.
  - theLiminator 4y ago
    Yeah, i think one potential way to use blackbox ai in newer systems is having guardrails that are validated as safe (but perhaps non-optimal) and ensuring that the ai takes action within that sample space. Obviously this is hard problem, but might open the doors for policies (in self-driving cars, for example) to be entirely ai driven.
    - earthboundkid 4y ago
      Obviously the solution is to get the LLM to output Prolog. Give it positive feedback if the Prolog compiles. :-)
      - 01100011 4y ago
        A friend of mine was just telling me how he asked GPT-3 to write a simple program in Prolog and it seemed to get it right. He didn't try compiling it, but he has enough experience w/ Prolog to say that it was more or less correct. I'm pretty cynical on LLMs(i.e. they're not intelligent and won't take all our jobs soon), but am coming around on their importance and capabilities.
  - lincpa 4y ago
    [dead]
glitchc 4y ago
I see and I hear: "Don't be dazzled by AI computer vision's creative charm! Classical computer vision, though less flashy, remains crucial for solving real-world challenges and unleashing computer vision's true potential." Meant for those in classical computer vision before ML ate the field.
- potatoman22 4y ago
  If by classical CV they mean image processing and machine vision, I would say that's still true.
jedberg 4y ago
The real innovation will come one someone uses a Generative AI to make something, and then use a predictive AI to rate it's accuracy, making it go again until it passes the predictive AI. Basically a form of adversarial training/generation.
- arrow7000 4y ago
  Isn't this exactly how GANs work already?
  - jedberg 4y ago
    Yes. But from I've seen no one has applied it to the latest Generative AIs.
    - arrow7000 4y ago
      Maybe an adversarial approach was used in training these models in the first place?
      - sharemywin 4y ago
        It was they were' trained using reinforcement learning with human feedback to create the critic.
        jedberg 4y ago
        I hadn't thought about human feedback being an adversarial system, but I guess that makes sense, since it's basically a classifier saying "you got this wrong".
    - dereg 4y ago
      I’m pretty sure Anthropic’s Claude is doing that. https://scale.com/blog/chatgpt-vs-claude https://scale.com/blog/chatgpt-vs-claude
- ChikkaChiChi 4y ago
ElijahLynn 4y ago
This article could be improved by starting off stating what some examples of Predictive AI is, as they did with Generative AI.
redox99 4y ago
I wonder how good multimodal GPT4 is at ImageNet. (You give it the image and prompt it with the 1000 classes and ask it which one the image belongs to). I'm surprised ClosedAI didn't include this kind of benchmark. I guess it doesn't do too well?
- sharemywin 4y ago
  Here's something on Clip https://www.pinecone.io/learn/zero-shot-image-classification-clip/ https://www.pinecone.io/learn/zero-shot-image-classification...
patrulek 4y ago
Old-school, huh. The skynet is closer than we think i guess.
efitz 4y ago
I think that 100% of the actually useful use cases for generative AI could be described in two words: “supervised autocomplete”.
- orangecat 4y ago
  That's not wrong, but an ideal autocompleter is a near-omniscient superintelligence. "The optimal approach to curing Alzheimer's is ______". "The proof of the Riemann hypothesis is as follows: ______". "The best way for me to improve my life is _______".
  - kneebonian 4y ago
    I think the big difference is just being an Autocompleter is less concerned with generating something that is truthful, as in reflects the real world as we understand it described by physics, vs simply spitting out something that sounds good. Although we do have a litmus test in asking it "What is the meaning of life the universe and everything?"
    - gweinberg 4y ago
      Yes, exactly. An autocompleter is saying what the next words probably would be, not what it should be. It's like a chess program that tries to find the most likely move that a huan would make in the position rather than the best move.
      - efitz 4y ago
        That’s why I said “supervised” - in other words, someone competent in the domain and context is examining the output and correcting or discarding as necessary before use. “Unsupervised generative AI” is useless IMO.
wslh 4y ago
I would add that there are logic deductive and constraint systems that are more classical and work in some areas. It is not about a single method but we should he aware that AI is a superset of what we see.
phonebucket 4y ago
There is much more to generative models than building out language models and image models. Generative models are about characterising probability distributions. If you ever predict more than just the average of something using data, then you are doing generative modelling. The difference between generative modelling and predictive modelling is similar to the difference between stochastic modelling and deterministic modelling in the traditional applied mathematical sciences. Both have their place. Neither is overrated. Grab the best tool for the job.
tolciho 4y ago
As stated by John McCarthy--"I invented [AI] because we had to do something when we were trying to get money for a summer study" (the Lighthill debate)--this article passes the AI sniff test, or "please remember us predictive AI folks when you go to dole out your money" as all that is solid melts into PR.
kulkarniankita 4y ago
I love it but also waiting for the hype to settle down.
f0ld 4y ago
Or could it be possible that it was always going to end up like a black box it seems is it not? We will never truly understand the inner workings while it solves every problems that can be numbered by which it couldn't be solved with hardline algorithms previously. It's literally calling higher dimensional egrigores for answers or some blood magic Genie.

227 points by Buhljingo 4y ago

[dead]

When the generative model is autoregressive (autocomplete), it can easily be used as a predictor. All of the state of the art language models are tested against multiple choice exams and other types of prediction tasks. In fact, it's how they are trained...masking - https://www.microsoft.com/en-us/research/blog/mpnet-combines-strengths-of-masked-and-permuted-language-modeling-for-language-understanding https://www.microsoft.com/en-us/research/blog/mpnet-combines... For example: "Multiple-choice questions in 57 subjects (professional & academic)" - https://openai.com/research/gpt-4 https://openai.com/research/gpt-4

k8si 4y ago
For GPT4: "Pricing is $0.03 per 1,000 “prompt” tokens (about 750 words) and $0.06 per 1,000 “completion” tokens (again, about 750 words)." Meanwhile, there are off-shelf models that you can train very efficiently, on relevant data, privately, and you can run these on your own infrastructure. Yes, GPT4 is probably great at all the benchmark tasks, but models have been great at all the open benchmark tasks for a long time. That's why they have to keep making harder tasks. Depending on what you actually want to do with LMs, GPT4 might lose to a BERTish model in a cost-benefit analysis--especially given that (in my experience), the hard part of ML is still getting data/QA/infrastructure aligned with whatever it is you want to do with the ML. (At least at larger companies, maybe it's different at startups.)
- potatoman22 4y ago
  This is true, but the humans to develop the non-LLM solution are expensive and the OpenAI API is easy.
croes 4y ago
Being good at standardized tests isn't really a good measure. What happens with completely new questions from totally different subject. The generative model will produce nonsense.
- deleted 4y ago
  [deleted]

all2 4y ago

From TFA: TLDR; Don't be dazzled by generative AI's creative charm! Predictive AI, though less flashy, remains crucial for solving real-world challenges and unleashing AI's true potential. By merging the powers of both AI types and closing the prototype-to-production gap, we'll accelerate the AI revolution and transform our world. Keep an eye on both these AI stars to witness the future unfold.

jasfi 4y ago

[flagged]

ulrikhansen54 4y ago
There's about ~10% point improvement left (i.e, from 80% to 90%) before it starts to stagnate. We've seen the same with predictive models benchmarked on ImageNet et. al.
- whiplash451 4y ago
  By stagnate, you mean beating humankind at the task, right? :)
  - dimitrios1 4y ago
    It's funny to me we look at GPT4 scoring high on all these tests and think it's worth anything when educators and a lot of us here have been lamenting the standardized tests since Bush made it a preeminent feature of our country's education system. They are not a good measure of intelligence. They measure how well you can take a test.
    - dahdum 4y ago
      > They are not a good measure of intelligence. They measure how well you can take a test. The tests aren't trying to measure intelligence, but rather whether you've learned the material.
      - dimitrios1 4y ago
        Again, they are horrible at that.
    - kenjackson 4y ago
      Funny -- I literally had someone tell me this same thing this morning... but the exact same guy last week was arguing with me against the reduced importance of these same tests for college admissions. Last week he was arguing how critical these tests were for the college admissions process, but this morning the same tests are basically worthless. Not saying you hold the same opinions -- but I wouldn't be surprised if people's take on these tests is more about what is convenient for their psyche than any actual principled position.

draxil 4y ago

We are all struck with the novelty of generative AI, it needs time to settle. People will throw the universe at the wall and see what really sticks. To my mind generative AI is great at finding needles in the haystack of stuff we already know. Of course it just as often gives you a fake needle right now, just to see if you notice. On the other hand "traditional"/predictive AI is often better at the things we don't already know or understand.

version_five 4y ago

Seems like the person who wrote the blog works in "classical" deep learning. So do I, so here's the fairest take I can come up with: "AI" has for recent memory been a marketing term anyway. Deep learning and variations have had a good run at being what people mean when they refer to AI, probably overweighting towards big convolution based computer vision models. Now, "AI" in people's minds means generative models. That's it, it doesn't mean generative models are replacing CNNs, just like CNNs don't replace SVMs or regression or whatever. It's just that pop culture has fallen in love with something else.

JohnFen 4y ago
Spot on. I work with deep learning systems in industrial control, and generative models are simply ill-suited for this sort of work. Wrong tool for the job. But neither the traditional nor generative models are "AI" in the sense that normal people think when they hear "AI".
- fakedang 4y ago
  I'm curious about your work, because I worked on something similar during my grad school. What kind of applications in industry do you use deep learning systems for? Process control?
  - JohnFen 4y ago
    Yes, process control. It's used in coordination with vision systems to analyze work pieces, determine the best way of processing them, and direct other machinery how to do that processing.
    - fakedang 4y ago
      That's cool. If you don't mind me asking, would you have any shallow level stuff that I could read on about this? Even a website or a blog post would be great. In my grad school, we were working on something similar - using computer vision to analyze reactor flows to then change process variables. The results would be fed back into the system for RL. Too bad the project sorta froze after I graduated.

DeathArrow 4y ago

>investors have become only interested in companies building generative AI, relegating those working on predictive models to “old school” AI. If that is the definition of old school AI, I wonder how symbolic AI should be named.

snapcaster 4y ago
how about "useless with no successes of note" AI?
- deleted 4y ago
  [deleted]
- TuringTest 4y ago
  I hope you've never used the power grid or parcel shipping, as those are heavily optimized using symbolic AI.
- qorrect 4y ago
  What ? We all use it everyday, it's just that as soon as the problem was solved with 'old AI', everyone forgot it was an AI problem.
TuringTest 4y ago
> If that is the definition of old school AI It is not. Symbolic, deductive reasoning engines have the same claim to being old-school AI as predictive statistic models.

pretendscholar 4y ago

I’m not sure I understand a definition of AI that doesn’t include the ability to generate things.

croes 4y ago
The point is that AI is more than just generating more of the same data it was trained on.
WoodenChair 4y ago
> I’m not sure I understand a definition of AI that doesn’t include the ability to generate things. It depends how you define "generate." For example, is software that controls a robot arm generating anything? I guess it's generating the movements of the arm. But when people use the term "generative" with regards to machine learning models right now, they generally mean content—e.g. text or images for consumption.
- yunwal 4y ago
  Generative has a more technical meaning than that. Generative AI is essentially the opposite of a classifier. You give it a prompt that could mean many different things, and it gives you one of those things. A robotic arm could use generative AI, because there are many different sets of electrical signals that would result in success for, say, catching a ball. Classification is an example of a non-generative AI in that there is only 1 correct answer, but it still requires machine learning to acquire the classification function.
TuringTest 4y ago
You can use AI to validate things, i.e. to check that they conform to some specification. You may twist the language to say that they are generating a list of validations and errors, but even then it's definitely a different use case than merely creating new items.

whiplash451 4y ago

The author might be missing the fact that generative models can be used for "old-school" prediction tasks, with quite outstanding results. Their power does not only lie in their ability to _generate_ new data, but to _model_ existing data.

jasonjmcghee 4y ago
The biggest issue with using them in this way is how alien the failure modes are. Interpretable models with transparent loss functions are easy to grok. How LLMs might fail on a classic task is (afaict right now) difficult to predict.
- whiplash451 4y ago
  What is not transparent in the cross-entropy loss used in a large number of deep nets?
  - jasonjmcghee 4y ago
    I think there was a breakdown in communication here. If I train a classic deep net as a classifier and there are 5 possible classes, it will only ever output those 5 classes (unless there's a bug). With ChatGPT, for example, it could theoretically decide to introduce a 6th class - what I would call an alien failure mode, even if you explicitly told it not to. I think formally / provably constraining the output of LLM APIs will help mitigate these issues, rather than needing to use an embedding API / use the LLM as a featurizer and train another model on top of it.
    - calf 4y ago
      Formal proof is problematic because English has no formal specification. Some people are working on this, it's a nascent area bringing formal methods (model checking) to neural network models of computation. But it's an interesting fundamental issue that arises there, if you can't even specify the design intentions then how do you prove anything about it.
deleted 4y ago

peter_retief 4y ago

"So has generative AI been overhyped? Not exactly. Having generative models capable of delivering value is an exciting development. For the first time, people can interact with AI systems that don’t just automate but create an activity of which only humans were previously capable." Good answer but I feel that most users/people do not understand the difference between generative and predictive machine learning and that will probably cause unpredictable failures and false flags. So yes it has been overhyped in my opinion

Xelynega 4y ago
I think the issue is more with people marketing/talking about them as "AI". When I think AI I think of something like Skynet. I would assume something like Skynet would be good at chess, able to generate new text, and synthesize new images. I think when shown novel algorithms that can do those things and told by the people selling the algorithms that they are "AI", it's hard to disagree since they quack like an AI so it's easy to accept that these are the same "artificial intelligence" concept in our brains which we previously only had examples of from fiction. Basically I think it's overhyped by the use of the term "AI" and how easy we are to accept it generally. Some aspect of them being generative models could have been the term used to market/describe them, but instead a much broader term is used.
kenjackson 4y ago
IMO, it has been underhyped. We're seeing things with LLMs that a decade ago I'd say was multiple decades out, if not more. We're just years into generative approaches. And I think we'll more combinations of methods used in the future. The goal of AI has never been to build an all knowing perfect system. It has also never been to replicate the way the human brain works. But its been to build an artificial system that can learn -- and AGI specifically to be able to give the appearance of human learning. I feel like we've turned this corner where the question now is, "Can we build something that knows everything that has been documented and can also synthesize and infer all of that data at a level of a very smart human". The fact that this has become the new bar is IMO one of the biggest tech changes in history. Not the biggest, but up there.

seydor 4y ago

I m not sure it's overrated, but the concerns are very real. We love the model because it speaks our language as if it's "one of us", but this may be deceiving, and the complete lack of model for truth is disturbing. Making silly poems is fun but the real uses are in medicine and biology, fields that are so complex that they are probably impenetrable to the human mind. Can Reinforcement learning alone create a model for the truth? The Transformer does not seem to have one, it only works with syntax and referencing. How much % of truthfulness can we achieve, and is it good enough for scientific applications? If a blocker is found in the interface between the model and reality, it will be a huge disappointment

aaroninsf 4y ago
I am not so sure, there seems to be accumulating evidence that "finding the optimal solutions" means (requires) building a world model. Whether it's consistent with ground truth probably depends on what you mean by ground truth. Given the hypothesis that the optimal solution for deep learning presented with a given training set, is to represent (simulate) the formal systemic relationships that generated that set, by "modeling" such relationships (or discovering non-lossy optimized simplifications), I believe an implicit corollary, that the fidelity of simulation is only bounded by the information in the original data. Prediction: a big enough network, well enough trained, is capable of simulating with arbitrary fidelity, an arbitrarily complex system, to the point that lack of fidelity hits a noise floor. The testable bit of interest being whether such simulations predict novel states and outcomes (real world behavior) well enough. I don't see why they shouldn't, but the X-factor would seem to be the resolution and comprehensiveness of our training data. I can imagine toy domains like SHRDLU which are simple enough that we should be able to build large models well enough already to "model" them and tease this sort of speculation experimentally. I hope (assume) this is already being done...
- JohnFen 4y ago
  > there seems to be accumulating evidence that "finding the optimal solutions" means (requires) building a world model. Was this ever in doubt? This has been the case forever (even before "AI"), and I thought it was well-established. The fidelity of the model is the core problem. What "AI" is really providing is a shortcut that allows the creation of better models. But no model can ever be perfect, because the value of them is that they're an abstraction. As the old truism goes, a perfect map of a terrain would necessarily be indistinguishable from the actual terrain.

kyleyeats 4y ago

I'm working on an old-school AI personal project right now. I don't know how long that lasts. The generative stuff is more and more tempting. It rewards the horrible micromanager in me like nothing else.

nathias 4y ago

after the era of low hanging fruits of generative AI will be over I'm sure there will be a return to other approaches

GuB-42 4y ago

Is there a fundamental difference? I mean, the only thing GPT does is predict the next word, which makes it not so different from a compression algorithm. And diffusion models (the image generating stuff) are essentially fancy denoisers. Depending on how you assemble the big building blocks, you get generation or you get prediction.

deleted 4y ago
[deleted]
sweezyjeezy 4y ago
Depends how far you take the word 'fundamental', on the one hand yeah most DL systems are trying to predict something, and they generally have some concept of compression built in. But in terms of the steps to curate a dataset, train, test, iterate and actually use the model for a given end goal - they are pretty fundamentally different.
- sharemywin 4y ago
  I think the thing is though in Large multi models you give it all the data and test it against everything. And it generally does better across most of the benchmarks.
  - sweezyjeezy 4y ago
    That depends entirely on the use-case - for example if you wanted to build an AI to operate a self-driving car, just training on unlabelled data scraped from the internet is only going to get you so far. It doesn't learn how to do EVERYTHING (not yet at least).
baq 4y ago
GPT-3.5 is not a Markov chain, this is trivially true. While ‘predicts the next word’ is true, the mechanism of it is of interest and that is most certainly not trivial.

EGreg 4y ago

Yes! Just like HN is anti blockchain but super pro AI. It seems most applications of generative AI at scale will havd a huge negative for society, far worse than anything blockchain could have brought about.

deleted 4y ago
[deleted]

uoaei 4y ago

Generative methods per se are pretty sick and dope, and are still useful for many things beyond art generation.

kmeisthax 4y ago

People calling neural-net classifiers "old-school" AI confused me. For a second I thought they were talking about the really old "expert systems" with everything being a pile of hard-coded rules.

01100011 4y ago
It still feels like there's a place for these rule based systems(Prolog?) to at least place some constraints on the output of non-deterministic, generative AI. If nothing else, have a generative AI generate the ruleset so you have some explicit rules you can audit from time to time.
- theLiminator 4y ago
  Yeah, i think one potential way to use blackbox ai in newer systems is having guardrails that are validated as safe (but perhaps non-optimal) and ensuring that the ai takes action within that sample space. Obviously this is hard problem, but might open the doors for policies (in self-driving cars, for example) to be entirely ai driven.
  - earthboundkid 4y ago
    Obviously the solution is to get the LLM to output Prolog. Give it positive feedback if the Prolog compiles. :-)
    - 01100011 4y ago
      A friend of mine was just telling me how he asked GPT-3 to write a simple program in Prolog and it seemed to get it right. He didn't try compiling it, but he has enough experience w/ Prolog to say that it was more or less correct. I'm pretty cynical on LLMs(i.e. they're not intelligent and won't take all our jobs soon), but am coming around on their importance and capabilities.
- lincpa 4y ago
  [dead]

glitchc 4y ago

I see and I hear: "Don't be dazzled by AI computer vision's creative charm! Classical computer vision, though less flashy, remains crucial for solving real-world challenges and unleashing computer vision's true potential." Meant for those in classical computer vision before ML ate the field.

potatoman22 4y ago
If by classical CV they mean image processing and machine vision, I would say that's still true.

jedberg 4y ago

The real innovation will come one someone uses a Generative AI to make something, and then use a predictive AI to rate it's accuracy, making it go again until it passes the predictive AI. Basically a form of adversarial training/generation.

arrow7000 4y ago
Isn't this exactly how GANs work already?
- jedberg 4y ago
  Yes. But from I've seen no one has applied it to the latest Generative AIs.
  - arrow7000 4y ago
    Maybe an adversarial approach was used in training these models in the first place?
    - sharemywin 4y ago
      It was they were' trained using reinforcement learning with human feedback to create the critic.
      - jedberg 4y ago
        I hadn't thought about human feedback being an adversarial system, but I guess that makes sense, since it's basically a classifier saying "you got this wrong".
  - dereg 4y ago
    I’m pretty sure Anthropic’s Claude is doing that. https://scale.com/blog/chatgpt-vs-claude https://scale.com/blog/chatgpt-vs-claude
ChikkaChiChi 4y ago

ElijahLynn 4y ago

This article could be improved by starting off stating what some examples of Predictive AI is, as they did with Generative AI.

redox99 4y ago

I wonder how good multimodal GPT4 is at ImageNet. (You give it the image and prompt it with the 1000 classes and ask it which one the image belongs to). I'm surprised ClosedAI didn't include this kind of benchmark. I guess it doesn't do too well?

sharemywin 4y ago
Here's something on Clip https://www.pinecone.io/learn/zero-shot-image-classification-clip/ https://www.pinecone.io/learn/zero-shot-image-classification...

patrulek 4y ago

Old-school, huh. The skynet is closer than we think i guess.

efitz 4y ago

I think that 100% of the actually useful use cases for generative AI could be described in two words: “supervised autocomplete”.

orangecat 4y ago
That's not wrong, but an ideal autocompleter is a near-omniscient superintelligence. "The optimal approach to curing Alzheimer's is ______". "The proof of the Riemann hypothesis is as follows: ______". "The best way for me to improve my life is _______".
- kneebonian 4y ago
  I think the big difference is just being an Autocompleter is less concerned with generating something that is truthful, as in reflects the real world as we understand it described by physics, vs simply spitting out something that sounds good. Although we do have a litmus test in asking it "What is the meaning of life the universe and everything?"
  - gweinberg 4y ago
    Yes, exactly. An autocompleter is saying what the next words probably would be, not what it should be. It's like a chess program that tries to find the most likely move that a huan would make in the position rather than the best move.
    - efitz 4y ago
      That’s why I said “supervised” - in other words, someone competent in the domain and context is examining the output and correcting or discarding as necessary before use. “Unsupervised generative AI” is useless IMO.

wslh 4y ago

I would add that there are logic deductive and constraint systems that are more classical and work in some areas. It is not about a single method but we should he aware that AI is a superset of what we see.

phonebucket 4y ago

There is much more to generative models than building out language models and image models. Generative models are about characterising probability distributions. If you ever predict more than just the average of something using data, then you are doing generative modelling. The difference between generative modelling and predictive modelling is similar to the difference between stochastic modelling and deterministic modelling in the traditional applied mathematical sciences. Both have their place. Neither is overrated. Grab the best tool for the job.

tolciho 4y ago

As stated by John McCarthy--"I invented [AI] because we had to do something when we were trying to get money for a summer study" (the Lighthill debate)--this article passes the AI sniff test, or "please remember us predictive AI folks when you go to dole out your money" as all that is solid melts into PR.

kulkarniankita 4y ago

I love it but also waiting for the hype to settle down.

f0ld 4y ago

Or could it be possible that it was always going to end up like a black box it seems is it not? We will never truly understand the inner workings while it solves every problems that can be numbered by which it couldn't be solved with hardline algorithms previously. It's literally calling higher dimensional egrigores for answers or some blood magic Genie.