11 ms·

The point is that now we're at the point of diminishing return for increasing model size, unless we find a better modeling architecture than Transformer. I thi

The point is that now we're at the point of diminishing return for increasing model size, unless we find a better modeling architecture than Transformer. I think this is likely true; while all the other companies underestimated the capability of transformer (including Google itself!), OpenAI made a fairly accurate bet on the transformer based on the scaling law, put all the efforts to squeeze it until the last drop and took all the rewards. It's likely that GPT-4 is on the optimal spot between cost and performance and there won't be significant improvements on performance in a near future. I guess the next task would be more on efficiency, which has a significant implication on its productionization.

fauxpause_ 4y ago
Seems like a wild claim to make without any examples of gpt models which are bigger and no demonstrably better.
- xipix 4y ago
  Perhaps (a) there do exist bigger models that weren't better or (b) this model isn't better than somewhat smaller ones. Perhaps the CEO has seen diminishing returns.
  - fauxpause_ 4y ago
    Sure. It is possible there is evidence that is not shared. It’s dumb to assume this is the case though.
- mnky9800n 4y ago
  or like a curve of model complexity versus results or whatever showing it asymptotically approaches whatever. actually there was a great paper from microsoft research from like 2001 on spam filtering where they demonstrated that model complexity necessary for spam filtering went down as the size of the data set went up. That paper, which i can't seem to find now, had a big impact on me as a researcher because it so clearly demonstrated that small data is usually bad data and sophisticated models are sometimes solving problems will small data sets instead of problems with data. of course this paper came out the year friedman published his gradient boosting paper, i think random forest also was only recently published then as well (i think there is a paper from 1996 about RF and briemans two cultures paper came out this year where he discusses RF i believe), and this is a decade before gpu based neural networks. So times are different now. But actually i think the big difference is these days i probably ask chatgpt to write the boiler plate code for a gradient boosted model that takes data out of a relational database instead of writing it myself.
  - nomel 4y ago
    > model complexity necessary for spam filtering went down as the size of the data set went up My naive conclusion in that this means there are still massive gains to be had, since, for example, something like ChatGPT is just text, and the phrase "a picture is worth a thousand words" seems incredibly accurate, from my perspective. There's an incredible amount of non-text data out there still. Especially technical data. Is there any merit to this belief?
    - jacobr1 4y ago
      Yes. One of the frontiers of current research seems to be multi-modal models.
    - cjbprime 4y ago
      GPT-4 is actually multi-modal, not text. ChatGPT does not yet expose image upsubmission to it. But it's part of how the model was trained already.
    - throwaway2037 4y ago
      Excellent points in your post. You wrote: There's an incredible amount of non-text data out there still. Especially technical data. "Especially technical data." What does this part mean? Initially, I thought you meant things like images and video, but now I am confused.
      - sseagull 4y ago
        They might mean numerical data like scientific simulation data, sensor data, polling data, statistics, etc.
      - nomel 4y ago
        Schematics (of any sort), block diagrams, general spatial awareness (including anything related to puzzle pieces/packing, like circuit layout), most physics problems involving force diagrams, anything mechanical, etc. The text representation of any of these is ludicrously more complex than simple images. If you sit someone down, that works in one of these fields, you'll quickly see the limitations. It'll try to represent the concepts as text, with ascii art or some "attempt" at an ascii file format that can be used to draw, and its "reasoning" about these things is much more limited. I think most people interacting with GPT are in a text-only (and especially programming) bubble.
- summerlight 4y ago
  https://twitter.com/SmokeAwayyy/status/1646670920214536193 https://twitter.com/SmokeAwayyy/status/1646670920214536193 Sam explicitly said that there won't be GPT-5 in the near future, which is pretty clear evidence unless he's blatantly lying in public speaking.
  - kjellsbells 4y ago
    Well, "no GPT-5" isn't the same as saying "no new trained model", especially in the realm of marketing. Welcome to "GPT 2024" could be his next slogan.
  - thehumanmeat 4y ago
    That is one AI CEO out of 10,000. Just because OpenAI may not be interested in a larger model in the short term doesn't mean nobody else won't pursue it.
    - not2b 4y ago
      Others might pursue a smaller model that works as well as a larger model. If that can be done, whoever does it can very effectively compete on price/performance. It seems that to assume otherwise (the only way to improve is to get bigger) is to assume that OpenAI already has found the optimal architecture. That seems unlikely.
  - imatworkyo 4y ago
    Google recently said transformers wouldn't work ...
- hackerlight 4y ago
  It's not a wild claim when you have empirically well-validated scaling laws which make this very prediction.
  - fauxpause_ 4y ago
    That’s a generous usage of the term “empirically well-validated” and “law”
- mensetmanusman 4y ago
  Better on which axis? Do you want an AI that takes one hour to respond to? Some would for certain fields, but getting something fast and cheap is going to be hard now that Moore’s law is over.
  - AlecSchueler 4y ago
    Don't we all agree that GPT4 is "better" than GPT3? How are we evaluating that if the axis is such a mystery. Yeah maybe we can't quantify it like I can't tell you one writer is better than another in a quantitative but we can both still read their work and come to an understanding.
    - nwlieb 4y ago
      The runtime is quadratic for a given context size, although it seems like there is some progress on this front https://gwern.net/note/attention https://gwern.net/note/attention
    - MichaelZuo 4y ago
      Exponential scaling for a presumable GPT-5 suggests it's response time will be unusably long for the vast majority of use cases, and probably cost multiple dollars USD per query. Not to mention there doesn't actually exist enough English text data in the world to even double GPT-4's training set.
      - epups 4y ago
        Compute will also scale exponentially in coming years. The data source limitation seems to be a harder barrier, I think many companies are experimenting with AI generated content for training at this point.
        MichaelZuo 4y ago
        > Compute will also scale exponentially in coming years. Cost per transistor scaling has already plateaued or perhaps even inverted with TSMC's latest and greatest. And the new chips, even after 25 layers of EUV lithography, more than doubling the previous record, and an extra year of fine tuning, has total SRAM size scaling of -5% and logic scaling of -42%. These are numbers verified by experienced semi people.
- Barrin92 4y ago
  Both ChatGPT 3.5 and 4 literally fail the question: "What is the third letter in the third word of this sentence" When you've spent 100 million on training the thing and it fails on 1st grade ordinality I think it's fair to say you may not be on the right path
  - mcbuilder 4y ago
    But yet it can understand a json data schema from example and write javascript to interact with a library that I fed it and asked it to understand. Yes, I know its limitations, but it can also surprise me.
    - bccdee 4y ago
      The problem with basic programming questions like this is that there are a million elementary online tutorials for doing this or that with a json schema. "Simple programming questions based on commonly-used technology" are something it's been very heavily trained on.
  - m00x 4y ago
    These specific questions are very hard for an AI to answer. Just like humans suck at calculating numbers, AIs aren't good at sparse self-questioning. They're extremely good at other tasks, like taking very difficult tests that require a lot of knowledge storage. It's pretty obvious they're on the right path for what they're trying to achieve.
  - drippingfist 4y ago
    "The third word of this sentence is "the," and its third letter is "e." - GPT-4
    - foobiekr 4y ago
      Unfortunately it seems clear that openai trains gptX on common test questions. They still fail novel ones.
- bhouston 4y ago
  I suspect you are right. We may be stuck at the gpt4 sizes for a bit just because of hardware costs though. As they get bigger it costs too much to run them until our hardware becomes more optimal for these large models at 4 bits or so. I think the YouTube videos is going to be the next big training set. A transformer trained on all text and all of YouTube will be killer amazing at so much. I bet it can understand locomotion and balance and body control from YouTube. I wonder if TPUs, like Google's Tensor chip, will beat out GPUs when it comes to image/video based training?
  - TaylorAlexander 4y ago
    > I wonder if TPUs, like Google's Tensor chip, will beat out GPUs when it comes to image/video based training? One of the OpenAI guys was talking about this. He said the specific technology does not matter, it is just a cost line item. They don't need to have the best chip tech available as long as they have enough money. That said I am curious if anyone else can really comment on this. It seems like as we get to very large and expensive models we will produce more and more specialized technology.
    - bhouston 4y ago
      > They don't need to have the best chip tech available as long as they have enough money. That sounds like someone who is "Blitzscaling." Costs do not matter in those cases, just acquiring customers and marketshare. But for the rest of us, who will see benefits but are not trying to win a $100B market, we will cost optimize.
      - TaylorAlexander 4y ago
        Yes, agreed. I would like to run large models at home without serious expense.
    - daydream 4y ago
      Whether or not cost matters much depends on your perspective. If you’re OpenAI and GPT4 is just a step on the way to AGI, and you can amortize that huge cost over the hundreds of millions in revenue you’re gonna pull in from subscriptions and API use… then sure you’re probably not very cost sensitive. It could be 20% cheaper or 50% more expensive, whatever, it’s so good your customers will use it at a wide range of costs. And you have truckloads of money from Microsoft anyways. If you’re a company or a developer trying to build a feature, whole new product, or an entire company on top of GPT then that cost matters a whole lot. The difference between $0.06 and $0.006 per turn could be infeasible vs. shippable. If you’re trying to compete with OpenAI then you’re probably doing everything possible to reduce that training cost. So, whether or not it matters - it really depends.
- blast 4y ago
  OpenAI may have those internally though.
- kccqzy 4y ago
  If OpenAI's CEO is making this claim, don't you think he has internal data backing up the claim?
  - wetpaws 4y ago
    He might or might not, this is a point of the claim.
  - smeyer 4y ago
    No. I don't always assume that just because a CEO makes a public statement they have internal data backing up the claim. Sometimes they do! Other times, they have data but are misinterpreting it or missing something, but it's impossible to tell if the data is just internal. Other times they're making a statement without data based on their personal beliefs. Other times, they don't even think the statement is true but are saying it for messaging, marketing, or communication reasons! Like the previous commenter, I'd be much more confident an asymptote was reached if it was being demonstrated publicly.
  - zamnos 4y ago
    Only OpenAI and its CEO know the full details on GPT-4's sizes so that's entirely possible. But since it's an internal secret, there's nothing compelling him to tell the truth. For all we know, he has internal data backing up the opposite of the claim but is making this claim so as to discourage potential competitors from spending the money training an even bigger and competitive ML model. Sending potential competitors off on a wide goose chase that, when pushed, he can just say "oh our internal data (that no one outside of a trusted few have seen) said otherwise". I have no idea if sama is such a person, but you must admit that the possibility exists.
    - imtringued 4y ago
      [flagged]
  - worthless-trash 4y ago
    You'd hope so, but unless people put their evidence in public, it could simply be a tool to manipulate the public's expectations or competitors behavior. I'll get downvoted for this, apples previous CEO was consistently inaccurate about company innovation and performance numbers.
- deleted 4y ago
  [deleted]
bitL 4y ago
Transformers were known that they kept scaling up with more parameters and more training data so if the Open AI hit the limits of this scaling that would be a very important milestone in AI.
- fastball 4y ago
  I think performance will keep scaling, the main problem is cost. If you want to feed longer strings of tokens in, the price is going to keep going up.
GaggiX 4y ago
I think the next step is multimodality, GPT-4 can "see" probably using a method similar to miniGPT-4, so the embeddings are aligned using Q-former (or something similar), the next step would be to actually predict image tokens using the LM loss, this way it would be able to use the knowledge gained by "seeing" on other tasks like: making actual good ASCII art, making SVG that makes sense, and on a less superficial level having a better world model.
KhoomeiK 4y ago
Further improvements in efficiency need not come from alternative architectures. They'll likely also come from novel training objectives, optimizers, data augmentations, etc.
- PartiallyTyped 4y ago
  There's only so much first order derivatives can do for you. My bet is on sparsity, lottery tickets and symmetries.
  - ActorNightly 4y ago
    If you think about it, Transformers were basically a way to just generalize convolution - instead of a fixed kernel shape in the sense of image processing, you now have a learned kernel arbitrary shape. Big advancement in terms of what they allowed, but fundamentally not really a new concept. While these things represent a fundamental way we store information as humans, these have very little to do with actual reasoning. My bet is that Hebbian learning is going to see a resurgence. Basically the architecture needs to be able to partition data domains while drawing connections between them, and being able to run internal prediction mechanisms.
    - PartiallyTyped 4y ago
      If we wanted to generalize this further, attention is 'just' an instance of graph convolution, encoder-only models like Bert are complete graphs, decoder-only models like GPT are still complete graphs but the information flow is not bi-directional as in Bert; instead a node provides information to all subsequent nodes only giving rise to the causal nature. However, I don't think Hebbian learning will see a resurgence except maybe if it motivates some kind of pruning mechanism. I think that Sutton was right in 'The bitter lesson', the problem seems to be that we are hitting the limits of what we can do with our compute.
chubs 4y ago
Does this mean we’ve reached the next AI winter? This is as good as it gets for quite a long time? Honest question :) perhaps this will postpone everyone’s fears about the singularity…
- frozenport 4y ago
  No. Winter means people have lost interest in the research. If anything successes in ChatGPT etc will be motivation for continued efforts.
  - mkl 4y ago
    Winter means people have lost funding for the research. The ongoing productionising of large language models and multimodal models mean that that probably won't happen for quite a while.
- ericabiz 4y ago
  Many years ago, there was an image that floated around with Craigslist and all the websites that replaced small parts of it—personals, for sale ads, etc. It turned out the way to beat Craigslist wasn’t to build Yet Another Monolithic Craigslist, but to chunk it off in pieces and be the best at that piece. This is analogous to what’s happening with AI models. Sam Altman is saying we have reached the point where spending $100M+ trying to “beat” GPT-4 at everything isn’t the future. The next step is to chunk off a piece of it and turn it into something a particular industry would pay for. We already see small sprouts of those being launched. I think we will see some truly large companies form with this model in the next 5-10 years. To answer your question, yes, this may be as good as it gets now for monolithic language models. But it is just the beginning of what these models can achieve.
  - robocat 4y ago
    https://www.today.com/money/speculation-craigslist-slowly-dying-124007 https://www.today.com/money/speculation-craigslist-slowly-dy... from 2011 - is that what you were thinking of? Strange how few of those logos have survived, and how many new logos would now be on it. It would be interesting to see a modernised version.
deleted 4y ago
[deleted]
hesdeadjim 4y ago
Seems like a gigantic increase in the context window would be a great next step. GPT-4 is so damned good I tend to only become disappointed when it is forced to forget history.
worrycue 4y ago
> (including Google itself!) Bet Google won’t make that mistake again, i.e. it won’t publish as much and will be much more careful about what it publishes, least they give a competitor a useful tool and get nothing in return - when the competitor (in this case very ironically named) goes full commercial and close source everything they can. Open collaboration in AI, at least when it comes to corporations, might have come to an end.
- urthor 4y ago
  Sun Tzu. When you are weak pretend to be strong. When you are strong, pretend to be weak.
  - RecycledEle 4y ago
    My thought exactly. I assume this is to reduce the competition.
- hackernewds 4y ago
  The scale of the damage Open AI has done to the trust ecosystem with soliciting not just the work, but also massive fundraising and then privatizing the profits is almost unprecedented and permanent.
  - imposter 4y ago
    > trust ecosystem What trust ecosystem are you talking about ? It was a lack of foresight by google on their own discovery of transformers, and it would probably have been sitting in dust or been killed off by the time it would have taken them to reach GPT-2 level of progress.
    - anthlax 4y ago
      The trust ecosystem of the AI community - most every breakthrough was published publicly until OpenAI decided to take advantage of that. Besides that, this comment contained a ton of statements on what “would” have happened had Google not published. Interesting but worthless way to defend openAI’s actions.
BeenAGoodUser 4y ago
I really want to see a gpt-4-turbo model that's almost as inexpensive as gpt-3.5-turbo
alphabetatheta 4y ago
The main question here is how soon will the model architecture improvements come. I fathom it won't be that long till OpenAI or Google come up with something better. The tricky piece here is that these companies will now keep their models more proprietary to make profits. Can a competitor company figure out an on par or better model architecture in time?
deleted 4y ago
[deleted]
pengaru 4y ago
> 640K ought to be enough for anybody
kvetching 4y ago
I think there's a single point that we need to keep an eye out for; A model that got to the point where it's possible for it to propose new architectures, improve optimization & efficiency. In other words, the extremely massive model that could do this doesn't need to be available to the public. It's sole purpose should be to be used internally by a company to improve itself. This is actually the point at which many say could lead to the singularity.
ckdarby 4y ago
If you use GPT4 as a "routing" service to send down to other models the journey is only beginning and most of the industry hasn't understood we're well beyond a doubling at the moment.
Alifatisk 4y ago
> while all the other companies underestimated the capability of transformer (including Google itself!), OpenAI made a fairly accurate bet on the transformer based on the scaling law, put all the efforts to squeeze it until the last drop and took all the rewards. You hit the nail
Tepix 4y ago
> It's likely that GPT-4 is on the optimal spot between cost and performance and there won't be significant improvements on performance in a near future. Given the amount of resources being thrown at AI right now, i consider this to be very unlikely indeed.
- hnfong 4y ago
  I think GP's comment is probably apt if understood together with the qualification of "unless we find a better modeling architecture than Transformer".

fauxpause_ 4y ago

Seems like a wild claim to make without any examples of gpt models which are bigger and no demonstrably better.

xipix 4y ago
Perhaps (a) there do exist bigger models that weren't better or (b) this model isn't better than somewhat smaller ones. Perhaps the CEO has seen diminishing returns.
- fauxpause_ 4y ago
  Sure. It is possible there is evidence that is not shared. It’s dumb to assume this is the case though.
mnky9800n 4y ago
or like a curve of model complexity versus results or whatever showing it asymptotically approaches whatever. actually there was a great paper from microsoft research from like 2001 on spam filtering where they demonstrated that model complexity necessary for spam filtering went down as the size of the data set went up. That paper, which i can't seem to find now, had a big impact on me as a researcher because it so clearly demonstrated that small data is usually bad data and sophisticated models are sometimes solving problems will small data sets instead of problems with data. of course this paper came out the year friedman published his gradient boosting paper, i think random forest also was only recently published then as well (i think there is a paper from 1996 about RF and briemans two cultures paper came out this year where he discusses RF i believe), and this is a decade before gpu based neural networks. So times are different now. But actually i think the big difference is these days i probably ask chatgpt to write the boiler plate code for a gradient boosted model that takes data out of a relational database instead of writing it myself.
- nomel 4y ago
  > model complexity necessary for spam filtering went down as the size of the data set went up My naive conclusion in that this means there are still massive gains to be had, since, for example, something like ChatGPT is just text, and the phrase "a picture is worth a thousand words" seems incredibly accurate, from my perspective. There's an incredible amount of non-text data out there still. Especially technical data. Is there any merit to this belief?
  - jacobr1 4y ago
    Yes. One of the frontiers of current research seems to be multi-modal models.
  - cjbprime 4y ago
    GPT-4 is actually multi-modal, not text. ChatGPT does not yet expose image upsubmission to it. But it's part of how the model was trained already.
  - throwaway2037 4y ago
    Excellent points in your post. You wrote: There's an incredible amount of non-text data out there still. Especially technical data. "Especially technical data." What does this part mean? Initially, I thought you meant things like images and video, but now I am confused.
    - sseagull 4y ago
      They might mean numerical data like scientific simulation data, sensor data, polling data, statistics, etc.
    - nomel 4y ago
      Schematics (of any sort), block diagrams, general spatial awareness (including anything related to puzzle pieces/packing, like circuit layout), most physics problems involving force diagrams, anything mechanical, etc. The text representation of any of these is ludicrously more complex than simple images. If you sit someone down, that works in one of these fields, you'll quickly see the limitations. It'll try to represent the concepts as text, with ascii art or some "attempt" at an ascii file format that can be used to draw, and its "reasoning" about these things is much more limited. I think most people interacting with GPT are in a text-only (and especially programming) bubble.
summerlight 4y ago
https://twitter.com/SmokeAwayyy/status/1646670920214536193 https://twitter.com/SmokeAwayyy/status/1646670920214536193 Sam explicitly said that there won't be GPT-5 in the near future, which is pretty clear evidence unless he's blatantly lying in public speaking.
- kjellsbells 4y ago
  Well, "no GPT-5" isn't the same as saying "no new trained model", especially in the realm of marketing. Welcome to "GPT 2024" could be his next slogan.
- thehumanmeat 4y ago
  That is one AI CEO out of 10,000. Just because OpenAI may not be interested in a larger model in the short term doesn't mean nobody else won't pursue it.
  - not2b 4y ago
    Others might pursue a smaller model that works as well as a larger model. If that can be done, whoever does it can very effectively compete on price/performance. It seems that to assume otherwise (the only way to improve is to get bigger) is to assume that OpenAI already has found the optimal architecture. That seems unlikely.
- imatworkyo 4y ago
  Google recently said transformers wouldn't work ...
hackerlight 4y ago
It's not a wild claim when you have empirically well-validated scaling laws which make this very prediction.
- fauxpause_ 4y ago
  That’s a generous usage of the term “empirically well-validated” and “law”
mensetmanusman 4y ago
Better on which axis? Do you want an AI that takes one hour to respond to? Some would for certain fields, but getting something fast and cheap is going to be hard now that Moore’s law is over.
- AlecSchueler 4y ago
  Don't we all agree that GPT4 is "better" than GPT3? How are we evaluating that if the axis is such a mystery. Yeah maybe we can't quantify it like I can't tell you one writer is better than another in a quantitative but we can both still read their work and come to an understanding.
  - nwlieb 4y ago
    The runtime is quadratic for a given context size, although it seems like there is some progress on this front https://gwern.net/note/attention https://gwern.net/note/attention
  - MichaelZuo 4y ago
    Exponential scaling for a presumable GPT-5 suggests it's response time will be unusably long for the vast majority of use cases, and probably cost multiple dollars USD per query. Not to mention there doesn't actually exist enough English text data in the world to even double GPT-4's training set.
    - epups 4y ago
      Compute will also scale exponentially in coming years. The data source limitation seems to be a harder barrier, I think many companies are experimenting with AI generated content for training at this point.
      - MichaelZuo 4y ago
        > Compute will also scale exponentially in coming years. Cost per transistor scaling has already plateaued or perhaps even inverted with TSMC's latest and greatest. And the new chips, even after 25 layers of EUV lithography, more than doubling the previous record, and an extra year of fine tuning, has total SRAM size scaling of -5% and logic scaling of -42%. These are numbers verified by experienced semi people.
Barrin92 4y ago
Both ChatGPT 3.5 and 4 literally fail the question: "What is the third letter in the third word of this sentence" When you've spent 100 million on training the thing and it fails on 1st grade ordinality I think it's fair to say you may not be on the right path
- mcbuilder 4y ago
  But yet it can understand a json data schema from example and write javascript to interact with a library that I fed it and asked it to understand. Yes, I know its limitations, but it can also surprise me.
  - bccdee 4y ago
    The problem with basic programming questions like this is that there are a million elementary online tutorials for doing this or that with a json schema. "Simple programming questions based on commonly-used technology" are something it's been very heavily trained on.
- m00x 4y ago
  These specific questions are very hard for an AI to answer. Just like humans suck at calculating numbers, AIs aren't good at sparse self-questioning. They're extremely good at other tasks, like taking very difficult tests that require a lot of knowledge storage. It's pretty obvious they're on the right path for what they're trying to achieve.
- drippingfist 4y ago
  "The third word of this sentence is "the," and its third letter is "e." - GPT-4
  - foobiekr 4y ago
    Unfortunately it seems clear that openai trains gptX on common test questions. They still fail novel ones.
bhouston 4y ago
I suspect you are right. We may be stuck at the gpt4 sizes for a bit just because of hardware costs though. As they get bigger it costs too much to run them until our hardware becomes more optimal for these large models at 4 bits or so. I think the YouTube videos is going to be the next big training set. A transformer trained on all text and all of YouTube will be killer amazing at so much. I bet it can understand locomotion and balance and body control from YouTube. I wonder if TPUs, like Google's Tensor chip, will beat out GPUs when it comes to image/video based training?
- TaylorAlexander 4y ago
  > I wonder if TPUs, like Google's Tensor chip, will beat out GPUs when it comes to image/video based training? One of the OpenAI guys was talking about this. He said the specific technology does not matter, it is just a cost line item. They don't need to have the best chip tech available as long as they have enough money. That said I am curious if anyone else can really comment on this. It seems like as we get to very large and expensive models we will produce more and more specialized technology.
  - bhouston 4y ago
    > They don't need to have the best chip tech available as long as they have enough money. That sounds like someone who is "Blitzscaling." Costs do not matter in those cases, just acquiring customers and marketshare. But for the rest of us, who will see benefits but are not trying to win a $100B market, we will cost optimize.
    - TaylorAlexander 4y ago
      Yes, agreed. I would like to run large models at home without serious expense.
  - daydream 4y ago
    Whether or not cost matters much depends on your perspective. If you’re OpenAI and GPT4 is just a step on the way to AGI, and you can amortize that huge cost over the hundreds of millions in revenue you’re gonna pull in from subscriptions and API use… then sure you’re probably not very cost sensitive. It could be 20% cheaper or 50% more expensive, whatever, it’s so good your customers will use it at a wide range of costs. And you have truckloads of money from Microsoft anyways. If you’re a company or a developer trying to build a feature, whole new product, or an entire company on top of GPT then that cost matters a whole lot. The difference between $0.06 and $0.006 per turn could be infeasible vs. shippable. If you’re trying to compete with OpenAI then you’re probably doing everything possible to reduce that training cost. So, whether or not it matters - it really depends.
blast 4y ago
OpenAI may have those internally though.
kccqzy 4y ago
If OpenAI's CEO is making this claim, don't you think he has internal data backing up the claim?
- wetpaws 4y ago
  He might or might not, this is a point of the claim.
- smeyer 4y ago
  No. I don't always assume that just because a CEO makes a public statement they have internal data backing up the claim. Sometimes they do! Other times, they have data but are misinterpreting it or missing something, but it's impossible to tell if the data is just internal. Other times they're making a statement without data based on their personal beliefs. Other times, they don't even think the statement is true but are saying it for messaging, marketing, or communication reasons! Like the previous commenter, I'd be much more confident an asymptote was reached if it was being demonstrated publicly.
- zamnos 4y ago
  Only OpenAI and its CEO know the full details on GPT-4's sizes so that's entirely possible. But since it's an internal secret, there's nothing compelling him to tell the truth. For all we know, he has internal data backing up the opposite of the claim but is making this claim so as to discourage potential competitors from spending the money training an even bigger and competitive ML model. Sending potential competitors off on a wide goose chase that, when pushed, he can just say "oh our internal data (that no one outside of a trusted few have seen) said otherwise". I have no idea if sama is such a person, but you must admit that the possibility exists.
  - imtringued 4y ago
    [flagged]
- worthless-trash 4y ago
  You'd hope so, but unless people put their evidence in public, it could simply be a tool to manipulate the public's expectations or competitors behavior. I'll get downvoted for this, apples previous CEO was consistently inaccurate about company innovation and performance numbers.
deleted 4y ago
[deleted]

bitL 4y ago

Transformers were known that they kept scaling up with more parameters and more training data so if the Open AI hit the limits of this scaling that would be a very important milestone in AI.

fastball 4y ago
I think performance will keep scaling, the main problem is cost. If you want to feed longer strings of tokens in, the price is going to keep going up.

GaggiX 4y ago

I think the next step is multimodality, GPT-4 can "see" probably using a method similar to miniGPT-4, so the embeddings are aligned using Q-former (or something similar), the next step would be to actually predict image tokens using the LM loss, this way it would be able to use the knowledge gained by "seeing" on other tasks like: making actual good ASCII art, making SVG that makes sense, and on a less superficial level having a better world model.

KhoomeiK 4y ago

Further improvements in efficiency need not come from alternative architectures. They'll likely also come from novel training objectives, optimizers, data augmentations, etc.

PartiallyTyped 4y ago
There's only so much first order derivatives can do for you. My bet is on sparsity, lottery tickets and symmetries.
- ActorNightly 4y ago
  If you think about it, Transformers were basically a way to just generalize convolution - instead of a fixed kernel shape in the sense of image processing, you now have a learned kernel arbitrary shape. Big advancement in terms of what they allowed, but fundamentally not really a new concept. While these things represent a fundamental way we store information as humans, these have very little to do with actual reasoning. My bet is that Hebbian learning is going to see a resurgence. Basically the architecture needs to be able to partition data domains while drawing connections between them, and being able to run internal prediction mechanisms.
  - PartiallyTyped 4y ago
    If we wanted to generalize this further, attention is 'just' an instance of graph convolution, encoder-only models like Bert are complete graphs, decoder-only models like GPT are still complete graphs but the information flow is not bi-directional as in Bert; instead a node provides information to all subsequent nodes only giving rise to the causal nature. However, I don't think Hebbian learning will see a resurgence except maybe if it motivates some kind of pruning mechanism. I think that Sutton was right in 'The bitter lesson', the problem seems to be that we are hitting the limits of what we can do with our compute.

chubs 4y ago

Does this mean we’ve reached the next AI winter? This is as good as it gets for quite a long time? Honest question :) perhaps this will postpone everyone’s fears about the singularity…

frozenport 4y ago
No. Winter means people have lost interest in the research. If anything successes in ChatGPT etc will be motivation for continued efforts.
- mkl 4y ago
  Winter means people have lost funding for the research. The ongoing productionising of large language models and multimodal models mean that that probably won't happen for quite a while.
ericabiz 4y ago
Many years ago, there was an image that floated around with Craigslist and all the websites that replaced small parts of it—personals, for sale ads, etc. It turned out the way to beat Craigslist wasn’t to build Yet Another Monolithic Craigslist, but to chunk it off in pieces and be the best at that piece. This is analogous to what’s happening with AI models. Sam Altman is saying we have reached the point where spending $100M+ trying to “beat” GPT-4 at everything isn’t the future. The next step is to chunk off a piece of it and turn it into something a particular industry would pay for. We already see small sprouts of those being launched. I think we will see some truly large companies form with this model in the next 5-10 years. To answer your question, yes, this may be as good as it gets now for monolithic language models. But it is just the beginning of what these models can achieve.
- robocat 4y ago
  https://www.today.com/money/speculation-craigslist-slowly-dying-124007 https://www.today.com/money/speculation-craigslist-slowly-dy... from 2011 - is that what you were thinking of? Strange how few of those logos have survived, and how many new logos would now be on it. It would be interesting to see a modernised version.

deleted 4y ago

[deleted]

hesdeadjim 4y ago

Seems like a gigantic increase in the context window would be a great next step. GPT-4 is so damned good I tend to only become disappointed when it is forced to forget history.

worrycue 4y ago

> (including Google itself!) Bet Google won’t make that mistake again, i.e. it won’t publish as much and will be much more careful about what it publishes, least they give a competitor a useful tool and get nothing in return - when the competitor (in this case very ironically named) goes full commercial and close source everything they can. Open collaboration in AI, at least when it comes to corporations, might have come to an end.

urthor 4y ago
Sun Tzu. When you are weak pretend to be strong. When you are strong, pretend to be weak.
- RecycledEle 4y ago
  My thought exactly. I assume this is to reduce the competition.
hackernewds 4y ago
The scale of the damage Open AI has done to the trust ecosystem with soliciting not just the work, but also massive fundraising and then privatizing the profits is almost unprecedented and permanent.
- imposter 4y ago
  > trust ecosystem What trust ecosystem are you talking about ? It was a lack of foresight by google on their own discovery of transformers, and it would probably have been sitting in dust or been killed off by the time it would have taken them to reach GPT-2 level of progress.
  - anthlax 4y ago
    The trust ecosystem of the AI community - most every breakthrough was published publicly until OpenAI decided to take advantage of that. Besides that, this comment contained a ton of statements on what “would” have happened had Google not published. Interesting but worthless way to defend openAI’s actions.

BeenAGoodUser 4y ago

I really want to see a gpt-4-turbo model that's almost as inexpensive as gpt-3.5-turbo

alphabetatheta 4y ago

The main question here is how soon will the model architecture improvements come. I fathom it won't be that long till OpenAI or Google come up with something better. The tricky piece here is that these companies will now keep their models more proprietary to make profits. Can a competitor company figure out an on par or better model architecture in time?

deleted 4y ago

[deleted]

pengaru 4y ago

> 640K ought to be enough for anybody

kvetching 4y ago

I think there's a single point that we need to keep an eye out for; A model that got to the point where it's possible for it to propose new architectures, improve optimization & efficiency. In other words, the extremely massive model that could do this doesn't need to be available to the public. It's sole purpose should be to be used internally by a company to improve itself. This is actually the point at which many say could lead to the singularity.

ckdarby 4y ago

If you use GPT4 as a "routing" service to send down to other models the journey is only beginning and most of the industry hasn't understood we're well beyond a doubling at the moment.

Alifatisk 4y ago

> while all the other companies underestimated the capability of transformer (including Google itself!), OpenAI made a fairly accurate bet on the transformer based on the scaling law, put all the efforts to squeeze it until the last drop and took all the rewards. You hit the nail

Tepix 4y ago

> It's likely that GPT-4 is on the optimal spot between cost and performance and there won't be significant improvements on performance in a near future. Given the amount of resources being thrown at AI right now, i consider this to be very unlikely indeed.

hnfong 4y ago
I think GP's comment is probably apt if understood together with the qualification of "unless we find a better modeling architecture than Transformer".