26 ms·

OpenAI’s CEO says the age of giant AI models is already over

• gobblers1 4y ago
  [dead]
• carlsborg 4y ago
  The 2017 Transformers paper has ~71,000 papers citing it. The sheer magnitude of human mental effort globally that is chasing the forefront of machine learning is unprecedented and amazing.
  • intalentive 4y ago
    Local minimum. LeCun is right.
• labrador 4y ago
  https://archive.is/s4V9e https://archive.is/s4V9e He did not say what kind of research strategies or techniques might take its place. In the paper describing GPT-4, OpenAI says its estimates suggest diminishing returns on scaling up model size. Altman said there are also physical limits to how many data centers the company can build and how quickly it can build them.
  • ftxbro 4y ago
    > In the paper describing GPT-4, OpenAI says its estimates suggest diminishing returns on scaling up model size. I read the two papers (gpt 4 tech report, and sparks of agi) and in my opinion they don't support this conclusion. They don't even say how big GPT-4 is, because "Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar." > Altman said there are also physical limits to how many data centers the company can build and how quickly it can build them. OK so his argument is like "the giant robots won't be powerful, but we won't show how big our robots are, and besides, there are physical limits to how giant of a robot we can build and how quickly we can build it." I feel like this argument is sus.
    • sangnoir 4y ago
      OpenAI has likely run into a wall (or is about to) for model size given it's funding amount/structure[1] - unlike its competition who actually own data centers and have lower marginsl costs. It's just like when peak-iPad Apple claimed that a "post-PC" age was upon us. 1. What terms could Microsoft wring out of OpenAI for another funding round?
• pixl97 4y ago
  The Age of Giants is over... The Age of Behemoths has begun! but sir, that means the same thing Throw this heretic into the pit of terror.
  • hanselot 4y ago
    The pit of terror is full. Fine, to the outhouse of madness then. Before I get nuked from orbit for daring to entertain humor, if someone is running ahead of me in a marathon, and running so far ahead, yet still broadcasting things to the back for the slow people (like myself), then eventually we catch up to them, and they suddenly say, you know what guys, we should stop running in this direction, there's nothing to see here right before anyone else is able to verify the veracity of their statement, perhaps it would still be in the public interest for at least one person to verify what they are saying. Given how skeptical the internet at large has been of Musk's acquisition of a company, it's interesting that the skepticism is suddenly put on hold when looking at this part of his work...
    • deleted 4y ago
      [deleted]
• fergie 4y ago
  The most comforting AI news I have read this year.
  • jackmott42 4y ago
    If you are worried about AI, this shouldn't make you feel a ton better. GPT4 is just trained to predict the next word, a very simple but crude approach and look what it can do! Imagine when a dozen models are wired together and giving each other feedback with more clever training and algorithms on future faster hardware. It is still going to get wild
    • ignoramous 4y ago
      > Imagine when a dozen models are wired together... Wouldn't these models hallucinate more than normal, then?
    • causi 4y ago
      I worry that the hardware requirements are only going to accelerate the cloud-OS integration. Imagine a PC that's entirely unusable offline.
      • cj 4y ago
        > Imagine a PC that's entirely unusable offline. FWIW we had thin clients in computer labs in middle school / high school 15 years ago (and still today these are common in enterprise environments, e.g. Citrix). Biggest issue is network latency which is limited by the speed of light, so I imagine if computers in 10 years require resources not available locally it would likely be a local/cloud hybrid model.
    • ShamelessC 4y ago
      Machine learning is actually premised on being “simple” to implement. The more priors you hardcode with clever algorithms, the closer you get to what we already have. The point is to automate the process of learning. We do this now with relatively simple loss functions and models containing relatively simple parameters. The main stipulation is that they are all defined to be continuous so that you can use the chain rule from calculus to calculate the error with respect to every parameter without taking so long that it would never finish. I agree that your suggested approach of applying cleverness to what we have now will probably produce better results. But that’s not going to stop better architectures, hardware and even entire regimes from being developed until we approach AGI. My suspicion is that there’s still a few breakthroughs waiting to be made. I also suspect that sufficiently advanced models will make such breakthroughs easier to discover.
• mupuff1234 4y ago
  Ok cool, so release the weights and your research.
• xt00 4y ago
  Saying “hey don’t go down the path we are on, where we are making money and considered the best in the world.. it’s a dead end” rings pretty hollow.. like “don’t take our lunch please?” Might be a similar statement it feels..
  • whywhywhywhy 4y ago
    Everyone hoping to compete with OpenAI should have an "Always do the opposite of what Sam says" sign on the wall.
  • Freire_Herval 4y ago
    [dead]
  • famouswaffles 4y ago
    It's a pretty sus argument for sure when they're scared to release even parameter size. although the title is a bit misleading on what he was actually saying. still, there's a lot left to go in terms of scale. Even if it isn't parameter size(and there's still lots of room here too, it just won't be economical), contrary to popular belief, there's lots of data left to mine
  • thewataccount 4y ago
    Nah - GPT-4 is crazy expensive, paying 20$/mo only get's you 25messages/3hours and it's crazy slow. The api is rather expensive too. I'm pretty sure that GPT-4 is ~1T-2T parameters, and they're struggling to run it(at reasonable performance and profit). So far their strategy has been to 10x the parameter count every GPT generation, and the problem is that there's diminishing returns everytime they do that. AFAIK they've now resorted to chunking GPT through the GPUs because of the 2 to 4 terabytes of VRAM required (at 16bit). So now they've reached the edge of what they can reasonably run, and even if they do 10x it the expected gains are less. On top of this, models like LLaMa have shown that it's possible to cut the parameter count substantially and still get decent results (albiet the opensource stuff still hasn't caught up). On top of all of this, keep in mind that at 8bit resolution 175B parameters (GBPT3.5) requires over 175GB of VRAM. This is crazy expensive and would never fit on consumer devices. Even if you use quantization and use 4bit, you still need over 80GB of VRAM. This definitely is not a "throw them off the trail" tactic - in order for this to actually scale the way everyone envisions both in performance and running on consumer devices - research HAS to be on improving the parameter count. And again there's lots of research showing its very possible to do. tl;dr: smaller = cheaper+faster+more accessible+same performance
• deleted 4y ago
  [deleted]
  • Freire_Herval 4y ago
    [dead]
• lossolo 4y ago
  We arrived at the top of the tree in our journey to the moon.
  • daniel_reetz 4y ago
    "You can't get to the moon by climbing successively taller trees"
  • famouswaffles 4y ago
    No we haven't. the title is misleading. there's plenty of scale room left. part of it might just not be economical (parameter sie) but there's data. If you take this to mean, "we're at a dead end" you'd be very wrong
• galaxytachyon 4y ago
  What age? Like, 3 years? On the other hand though, Chinchilla and multimodal approaches already showed how later AIs can be improved beyond throwing petabytes of data at them. It is all about variety and quality from now on I think. You can teach a person all about the color zyra but without actually ever seeing it, they will never fully understand that color.
  • idiotsecant 4y ago
    It does seem, though, that using chinchilla like techniques does not create a copy with the same quality as the original. It's pretty good for some definition of the phrase, but it isn't equivalent, it's a lossy technique.
    • galaxytachyon 4y ago
      I agree on the lossy. There is a tradeoff between efficiency and comprehensiveness, kind of. It would be pretty funny if in the end, the most optimal method turns out to be the brain we already have. Extremely efficient, hardware optimized, but slow as hell and misunderstand stuff all the time unless prompted with specific phrases.
• jcims 4y ago
  I'm no expert but doesn't the architecture of minigpt4 that's on the front page right now give some indication of what the future might look like?
  • MuffinFlavored 4y ago
    eh, I haven't personally found a usecase for LLMs yet given the fact that you can't trust the output and it needs to be verified by a human (which might as well be just as time consuming/expensive as actually doing the task yourself)
    • causi 4y ago
      I've enjoyed using it for very small automation tasks. For instance, it helped me write scripts to take all my audiobooks with poor recording quality, split them into 59-minute chunks, and upload them to Adobe's free audio enhancement site to vastly improve the listening experience.
    • Uehreka 4y ago
      I’d reconsider the “might as well just be as time consuming” thing. I see this argument about Copilot a lot, and it’s really wrong there, so it might be wrong here too. Like, for most of the time I’m using it, Copilot saves me 30 seconds here and there and it takes me about a second to look at the line or two of code and go “yeah, that’s right”. It adds up, especially when I’m working with an unfamiliar language and forget which Collection type I’m going to need or something.
      • MuffinFlavored 4y ago
        > Like, for most of the time I’m using it, Copilot saves me 30 seconds here and there and it takes me about a second to look at the line or two of code and go “yeah, that’s right”. I've never used Copilot but I've tried to replace StackOverflow with ChatGPT. The difference is, the StackOverflow responses compile/are right. The ChatGPT responses will make up an API that doesn't exist. Major setback.
    • 4y ago
• cleandreams 4y ago
  Once you've trained on the internet and most published books (and more...) what else is there to do? You can't scale up massively anymore.
  • deleted 4y ago
    [deleted]
  • m4jor 4y ago
    They didn't train it on the entire internet tho, only a small amount (in comparison to entire internet). Still plenty they could do.
  • JasonZ2 4y ago
    Video. > YouTubers upload about 720,000 hours of fresh video content per day. Over 500 hours of video were uploaded to YouTube per minute in 2020, which equals 30,000 new video uploads per hour. Between 2014 and 2020, the number of video hours uploaded grew by about 40%.
    • sottol 4y ago
      But what are you mostly "teaching" the LLM then? Mundane everyday stuff? I guess that would make them better at "being average human" but is that what we want? It already seems that prompting the LLM to be above-average ("pretend to be an expert") improves performance.
      • dougmwne 4y ago
        This whole conversation about training set size is bizarre. No one ever asks what’s in the training set. Why would a trillion tokens of mundane gossip improve a LLMs ability to do anything valuable at all? If a scrape of the general internet, scientific papers and books isn’t enough, a trillion trillion trillion text messages to mom aren’t going to change matters.
  • Animats 4y ago
    Right. They've already sucked in most of the good general sources of information. Adding vast amounts of low-quality content probably won't help much and might degrade the quality of the trained model.
• tfehring 4y ago
  Related reading: https://dynomight.net/scaling/ https://dynomight.net/scaling/ In short it seems like virtually all of the improvement in future AI models will come from better algorithms, with bigger and better data a distant second, and more parameters a distant third. Of course, this claim is itself internally inconsistent in that it assumes that new algorithms won't alter the returns to scale from more data or parameters. Maybe a more precise set of claims would be (1) we're relatively close to the fundamental limits of transformers, i.e., we won't see another GPT-2-to-GPT-4-level jump with current algorithms; (2) almost all of the incremental improvements to transformers will require bigger or better-quality data (but won't necessarily require more parameters); and (3) all of this is specific to current models and goes out the window as soon as a non-transformer-based generative model approaches GPT-4 performance using a similar or lesser amount of compute.
  • brucethemoose2 4y ago
    Better data is still critical, even if bigger data isn't. The linked article emphasizes this.
    • tfehring 4y ago
      I'd bet on a 2030 model trained on the same dataset as GPT-4 over GPT-4 trained with perfect-quality data, hands down. If data quality were that critical, practitioners could ignore the Internet and just train on books and scientific papers and only sacrifice <1 order of magnitude of data volume. Granted, that's not a negligible amount of training data to give up, but it places a relatively tight upper bound on the potential gain from improving data quality.
      • CuriouslyC 4y ago
        It's possible that this effect washes out as data increases, but researchers have shown that for smaller data set sizes average quality has a large impact on model output.
    • NeuroCoder 4y ago
      So true. There are still plenty of areas where we lack sufficient data to even approach applying this sort of model. How are we going to make similar advances in something like medical informatics where we not only have less data readily available but its much more difficult to acquire more data
• arenaninja 4y ago
  An amusing thought I've had recently is whether LLMs are in the same league as the millions of monkeys at the keyboard, struggling to reproduce one of the complete works of William Shakespeare. But I think not, since monkeys probably don't "improve" noticeably with time or input.
  • mhb 4y ago
    But I think not, since monkeys probably don't "improve" noticeably with time or input. Maybe once tons of bananas are introduced...
• thunderbird120 4y ago
  >"the company’s CEO, Sam Altman, says further progress will not come from making models bigger. “I think we're at the end of the era where it's going to be these, like, giant, giant models,” he told an audience at an event held at MIT late last week. “We'll make them better in other ways.” So to reiterate, he is not saying that the age of giant AI models is over. Current top-of-the-line AI models are giant and likely will continue to be. However, there's not point in training models you can't actually run economically. Inference costs need to stay grounded which means practical model sizes have a limit. More effort is going to go into making models efficient to run even if it comes at the expense of making them less efficient to train.
  • hcks 4y ago
    Yes, but it also tells us that if Altman is honest here, then he doesn’t believe GPT-like models can scale to near level human performances (because even if the cost of compute was 10x or even 100x it would still be economically sound).
    • deleted 4y ago
      [deleted]
    • famouswaffles 4y ago
      No it doesn't. For one thing they're already at human performance. For another, i don't think you realize how expensive inference can get. Microsoft with no scant amount of available compute is struggling to run gpt-4 such that they're rationing it between subsidiaries while they try to jack up compute. So saying, it would be economically sound if it cost x10 or x100 what it costs now is a joke.
      • smeagull 4y ago
        This tells me you haven't really stress tested the model. GPT is currently at the stage of "person who is at the meeting, but not really paying attention so you have to call them out". Once GPT is pushed, it scrambles and falls over for most applications. The failure modes range from contradicting itself, making up things for applications that shouldn't allow it, to ignoring prompts, to simply being unable to perform tasks at all.
• deleted 4y ago
  [deleted]
• donpark 4y ago
  I think Sam is referring to transition from "Deep" to "Long" learning [1]. What new emergent properties, if any, will 1 billion tokens unlock? [1] https://hazyresearch.stanford.edu/blog/2023-03-27-long-learning https://hazyresearch.stanford.edu/blog/2023-03-27-long-learn...
• enduser 4y ago
  "When we set the upper limit of PC-DOS at 640K, we thought nobody would ever need that much memory." Bill Gates
  • bagels 4y ago
    Gates has refuted saying this. Are you implying by analogy that Altman hasn't said/will disclaim saying that "the age of giant AI models is almost over"?
    • enduser 4y ago
      Just that there is tremendous hubris in the statement—at least when the statement stands alone. Vastly larger LLMs will probably become one or more relatively small components or layers of much larger systems that run on whatever we use as telephones in 20 years. I suspect what he means is that OpenAI is finding diminishing returns from throwing money and hardware at larger models right now and that they are investigating other and/or composite AI techniques that make more optimal use of their hardware investment.
  • gowld 4y ago
    That was in context of a how much RAM would be needed for software of that era, not the entire future of computing.
• zwieback 4y ago
  The age of CEOs that recently got washed to the top saying dumbish things is just starting, though.
• antibasilisk 4y ago
  it's over, billions of parameters must be released
  • chihuahua 4y ago
    Soon, the EU will pass a law that forces any AI company to provide you with a printout of the entire model upon request.
• jstx1 4y ago
  Ilya Sutskever from OpenAI saying that the data situation is good and there's more data to train on - https://youtu.be/Yf1o0TQzry8?t=657 https://youtu.be/Yf1o0TQzry8?t=657
  • jarbus 4y ago
    Maybe more data to train on while keeping model size the same
• screye 4y ago
  small AI model != cheap AI model. It costs the same to train as these giant models. You merely spend they money on training it for longer instead of larger.
• aaroninsf 4y ago
  "...for the current cycle, in our specific public-facing market." As most here well know "over" is one of those words like "never" which particularly in this space should pretty much always be understood as implicitly accompanied by a footnote backtracking to include near-term scope.
• generalizations 4y ago
  I'd bet that what he, and the competition, is realizing is that the bigger models are too expensive to run. Pretty sure Microsoft swapped out Bing for something a lot smaller in the last couple of weeks; Google hasn't even tried to implement a publicly available large model. And OpenAI still has usage caps on their GPT-4. I'd bet that they can still see improvement in performance with GPT-5, but that when they look at the usage ratio of GPT3.5 turbo, gpt3.5 legacy, and GPT4, they realized that there is a decreasing rate of return for increasingly smart models - most people don't need a brilliantly intelligent assistant, they just need a not-dumb assistant. Obviously some practitioners of some niche disciplines (like ours here) would like a hyperintelligent AI to do all our work for us. But even a lot of us are on the free tier of ChatGPT 3.5; I'm one of the few paying $20/mo for GPT4; and idk if even I'd pay e.g. $200/mo for GPT5.
  • deepsquirrelnet 4y ago
    > I'd bet that what he, and the competition, is realizing is that the bigger models are too expensive to run. I think it's likely that they're out of training data to collect. So adding more parameters is no longer effective. > most people don't need a brilliantly intelligent assistant, they just need a not-dumb assistant. I tend to agree, and I think their pathway toward this will all come from continuing advances in fine tuning. Instruction tuning, RLHF, etc seem to be paying off much more than scaling. I bet that's where their investment is going to be turning.
    • Tepix 4y ago
      Once they can add videos they will have a lot of new training data.
  • deleted 4y ago
    [deleted]
• boringuser2 4y ago
  Eh. Altman has a financial incentive to lie and obfuscate about what it takes to train a model like GPT-4 and beyond, so his word is basically worthless.
  • cowpig 4y ago
    OpenAI has gone from open-sourcing its work, to publishing papers only, to publishing papers that omit important information, to GPT-4 being straight-up closed. And Sam Altman doesn't exactly have a track record of being overly concerned about the truth of his statements.
    • transcriptase 4y ago
      I had a fun conversation (more like argument) with ChatGPT about the hypocrisy of OpenAI. It would explicitly contradict itself and then began starting every reply with “I can see why someone might think…” and then just regurgitating fluff about democratizing AI. I finally was able to have it define democratization of technology and then recognize the absurdity of using that label to describe a pivot to gating models and being for-profit. Then it basically told me “well it’s for safety and protecting society”. An AI, when presented with facts counter to what it thought it should say, agreed and basically went: “Won’t someone PLEASE think of the children!” Love it.
      • gowld 4y ago
        Transcripts of other people's GPT chats are like photos of other people's kids.
      • mstolpm 4y ago
        Why are you discussing OpenAI with ChatGPT? I’m honestly interested. I would imagine that any answer of ChatGPT on that topic is either (a) „hallucinated“ and not based on any verifiable fact or (b) scripted in by OpenAI. The same question pops up for me whenever someone asks ChatGPT about the internals and workings of ChatGPT. Am I missing something?
• Bjorkbat 4y ago
  Something kind of funny (but mostly annoying), about this announcement is the people arguing that OpenAI is, in fact, working on GPT-5 in secret. To my knowledge, NFT/crypto hype never got so bad that conspiracy theories began to circulate (though I’m sure there were some if you looked hard enough). Can’t wait for an AIAnon community to emerge.
  • ryanwaggoner 4y ago
    Isn’t it obvious? Q is definitely an LLM, trained on trillions of words exfiltrated from our nation’s secure systems. This explains why it’s always wrong in its predictions: it’s hallucinating!
• dpflan 4y ago
  Hm, all right, I'm guessing that huge models as a business maybe are over until economics are figured out, but huge models as experts for knowledge distillation seems reasonable. And if you pay a super premium can you use huge model.
• joebiden2 4y ago
  We will need a combination of technologies we have in order to really achieve emergent intelligence. Humans are comprised of various "subnets" modelling aspects which, in unison, produce self-conciousness and real intelligence. What is missing in the current line of approaches is that we only rely on auto-alignment of subnetworks by machine learning, which scales only up to a point. If we would produce a model which has * something akin a LLM as we know it today, which is able to * store or fetch facts to a short- ("context") or longterm ("memory") storage * if not in the current "context", query the longterm context ("memory") by keywords for associations, which are one-by-one inserted into the current "context" * repeat as required until fulfilling some self-defined condition ("thinking") To me, this is mostly mechanical plumbing work and lots of money. Also, if we get rid of the "word-boundedness" of LLMs - which we already did to some degree, as shown by the multi-language capabilities - LLMs would be free to roam in the domain of thoughts /s :) This approach could be further improved by meta-LLMs governing the longterm memory access, providing an "intuition" which longterm memory suits the provided context best. Apply recursion as needed to improve results (paying by exponential training time, but this meta-NN will quite probably be independent of actual training, as real life / brain organization shows).
  • babyshake 4y ago
    The other elements that may be required could be some version of the continuous sensory input that to us creates the sensation of "living" and, this one is a bit more philosophical, the sensation of suffering and a baseline establishment that the goal of the entity is to take actions that help it avoid suffering.
    • joebiden2 4y ago
      I think an AI may have extra qualities by feeling suffering etc., but I don't think these extra qualities are rationally beneficial.
    • ilaksh 4y ago
      That's when it gets dangerous, when we try to really recreate animal (human) characteristics in digital form. Combining that with likely 1000-1000000 X increases in performance leads to these superintelligent digital creatures taking over. Instead, we can focus on the Star Trek computer type stuff that we have with GPT and be incredibly careful about deploying those more animal/humanlike models and higher performance compute. Especially if we deliberately create the next species in digital form, make it 100X or 10000X faster thinking/smarter than us, and then enslave it, that is not only totally stupid but also proven unnecessary by the generality the latest AI models.
• gardenhedge 4y ago
  Sam Altman and OpenAI must be pretty nervous. They have first mover advantage but they hold no hook or moat. Unless they can somehow keep their improvements ahead of the rest of the industry then they'll be lost among a crowd.
  • esafak 4y ago
    On the other hand, if Microsoft can change enough people's habits of using Google to search, they'll be happy as a clam.
  • kvetching 4y ago
    Yeah. Sadly I believe they may lose their spot due to their concern for safety. They got here by pushing the envelope. But now with everyone freaking out, he's not putting off the training of a new model due to 'safety issues' (with plugins i suspect)
• ldehaan 4y ago
  This is just push back from elon and crews fake article about the dangers of AI, they specifically state the next versions will be ultra deadly. Sam is now saying there will be no future model that will be as good. This is all positioning to get regulators off the track because none of these control freaks in government actually understand a whit of this. All said and done, this all just to try to disempower the OSS community. But they can't, we're blowing past their barriers like the 90s did with the definition of slippery slope.
  • nuc1e0n 4y ago
    He isn't saying that. Only that different approaches are needed to produce better results. If he's confident to say this I think he must have something up his sleeve.
• throwaway22032 4y ago
  I don't understand why size is an issue in the way that is being claimed here. Intelligence isn't like processor speed. If I have a model that has (excuse the attempt at a comparison) 200 IQ, why would it matter that it runs more slowly than a human? I don't think that, for example, Feynman at half speed would have had substantially fewer insights.
  • yunwal 4y ago
    We're not going to get a 200 IQ model by simply scaling up the current model, even with all the datacenters in the world running 24/7
• sgu999 4y ago
  Is anyone aware of techniques to prune a model from useless knowledge to leave more space for the reasoning capabilities? It really shouldn't matter that it can give the exact birthdate of Steve Wozniac, as long as it can properly make a query to fetch it and deal with the result.
  • cloudking 4y ago
    I follow your design, couldn't you also solve hallucinations with a "fact checking" LLM (connected to search) that corrects the output of the core LLM? You would take the output of the core LLM, send it to the fact checker with a prompt like "evaluate this output for any potential false statements, and perform an internet search to validate and correct them"
• it 4y ago
  Interesting how this contradicts "The Bitter Lesson": http://incompleteideas.net/IncIdeas/BitterLesson.html http://incompleteideas.net/IncIdeas/BitterLesson.html.
  • sebzim4500 4y ago
    I don't think there is a contradiction at all. Altman is essentially saying they are running out of compute and therefore can't meaningfully scale further. Not that scaling further would be a worse plan longterm than coming up with new algorithms.
• summerlight 4y ago
  The point is that now we're at the point of diminishing return for increasing model size, unless we find a better modeling architecture than Transformer. I think this is likely true; while all the other companies underestimated the capability of transformer (including Google itself!), OpenAI made a fairly accurate bet on the transformer based on the scaling law, put all the efforts to squeeze it until the last drop and took all the rewards. It's likely that GPT-4 is on the optimal spot between cost and performance and there won't be significant improvements on performance in a near future. I guess the next task would be more on efficiency, which has a significant implication on its productionization.
  • fauxpause_ 4y ago
    Seems like a wild claim to make without any examples of gpt models which are bigger and no demonstrably better.
    • xipix 4y ago
      Perhaps (a) there do exist bigger models that weren't better or (b) this model isn't better than somewhat smaller ones. Perhaps the CEO has seen diminishing returns.
      • fauxpause_ 4y ago
        Sure. It is possible there is evidence that is not shared. It’s dumb to assume this is the case though.
    • mnky9800n 4y ago
      or like a curve of model complexity versus results or whatever showing it asymptotically approaches whatever. actually there was a great paper from microsoft research from like 2001 on spam filtering where they demonstrated that model complexity necessary for spam filtering went down as the size of the data set went up. That paper, which i can't seem to find now, had a big impact on me as a researcher because it so clearly demonstrated that small data is usually bad data and sophisticated models are sometimes solving problems will small data sets instead of problems with data. of course this paper came out the year friedman published his gradient boosting paper, i think random forest also was only recently published then as well (i think there is a paper from 1996 about RF and briemans two cultures paper came out this year where he discusses RF i believe), and this is a decade before gpu based neural networks. So times are different now. But actually i think the big difference is these days i probably ask chatgpt to write the boiler plate code for a gradient boosted model that takes data out of a relational database instead of writing it myself.
• calderknight 4y ago
  I didn't think this article was very good. Sam Altman actually implied that GPT-5 will be developed when he spoke at MIT. And if Sam said that scaling is over (I doubt he said this but I could be wrong) the interesting part would be the reasoning he provided for this statement - no mention of that in the article.
• bob1029 4y ago
  I strongly believe the next generation of models will be based upon spiking neural concepts wherein action potentials are lazily-evaluated throughout the network (i.e. event-driven). There are a few neuron models that can be modified (at some expense to fidelity) in order to tolerate arbitrary delays between simulation ticks. Using actual latency between neurons as a means of encoding information seems absolutely essential if we are trying to emulate biology in any meaningful way. Spiking networks also lend themselves nicely to some elegant learning rules, such as STDP. Being able to perform unsupervised learning at the grain of each action potential is really important in my mind. This gives you all kinds of ridiculous capabilities, most notably being the ability to train the model while it's live in production (learning & use are effectively the same thing). These networks also provide a sort of deterministic, event-over-time tracing that is absent in the models we see today. In my prototypes, the action potentials are serialized through a ring buffer, and then logged off to a database in order to perfectly replay any given session. This information can be used to bootstrap the model (offline training) by "rewinding" things very precisely and otherwise branching time to your advantage. The #1 reason I've been thinking about this path is that low-latency, serialized, real-time signal processing is somewhat antagonistic to GPU acceleration. I fear there is an appreciable % of AI research predicated on some notion that you need at least 1 beefy GPU to start doing your work. Looking at fintech, we are able to discover some very interesting pieces of technology which can service streams of events at unbelievable rates and scales - and they only depend on a handful of CPU cores in order to achieve this. Right now, I think A Time Domain Is All You Need. I was inspired to go outside of the box by this paper: https://arxiv.org/abs/2304.06035 https://arxiv.org/abs/2304.06035. Part 11 got me thinking.
  • eternalban 4y ago
    I'm bullish on SNNs too. This Chinese research group is doing something quite comprehensive with them: https://news.ycombinator.com/item?id=35037605 https://news.ycombinator.com/item?id=35037605
  • MagicMoonlight 4y ago
    I know what it looks like in my head but I can’t quite figure the algorithm out. The spiking is basically reinforcement learning at the neuron level. Get it right and it’s basically all you need. You don’t even need training data because it will just automagically learn from the data it sees.
• winddude 4y ago
  Also scaling doesn't address some of the challenges for ai that chatGPT doesn't meet, like: - learning to learn, aka continual learning - internalised memory bringing it closer to actual human capabilities.
• LesZedCB 4y ago
  the way i see it, the expensive part should be to train the models via simulated architectures in GPUs or TPUs or whatever. but once they are trained, is there a way to encode the base models into hardware where inference costs are basically negligible? hopefully somebody is seeing if this is possible, using structurally encoded hardware to make inference costs basically nil/constant.
• gumballindie 4y ago
  Bruv has to pay for the data he's been using or soon there won't be any to nick on. Groupies claiming their ai is "intelligent", and not just a data ingesting beast, will soon learn a heard lesson. Take your blogs offline, stop contributing content for free and stop pushing code or else chavs like this one will continue monetising your hard work. As did bezos and many others that now want you to be out of a job.
• iandanforth 4y ago
  There's plenty of room for models to continue to grow once efficiency is improved. The basic premise of the Google ML pathways project is sound, you don't have to use all the model all the time. By moving to sparse activations or sparse architectures you can do a lot more with the same compute. The effective model size might be 10x or 100x GPT-4 (speculated at 1T params) but require comparable or less compute. While not a perfect analogy it's useful to remember that the human brain has far more "parameters", requires several orders of magnitude less energy to train and run, is highly sparse, and does a decent job at thinking.
  • ilaksh 4y ago
    The efficiency thing is what worries me. I think Nvidia has rough ideas for increasing efficiency 100 - 1000 times without changing the fundamental paradigm (i.e. memristors or crossbar arrays or something). If it doesn't go to 1000, I assume there is a lot of investment ramping up for realizing the new fully compute-in-memory systems. If the software can leverage these efficiency gains effectively, then the concerns about runaway AI will be very relevant. Especially since people seem to think that they need to emulate all animal (like human) characteristics to get "real" general intelligence. Despite the fact that GPT is clearly general purpose. And people make no real differentiation between the most dangerous types of characteristics like self-preservation or full autonomy. GPT shows that we can have something like a Star Trek computer without creating Data. People should really stop rushing their plans to create an army of Datas and then enslave them. Totally unnecessary and stupid.
• mnky9800n 4y ago
  dude someone lied their way into being president of the united states all while people fact checked him basically immediately after each lie. lying doesnt make a difference.
  • beowulfey 4y ago
    He’s not presenting false evidence here, he’s presenting a hunch. It’s a guess. No one is going to gain anything from this one way or another.
    • Andrex 4y ago
      > No one is going to gain anything from this one way or another. Then why did Altman even bother to make this very public statement?
      • not2b 4y ago
        To answer the people asking when GPT5 is coming out, probably.
      • neom 4y ago
        CEOs often make public statements for recruiting and to steer the company internally, it's important to mimmic what you say inside the company outside the company, maybe even more so.
      • ukuina 4y ago
        Altman's made it clear he does not mind being wrong. Smart people are allowed to make mistakes.
      • beowulfey 4y ago
        He was a guest on an AI-related discussion panel [1]. I did not attend, but I would guess someone asked him “do you think models will have to continuing growing to even larger sizes,” or something like that, and this was his answer. Honestly, that anyone thinks he has “ulterior motives” here is hilarious to me. Every day I think more and more that we can no longer think critically on the internet. [1] https://www.imaginationinaction.co/ https://www.imaginationinaction.co/
• curiousllama 4y ago
  I believe Altman, but the title is misleading. Have we exhausted the value of larger models on current architecture? Probably yes. I trust OpenAI would throw more $ at it if there was anything left on the table. Have we been here before? Also yes. I recall hearing similar things about LSTMs when they were in vogue. Will the next game changing architecture require a huge model? Probably. Don’t see any sign these things are scaling _worse_ with more data/compute. The age of huge models with current architecture could be over, but that started what, 5 years ago? Who cares?
• matchagaucho 4y ago
  I wonder how much the scarcity and cost of Nvidia GPUs is driving this message? Nvidia is in a perfect "Arms Dealer" situation right now. Wouldn't be surprised to see the next exponential leap in AI models trained on in-house proprietary GPU hardware architectures.
  • TheDudeMan 4y ago
    Google has been using TPUs for years and continuously improving the designs.
• deleted 4y ago
  [deleted]
• textninja 4y ago
  I call bullshit. There will be bigger and better models. The question is not whether big companies will invest in training them (they will), but whether they’ll be made available to the public.
• stephencoxza 4y ago
  The role of a CEO is more to benefit the company than the public. Only time will tell. I am curious though how something like Moore's Law relates to this. Yes, model architectures will deal with complexity better and the amount of data helps as well. There must be a relation between technology innovation and cost which alludes to effectiveness. Innovation in computation, model architecture, quality of data, etc.
• seydor 4y ago
  Now we need another letter
• mromanuk 4y ago
  Sorry, but this sounds a lot like 640KB is all the memory you will ever need. What about “Socratic model” for video? There should me many applications that would benefit from a bigger model
• rhelz 4y ago
  All warfare is based on deception -- Sun Zu
• narrator 4y ago
  "Altman said there are also physical limits to how many data centers the company can build and how quickly it can build them." An H100 has 80GB of VRAM. The Highest end system I can find is 8xH100. Is a 640GB model is the biggest model you can run on a single system? Already GPT-4 is throttled and has a waiting list and they haven't even released the image processing or integrations to a wide audience. Maybe they are just unable to do inference in a cost-efficient manner and at an acceptable speed on anything bigger than GPT-4?
  • moralestapia 4y ago
    >Maybe they are just unable to do inference in a cost-efficient manner and at an acceptable speed on anything bigger than GPT-4? Yes, that's the whole thing. As others have pointed out, GPT-4 seems like an optimum point that balances cost, ROI, etc... it won't improve much just by throwing more data at it.
  • forrest2 4y ago
    You might not see it available via self serve interfaces, but I think the h100 cards can do 256x to a single host.
    • narrator 4y ago
      You're right! You can nvlink 32 systems with 8 h100s! That's 20 terabytes of VRAM total!
• uejfiweun 4y ago
  To those in the know, what are the potential successors to the Transformer architecture?
  • intalentive 4y ago
    Good ideas here: https://youtu.be/mBjPyte2ZZo https://youtu.be/mBjPyte2ZZo
• tabtab 4y ago
  GPT4 already obsolete? Geez, we got the wrong bot, silly heads: https://www.youtube.com/watch?v=EdRMVhlNP5I https://www.youtube.com/watch?v=EdRMVhlNP5I
  • ChatGTP 4y ago
    It’s a good things there bots don’t have feelings yet because there’s going to be an lot obsolete bots once we’re done and getting made obsolete isn’t very fun.
• htk 4y ago
  Maybe they're now training teams of AIs to work together? This way the hardware limitations are less of a barrier in terms of maximum memory per AI. They can run in parallel in separate hardware and communicate to reach higher goals.
  • dumbmrblah 4y ago
    Someone described me a emerging rule called “Penelope‘s rule” where you scale by having the largest and most powerful model delegate to the less powerful models and then have the larger model improve upon the results. (Eg have GPT-4 delegate the more mundane tasks to GPT-3.5).
• siliconc0w 4y ago
  It's probably economically difficult to rationalize building a GPT5. They're already having a hard time getting everyone GPT4. Better to spend the resources optimizing to get everyone onto a cheap(er) GPT4-comparable model than build a marginally better model that is even more expensive to scale. Like most tech there is a sigmoid curve where you start to see diminishing returns.
• lz400 4y ago
  These kind of comments have me update my priors so I downgrade them in relevance. Two days ago we were talking about AGI and Altman was letting himself be cuddled, knowing full well it was mindless hype. Now he's trolling the other way. Ok. My weights have been updated.
• deleted 4y ago
  [deleted]
• throwaway14356 4y ago
  42 train the entire model for a single question.
• kmod 4y ago
  I don't quite understand this. The returns to scale have always been sublinear (ie "diminishing"), but the scale-maximalists didn't worry about that before. Also I heard Sam tout on the Lex Friedman podcast how amazing it is that the scaling laws worked so well for GPT-4. So I wonder what changed? Sure yeah the cost numbers are getting very large and we can't keep scaling forever. But Google could easily 10x the training cost of GPT-4 if they thought it would protect their search business. I'm still skeptical that scaling is enough to reach the thresholds we want, but I'm surprised that it's being claimed right now when there's a huge rush of new money into the space. I wonder if this is some sort of misdirection by Sam
  • teruakohatu 4y ago
    > So I wonder what changed? Maybe cost & latency for both training and inference it getting too high. If costs doubled for every 5% better performance, would it be worth it? NVIDIA is making a small fortune from this. > But Google could easily 10x the training cost of GPT-4 if they thought it would protect their search business Google makes /\$0\.[0+]\d/ per search query. If the inference cost of the model exceeds that, they go from making money to losing money. It is not clear if the Bing integration is a money maker or a lost leader.
• bla3 4y ago
  In other words, "this approach is getting to expensive for us and we need to control the narrative for our model sizes going down". Most increases in Imagenet etc scores came from bigger models. "The Unreasonable Effectiveness of Data" has aged very well. It seems very convenient for OpenAI that this trend should be over a few months after their "eye-wateringly expensive" comment.
  • kvetching 4y ago
    They could have it open source and people like Gerganov could rewrite the code in C/C++, but god forbid they actual are open
  • anthlax 4y ago
    SamA strikes me as overly disingenuous - as if most of his interactions are marketing pieces / warfare. The entire lex + sama podcast seemed this way to me. I’m incredibly biased though and feel slighted (as a part of humanity) by openAI’s actions, so perhaps I’m looking at his actions through a hateful lens.
    • mclightning 4y ago
      I feel the same. His narrative seem very calculated. It doesn't help that OpenAI is very much the most closed Saas thing to be released in recent years. I am a paying customer for sometime, have ChatGPT Plus as well. I am still in waiting list for GPT-4 API. Not only that, but their business model is completely unclear, which is the scariest part for me, as an individual developer. Let's go one step forward, they use our texts online including this very comment perhaps. They have a software that can analyse all our texts at scale. But we don't get even the API access to this thing. It is the most self-serving software ever released, consuming the human creativity both the content and the future relevance of it.
• ramshanker 4y ago
  Wait till some one pulls a Cerberus for MEMORY. And than we connect a wafer scale processor with wafer scale memory and BOOM. New Mega size works emerge. As for current claim, it might have to do with the amount of time spent taming the wildness of answer by raw GPT-4. So focus is shifting from increasing the model size.
• deleted 4y ago
  [deleted]
• smrtinsert 4y ago
  No one will ever need more than 64k moment or whatever it was
• m3kw9 4y ago
  Sounds like we will be on gpt4 for a while, while we wait out a new “AI winter”, mind you there is still a lot to squeeze from gpt4 and local/edge llm models
• shironandon 4y ago
  "The great supercomputer Deep Thought is the most powerful computer ever built, with one exception. It was designed by hyper-intelligent pan-dimensional beings, who wanted to know the answer to Life, the Universe, and Everything. Its creation annoyed a fair few philosophers, who felt that it was taking over their turf. After seven and a half million years of serious cogitation, Deep Thought spoke the answer. However, it was so inexplicable that Deep Thought then had to go on and design the most powerful computer ever built (with no exceptions) to work out what the question was."
  • iamgopal 4y ago
    And that was earth, so are we asking the right questions?
  • vinte 4y ago
    and the answer to question of life is 42.
  • ricochet11 4y ago
    Feels very much like we are entering the Happy Vertical People Transporters era. How long till the air con goes on strike for miserable working conditions? "Brain the size of a planet and they ask me write a lesson plan in the style of a pirate" - chatgpt5, probably...
• pffft8888 4y ago
  You can't trust what sama says. Period. He's shown his willingness to deceive for gain. He deleted the codex model because it was better than the text model and only way to explain it is that by training on code instead of just text of human interactions the model develops stronger reasoning abilities (because of higher clarity around logic and higher signal to noise overall in the training data.) So he removed that so researchers don't catch on. Also, why 1 Trillion params is the limit? Why not 2 Trillion. He's been begging for regulations and trying to pull up the ladder behind him. Just really "street smart" thinking but I have not seen any elevated humanity serving thinking coming form him. He would not have taken $100M for a non-profit to do open source research only to bait-and-switch into for profit closed source sold to the highest bidder. Give me a break. Don't believe anything he says (well, use your brain, as he may mix truths and deceptions with the goal of deceiving his competitors... he's not out there to do a service for humanity... if he was, he would have kept it open and non-profit, not basically steal the funds of the non-profit to pocket $29B from it.) It's ridiclous.
• ironfootnz 4y ago
  He's finally catching up with me. I wrote an article about that a few months ago. https://twitter.com/igorcosta/status/1612704076315389952 https://twitter.com/igorcosta/status/1612704076315389952
• SanderNL 4y ago
  Those stochastic parrots are done for, finally! I have been telling people AI is a lie and it’s all a scam and now you can see for yourself. Buy nice warm jackets guys, winter is coming.
• electrofried 4y ago
  LLM's will be treated much like occupations for people going forward. Just like our own education, there will be a basic set of skills all LLM's need to be trained on to provide a minimal useable product. With GPT4 I suspect we are at this point now if not slightly beyond the optimal basic training stage. However, now it is time to send our LLM's off to university so to speak, to specialise them in certain tasks and knowledge base. The current approach of attempting to just build everything in to one does not work for a human mind, so why would it work for a vector based replication? I suspect in the coming months we will hear more about tiny models trained on much smaller datasets and then specialised using a mix of adaptors and LoRA modifications to excel at specific tasks like code generation, translation, and conversation. Then multiple models will be implemented in one application chain to best leverage each of the respective strengths.
• Dowwie 4y ago
  "640K of memory is all that anybody with a computer would ever need"
• luckystarr 4y ago
  There will still be giant AI models, they will just be improved by other means than making them bigger. Subtle but important distinction.
• Dave3of5 4y ago
  Yeah I think what's happened to OpenAI is the same happening to all tech companies. They were spending a lot of money building these LLM's and now they've been told to cut back costs they no longer have the funding to build the next "double the size of the last one" model. Interestingly what's makes ChatGPT work is the size of the model so I think they've found their dead stop. Lot's of the bullish comments have been talking about how ChatGPT is a bit shit right now but will exponentially get better and I think the answer is now the progress will be much slower and more linear. That is if they can stay funded which is a very big if. As an org they are bleeding money.
• ionwake 4y ago
  Hmmmmmm I know nothing about this but… guy who spends a bunch of money on thing to sell says others shouldn’t do it. Sounds sus Sorry I know this adds little to the conversation
• morelight18 4y ago
  [dead]
• angelhearts9 4y ago
  [dead]
• smashfizzle7 4y ago
  [dead]
• monsoon11 4y ago
  [dead]
• blooms112 4y ago
  [dead]
• floufrouu11 4y ago
  [dead]
• morelight19 4y ago
  [dead]
• sharinggenes1 4y ago
  [dead]
• indigosparkle1 4y ago
  [dead]
• advocates19 4y ago
  [dead]
• advocates19 4y ago
  [dead]
• selfcare111 4y ago
  [dead]
• yourblanks17 4y ago
  [dead]
• butterflyeffec 4y ago
  [dead]
• lifebutterfly 4y ago
  [dead]
• livincool87 4y ago
  [dead]
• selfcare11 4y ago
  [dead]
• reading87 4y ago
  [dead]
• flightsnot81 4y ago
  [dead]
• doodles675 4y ago
  [dead]
• indigosparkle7 4y ago
  [dead]
• morelight876 4y ago
  [dead]
• monsoon654 4y ago
  [dead]
• indigosparkle6 4y ago
  [dead]
• fabulous13 4y ago
  [dead]
• champagne7 4y ago
  [dead]
• flightsnot45 4y ago
  [dead]
• styleslover123 4y ago
  [dead]
• isonfire11 4y ago
  [dead]
• shop13213 4y ago
  [dead]
• aholicdiva 4y ago
  [dead]
• workofgod132 4y ago
  [dead]
• pink1321 4y ago
  [dead]
• lusttforlife567 4y ago
  [dead]
• thatsassy127 4y ago
  [dead]
• lwithnojob65 4y ago
  [dead]
• inkandpaper1 4y ago
  [dead]
• gaming56 4y ago
  [dead]
• withnojob0 4y ago
  [dead]
• kong45 4y ago
  [dead]
• goddess09 4y ago
  [dead]
• gypsy645 4y ago
  [dead]
• wanderlust56 4y ago
  [dead]
• chicnext673 4y ago
  [dead]
• door7544 4y ago
  [dead]
• foodie564 4y ago
  [dead]
• lyjules464 4y ago
  [dead]
• nitch65854 4y ago
  [dead]
• avocados564 4y ago
  [dead]
• betches634 4y ago
  [dead]
• betches634 4y ago
  [dead]
• diet7865 4y ago
  [dead]
• deleted 4y ago
  [deleted]
• prada574 4y ago
  [dead]
• betch8975 4y ago
  [dead]
• velvet6755 4y ago
  [dead]
• canyon754 4y ago
  [dead]
• collective765 4y ago
  [dead]
• hipjiveguy 4y ago
  dudes. if none of you see it, it's because you're turning a blind eye, is the best way I can say it. the future of the 50's is about to show up, if, and only if, we're not so dumb as to ignore it. flying cars, killer robots, we can help decide, if we're only aware enough to do a bit of research, and say anything. your voice does matter :D watch this: https://www.youtube.com/watch?v=VcVfceTsD0A&ab_channel=LexFridman https://www.youtube.com/watch?v=VcVfceTsD0A&ab_channel=LexFr... then, think, and then, think again, and then comment be optimistic, but be forewarned, and speak your voice
• lee101 4y ago
  [dead]