4 ms·
Intel Announces Aurora GenAI, ChatGPT Competitor with 1T Parameters
- jjcm 3y agoHas there been any official confirmation of the number of parameters used in gpt4? Pages around the web seem to have conflicting reports on the number, ranging from 1T to 170T, but there doesn't seem to be anything official.
- mchusma 3y agoJust to expand your lower bar, some people suspect it is not much bigger than GPT 3.5 @ 175B parameters, but this is also speculation. I don't think there has been anything official.
- rileyphone 3y agoGiven how PALM2 had less params than the first one (540B) I think that makes sense.
- euclaise 3y agoCan't be less though, the latency difference between them implies that GPT-4 is significantly larger. This is unlike with PaLM, where latency decreased.
- macrolocal 3y ago1.4T according to Michael Douglas [1]. [1] https://youtu.be/B3Az2EONCHE https://youtu.be/B3Az2EONCHE
- belter 3y agoThat is correct. The open science made at OpenAI managed to publish a 100 page GPT4 technical report without using the word parameters one single time :-) ( I checked ).
- timhigins 3y agoAccording to Intel's own press release, "Argonne is spearheading an international collaboration to advance the project, including Intel; HPE; Department of Energy laboratories; U.S. and international universities; nonprofits; and international partners, such as RIKEN." Maybe the headline should say "Argonne Labs announces GPT-4 competitor" https://www.businesswire.com/news/home/20230522005289/en/Intel’s-Broad-Open-HPCAI-Portfolio-Powers-Performance-Generative-AI-for-Science https://www.businesswire.com/news/home/20230522005289/en/Int...
- alberth 3y agoIt’s interesting how it took OpenAI 7-years to produce GPT4, and in the matter of a few months Google/Intel/etc have released on par offerings. I wonder how much of a competitive moat actually exists in this space.
- deleted 3y ago[deleted]
- gberger 3y agoThere is no moat, according to Google https://news.ycombinator.com/item?id=35813322 https://news.ycombinator.com/item?id=35813322
- Sai_ 3y agoI think OpenAI is building a moat right in front of us through their plug-ins idea. Executed well, plug-ins could become ChatGPT's App Store equivalent. Once that happens, OpenAI is undoubtedly going to convert a portion of search traffic into agent delegated work i.e., people will use ChatGPT to start and terminate their searches essentially delegating the work of collating data from multiple sources to ChatGPT. FD - I've applied to release a ChatGPT plug-in and this space looks very interesting to me. PS - Earlier today, I submitted to HN a link to my blog where I analyzed plugins. The HN post sank without a trace but the I wrote my blogpost using the Wolfram plugin in ChatGPT and it was a breeze. I genuinely feel I've seen the future.
- r00fus 3y agoCompeting app stores are a thing. Apple's moat wasn't the App Store. It was better quality hardware and a unified experience. I have no doubt that large players can possibly lap OpenAI in terms of ecosystem surrounding a decent AI core. However, OpenAI has the leg up and it's their game to lose at this point.
- Sai_ 3y ago> Competing app stores are a thing This is only a factor if the underlying AI/LLM model becomes as good as OpenAI AND serves millions of people. While open source LLMs are soon going to gain parity with GPT-4/Bard, it is unlikely that they will hit the kind of usage numbers that an OpenAI/Google/Meta can deliver. Even there, I'm not sure Google/Bing will cannibalize their own product to allow third parties to inject data into a search interaction. ChatGPT is a fundamentally different product - it's a humanlike intelligence which is always ready to talk to you and assist you. We haven't ever had anything of comparable quality and reach. The nearest was Alexa but even she was limited to Amazon's catalog and shopping. It's like ChatGPT can be everyone's personal gofer and THAT is huge, imo.
- justinclift 3y ago> ... the target size for the new model is 1 trillion parameters. > Meanwhile, the target size for the free & public versions of ChatGPT is just 175 million in comparison. That's a 5.7x increase in number of the parameters. 5.7x seems off by several orders of magnitude, or do I need more caffeine?
- mediaman 3y agoIt's a typo in the article. GPT-3.5 is 175bn parameters, not 175 million. GPT-4 does not have publicly available parameter count, but is thought to have something similar to 1tn.
- justinclift 3y agoCool, thanks. :)
- redeux 3y agoThere is an obvious typo in the article. ChatGPT (i.e. GPT 3.5) has 175 billion parameters, no 175 million.
- simonw 3y ago"Announced at today's ISC23 keynote, the Aurora genAI model is going to be trained on General text, Scientific texts, Scientific data, and codes related to the domain." So this thing doesn't actually exist yet?
- barthelomew 3y agoYes, it does seem like it's concept stage.
- tpmx 3y agoI can do a decent Daniel Gackle (dang) impression: This looks like an announcement of an announcement, which isn't on topic for HN. https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sort=byDate&type=comment&query=%22announcement%20of%20an%20announcement%22 https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor... On HN, there's no harm in waiting for the actual thing: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&sort=byDate&type=comment&query=%22no%20harm%20in%20waiting%22%20by:dang https://hn.algolia.com/?dateRange=all&page=0&prefix=false&so...
- phkahler 3y agoWhy train on scientific texts when half the papers are BS?
- randomname93857 3y agoit would be good to train LLMs to distinguish BS articles from good ones. but it's rather unrealistic expectations
- A4ET8a8uTh0 3y agoI can only assume someone is working on a BS detector. And holy fuck, I would pay cash money for a way to identify it without having delve into it myself. Imagine not having to dive into paper only to find out that the population sample is so low that it shouldn't even be considered for submission or better yet have online news qualified as 'propaganda beneficial to side x'.
- gigatexal 3y agoso what's the api/tech powering the front end to this LLM api running on this super computer? and who pays for it?
- techwizrd 3y agoI'll be interested to see how this performs if/when it is released. I'm not convinced that simply increasing the number of parameters improves the models, and I don't think we should be putting out press releases "spec-chasing" increases in the number of parameters. I've also found in my research that larger models often do not perform better, but this is becoming more difficult to explain to non-technical folks due to irresponsible marketing. If we aren't careful about the bold marketing claims, we'll disillusion many people and reduce the potential for future growth—we don't want another AI Winter. There's a significant environmental cost to training these large models, and it's harder to understand, use, and control these very large models (even as a researcher).
- EGreg 3y agoWhat happened to “the scale hypothesis” LOL. Is scale really is all you need? Maybe this model can invent time travel and improve itself in the past..
- londons_explore 3y ago> I'm not convinced that simply increasing the number of parameters improves the models At every size people say this... And then someone comes out with a model 10x larger which outperforms the previous state of the art. Sure, you need to still take a little care with architecture choice and hyperparameter selection, but size really does seem to be king.
- weatherlite 3y ago> we don't want another AI Winter Speak for yourself...
- Der_Einzige 3y agoParameter count is strictly better IF the number of tokens (and ideally better quality tokens) trained on increases, and if the training is done for longer (most LLMs are way undertrained) Most of the huuuuuge models failed on most or all of these fronts and that's why they suck compared to Llama or Alpaca or Vicuna
- 3y ago
- londons_explore 3y agoThis kind of news isn't of interest until the model is made and tested side-by-side with GPT-4 and shown to at least be in the same ballpark of performance. So far, nobody has claimed to exceed GPT-4's performance (either by academic metrics or by youtubers subjective evaluations)
- sylware 3y agoI still don't know what are neural net "parameters". inputs+outputs? Number of neurons+synapes?
- phkahler 3y agoSynapse weights.
- sylware 3y agoHow are they distributed? How many neurons levels? And still, how many inputs and outputs has their neural net? Well, how are they "connected"? Their topology?
- jnwatson 3y agoThat is confidential information.
- PoignardAzur 3y agoLoot into "transformers". They're the architecture everybody is using these days; and it's relatively simple.
- sylware 3y agoUnable to get a bird eye of the connectome with those transformers. Something is not right here.
- immichaelwang 3y agoIsn't there more and more research coming out that at a certain point (200B~), parameters have significantly decreasing returns and it's better to just then do some supervised learning ontop of the base model?
- gmadsen 3y agocan you point to any of that? my understanding is that we haven't reached any dimishing returns with transformer models and scaling yet
- CuriouslyC 3y agoParameters don't have diminishing returns so much as we don't have enough (distinct) data to train models to use that many parameters efficiently.
- mirekrusin 3y agoIt smells like somebody forgot to flush PR from it. For starters future tense is embarrassing in this area. By the time they finish it, it may very likely be outdated. Also number of parameters is not all you need. It's enabler, yes, but you need tons of other things to fit together.
- londons_explore 3y agoIntel could just make every CPU/GPU they manufacture have a 'burn in test' for a week or so where it runs a bunch of self tests and then trains a massive network for them. If sales slow down, instead of having the excess stock in a box in a warehouse, have it training their neural nets.
- GaggiX 3y agoI don't see any reason why someone should train a 1T model, they probably don't have a big enough dataset to make it worth it, so with the same compute or less you can get a better model by training a smaller one (the chinchilla scale).
- davidkunz 3y ago> the target size for the new model is 1 trillion parameters. Meanwhile, the target size for the free & public versions of ChatGPT is just 175 million in comparison. That's a 5.7x increase in number of the parameters. Did nobody notice?
- htrp 3y ago>The foundations of the Intel Aurora genAI model are Megatron and DeepSpeed. Intel is PLANNING to train a genAI model with a target of 1bn parameters... using an architecture from 2019
- deleted 3y ago[deleted]
- godelski 3y agoThere's questions about why this announcement matters if it isn't trained. There's a few: 1) This is going to be using a publicly funded computer, which is the most powerful in the world AND was also announced today (btw, the specs are better than what was initially planned). The program gives justification for the machine and that public money (though in comparison to many things governments spend money on, this is very cheap). 2) THAT'S A LOT OF GPUS. This will, as far as I'm aware, be the biggest training every. GPT4 was supposedly trained on 10k A100s. Remember that Summit, the #3 computer, has >27k GPUs (V100s), but Aurora has almost 64k. This is going to be a big engineering feat in of itself. Now it won't make training 6x faster, but it will definitely be much better. It does say that the US government is taking this very seriously. That's the point of this kind of announcement: marketing, propaganda, and cool engineering project.
- ftxbro 3y agosomething to keep in mind is this datacenter will be doing so many other projects as well. it's not a dedicated LLM maker. Maybe the LLM training won't have access to all those GPUs, or maybe it will have only time sharing access split with the other projects.
- godelski 3y agoNot really a datacenter, it is a supercomputer. And yes, it is important to keep in mind that there aren't just single users. But with almost 64k nodes it isn't unreasonable for multiple teams to get 10k+ nodes at a time. I've seen runs on Summit use >50% capacity. Let's be clear, the teams aren't just going to start off using all the nodes at once. You step up.
- ftxbro 3y ago> Not really a datacenter, it is a supercomputer. ok but when openai made one it was called a datacenter
- ppaattrriicckk 3y agoI love bashing Intel as much as the other person, but looking at the number of tokens LLaMA[1] was trained on (1,400B tokens), the number of parameters of GPT-3[2] vs. the number of tokens it was trained on (175B params vs. 300B), it's not like the announced 1,000B param model is unreasonable. Taking 175/300*1400 yields 816B parameters - which is fairly close to 1,000B. Not outright the most efficient utilization of data, but as others have mentioned, there might still be something left to gain (from not solely optimizing for compute). E.g. look at Figure 9 in [3]. Although the model trained on the largest model obviously utilized the data most efficiently, it's not perfectly clear - to me at least - whether some over-parameterization will necessarily lead to a decrease of test/out-of-sample performance. Of course, LLaMA was trained on way fewer parameters relative to data set size. I've just mentioned LLaMA as a point of reference to the largest dataset known to me. [1] - https://arxiv.org/pdf/2302.13971.pdf https://arxiv.org/pdf/2302.13971.pdf [2] - https://arxiv.org/pdf/2005.14165.pdf https://arxiv.org/pdf/2005.14165.pdf [3] - https://arxiv.org/pdf/2001.08361.pdf https://arxiv.org/pdf/2001.08361.pdf EDIT: Formatting of references
- segmondy 3y agointel can't even in hardware, and somehow they think they can do so with software? Right! This is just for the clueless investors.