9 ms·
There is a startling acceleration of innovation in the field that GPT-4 illustrates. According to NVidia, LLM sizes have been increasing 10X per year for the la
by mmaunder 4y ago
There is a startling acceleration of innovation in the field that GPT-4 illustrates. According to NVidia, LLM sizes have been increasing 10X per year for the last few years. This tech is going to hit every aspect of society like a sledgehammer over the next 48 months.
- curiousgal 4y agoI guess you perception of society is severely limited if you think a fancy autocomplete is capable of changing every aspect of it.
- maxdoop 4y agoI have fun on these HN chats responding to comments like yours . It’s just fancy auto complete to you? You honestly can’t see the capability it has and extend it the future? What’s that saying about “it’s hard to get someone to understand something when their salary depends on their not understanding it”.
- HervalFreire 4y ago[flagged]
- _dain_ 4y ago> When these LLMs get normalized probably 5 years from now I'm going go back to these old threads and contact these people who are in self denial and throw it in their face. I'll just link this comment and be like I TOLD YOU, I TOLD YOU, YOU WERE WRONG. this isn't a kind thing to do.
- HervalFreire 4y ago[flagged]
- Jeff_Brown 4y agoNor rational. What a strange impulse.
- HervalFreire 4y ago[flagged]
- blurbleblurble 4y agoI feel very frustrated with these takes because instead of grappling with what we're going to do about it (like having a conversation) it's a flat, dismissive denial, and it isn't even grounded in the science, which says that "memory augmented large language models are computationally universal". So at the very least we're dealing with algorithms that can do anything a hand written program can do, except that they've been trained to do it using natural language in extremely flexible ways. I'm having a hard time seeing how "fancy autocomplete" is the right description for this.
- maxdoop 4y agoI agree 100%. I don't understand why we can't look at the potential, or even current, capabilities of these LLMs and have a real conversation about how it might impact things. Yet so many folks here just confidently dismiss it. "It doesn't even think!" -- OK, define thinking? "It doesn't create novel ideas!" OK -- what do most devs do every day? "It is wrong sometimes!" OK -- is it wrong more or less often than an average dev?
- spacemadness 4y agoIn case we forget the amazing predictive capabilities of HN there’s always https://news.ycombinator.com/item?id=9224 https://news.ycombinator.com/item?id=9224
- enord 4y ago«Computationally universal»? Are you quite sure? Hoo boy pass the nitroglycerin hooooo aaah ohh after 80 years Turing was proven right after all ghasp
- tel 4y agohttps://en.wikipedia.org/wiki/Universal_approximation_theorem https://en.wikipedia.org/wiki/Universal_approximation_theore...
- blurbleblurble 4y agoWell I don't understand the nitroglycerin reference and of course "computationally universal" doesn't mean "sentient" but the point is when you add the external memory they (Flan-U-PaLM 540B to be specific) have been demonstrated to be capable of simulating a specific, well defined Turing machine without special training. There are some other papers out there arguing this from a theoretical angle too, but this is the one whose title I quoted: https://arxiv.org/abs/2301.04589 https://arxiv.org/abs/2301.04589
- MVissers 4y agoFunny- That’s like saying the iPhone is a nothingburger. Who would buy such thing right? Or the internet? Who’d even want a computer at home to talk to other computers. This is going to infiltrate every text-based tool. Email, social media, text. Speech and video already incoming.
- bena 4y agoYes and no. But you're putting forth a "Bozo argument" by noting that that people also shit on the iPhone. "But the fact that some geniuses were laughed at does not imply that all who are laughed at are geniuses. They laughed at Columbus, they laughed at Fulton, they laughed at the Wright brothers. But they also laughed at Bozo the Clown." - Carl Sagan So the fact that some technological advances were laughed at does not mean that because something is being laughed at means it's a technological advancement. Basically all you've done is say you disagree with the poster. But neither of you have given any sort of reasoning for your positions. Personally, I agree with your final conclusion. Even as a "fancy autocomplete", this thing is pretty fucking fancy. To toss ChatGPT a basic idea and get back a multi-paragraph text on that idea that only needs to be proofread is pretty choice. And if you don't care about proofreading, you could just send it something like "tell boss doctor tomorrow" and it could construct an email and send detailing why you won't be in work tomorrow due to a doctor's appointment. Human communication just got a bit more complicated.
- antibasilisk 4y agoI've been seeing alot of this 'nothing ever happens' attitude lately, and I find it very odd. I feel like people are using it to cope with the rapid pace of change we're experiencing.
- tarruda 4y agoThat was what I thought until a few months ago when ChatGPT was released. I never cared much about LLMs because it always felt like a brute force method to solving problems. What I'm seeing now is that some kind of intelligence seem to emerge from these models, even though under the hoods it is just a bunch o matrix multiplications. Who's can say for sure that our own brains doesn't work similarly? Maybe human intelligence is something that emerges from similar primitives and AGI will simply be a really large language model.
- throwbadubadu 4y agoOh man... if it will be as stupid as us humans we are completely doomed. For now lets assume what it most likely is: not emerging anything, just a very good simulation of that emergeance. I mean we already did stupid chat bots in the 90s with convoluted if-else-spaghettis that imitated Eliza. That could already hold for some sentences... GPT-4 now holds much longer, but still you can easily drive it to the point where it starts "looping" and not provide much more value, especially if you are more expert => makes me so far hope that this length will grow more, but still.. in my "sessions" did not (gladly) find anything emerging beyond the first wow and surprising answers... not even close to that.
- tarruda 4y agoI have very limited understanding of how these systems are created, but as far as I know they are not programmed to simulate anything. It is just a system that outputs the most likely next word based on a really large dataset of training samples it was given. Yet you are still able to "chat" with these systems. Here's a session I just ran on llama.cpp (7 billion parameter version). Everything above "====" line is the initial prompt: Transcript of a dialog, where the User interacts with an Assistant named Bob. Bob is helpful, kind, honest, good at writing, and never fails to answer the User's requests immediately and with precision. User: Hello, Bob. Bob: Hello. How may I help you today? User: Please tell me the largest city in Europe. Bob: Sure. The largest city in Europe is Moscow, the capital of Russia. User: ==== Who are you? Bob: Who am I? I'm a virtual assistant, or a VA, for short. User: Do you have a name? Bob: Yes, I do. My name is Bob. User: How old are you? Bob: I am 19 years old. User: Do you know what is "Hacker News"? Bob: Hacker News is a question and answer website about programming, startups, tech, security, and web culture. This is a super simple example. Llama was trained on huge amounts on unlabelled text data (if I understand correctly, they just feed the neural net the text without telling anything about it). Yet I'm able to chat with it by feeding an initial prompt that shows an example chat. Isn't this an emergent property?
- ben_w 4y ago""" That riled me, but I held my temper as I asked, “What’s the matter with you?” He bellowed: “There’s nothing the matter with me. It’s you! You’re crazy if you think this fool contraption you’ve been wasting your time on will ever displace the horse.” """ - https://www.saturdayeveningpost.com/2017/01/get-horse-americas-skepticism-toward-first-automobiles/ https://www.saturdayeveningpost.com/2017/01/get-horse-americ...
- blurbleblurble 4y ago"Memory Augmented LLMs are Computationally Universal"
- deleted 4y ago[deleted]
- legulere 4y agoThe same was said 10 years ago. It's astonishing what can be done, but you can already see fundamental limits. I think it will raise productivity for some tasks, but not fundamentally change society.
- danielbln 4y agoCan you elaborate on fundamental limits?
- maxlamb 4y agoI’m guessing one is data. The limit would be once you’ve trained a LLM on all public (or even private) data. Sure you can still make some improvements or try to find some additional private data but still, a fundamental limit has been reached.
- macrolime 4y agoIs it even feasible any time soon to train an LLM on all of YouTube?
- danielbln 4y agoNapkin math, assuming around 156 million hours of video on all of Youtube: 156 million hours of YouTube videos 9,000 words/hour 6 characters/word (including space) First, let's find out the total number of characters: 9,000 words/hour \* 6 characters/word = 54,000 characters/hour Now, let's calculate the total number of characters for 156 million hours of YouTube videos: 54,000 characters/hour \* 156,000,000 hours = 8,424,000,000,000 characters Since 1 character is typically 1 byte, we can convert this to gigabytes: 8,424,000,000,000 bytes / (1024 \* 1024 \* 1024) ≈ 7,842.11 GB So, 8TB of text? Seems doable.
- macrolime 4y agoI mean the actual video, that's much bigger With a vision transformer each token may be around 16x16 pixels. I found an example where they use images of resolution 224x224 for training a vision transformer so if we go with that that 256 pixels per token and 50176 pixels per image, so 196 tokens per frame, 24 frames per second, that's 4704 tokens per second or 16934400 token / hour. In total we're at 2.6x10^15 tokens. GPT-3 was trained on 5x10^11 tokens, so YouTube done this way would be around four orders of magnitude more tokens that GPT-3 was trained on. GPT-3 was undertrained by 1-2 orders of magnitude, so the compute required to trained a model on YouTube would then be around 6 orders of magnitude higher than what was used to train GPT-3, so about one million times more. I did a linear regression on the training costs from cerebras(1) and came up with the formula (1901.67366*X)-197902.72715 where X is number of tokens in billions. Plugging in 5x10^15 tokens we get a training cost of 5 billion dollars. I guess a lot of optimizations could be done that would decrease the cost, so maybe its doable in a few years. 1. https://cirrascale.com/cerebras.php https://cirrascale.com/cerebras.php