12 ms·
ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model
- haint_ 4y agoFrom the provided example: Q: How would I make for loop in python? A: I can help you create an AI chat bot. It would talk to you like a human. (additional text that is not relevant to the prompt) It is just me or this does not seem right?
- zaptrem 4y agoThis is only a 1.5b parameter model. This is in line with that. GPT3.5 is ~175b params.
- cztomsik 4y agoGive it at least few examples. ~1B networks are not good in zero-shot. Also, don't expect to get answers for things it was not trained on. the_pile is not programming dataset. RWKV is important because it's fast, it can be trained in parallel and it gives very good results (compared to other networks trained on the same dataset).
- deleted 4y ago[deleted]
- totoglazer 4y agoThis might be an interesting language model. However people care about ChatGPT entirely due to its quality, which this doesn’t demonstrate yet.
- phist_mcgee 4y agoThe leap in public exposure wasn't so much GPT3 to GPT3.5, it was attaching a clean UI to the model, (with sane defaults) and allowing people to talk to it like a person. Suddenly it became something 'real' then. (This is purely talking about the public popularity of GPT)
- junipertea 4y agoThey also did reinforcement learning on top of a frozen trained model. It is considerably more than just attaching a UI as that would just finish sentences compared to answering questions. https://huggingface.co/blog/rlhf https://huggingface.co/blog/rlhf
- totoglazer 4y agoNo. ChatGPT’s UI is incredibly simple and basically exactly what ever chat bot test repl looks like. The delta of GPT3 -> ChatGPT is from the expanded context and control the model offers through fine tuning. Eg read the instructgpt paper to see the path on the way to ChatGPT.
- TJSomething 4y agoOne of the important parts of ChatGPT over plain GPT-3 is the reinforcement learning from human feedback to ensure alignment, without which it's not quite as good of a product for the public.
- tinsmith 4y agoThis is a remarkably good take that just didn't dawn on me until I read your comment. Even if ChatGPT had a lesser quality than the current iteration, the fact that they had a way for anyone to easily interact with it really was a homerun, snd can be for any software, really.
- b33j0r 4y agoMy family told me that ChatGPT came up from the pulpit AT CHURCH Me? I made a few comments like a scared luddite when ChatGPT solved two of my outstanding engineering problems instantly. I got better. But this is exactly right. The world in general now knows about AI and ML. It’s a pivot point. When something scares a seasoned engineer for a minute, and anyone can now make use of this… write it down in your diary as a moment in history
- sinuhe69 4y agoI wonder what kind of engineering problems you’ve talked with ChatGPT. Because its limited understanding of spatial problems, I found it hard to use it for mechanical engineering. How did you do it?
- deleted 4y ago[deleted]
- nl 4y agoFor those wondering how on earth they are getting decent results from a RNN without long range forgetting, I don't really know either! But they reference https://arxiv.org/abs/2105.14103 https://arxiv.org/abs/2105.14103 and the bottom section of https://github.com/BlinkDL/RWKV-LM https://github.com/BlinkDL/RWKV-LM has an explainer.
- beernet 4y ago[flagged]
- deleted 4y ago[deleted]
- leodriesch 4y agoThe readme does not seem to be geared towards people not familiar with the topic. My questions: - Is this on the run on consumer GPU scale, or run on 8 A100 scale or you can’t run it yourself ever scale? - How does it compare to other language models in quality/abilities? - What is the training data?
- zone411 4y agoThe large model weights are 14B, so at 16 bits per weight, it won't quite fit on one 3090 or 4090.
- deleted 4y ago[deleted]
- aljungberg 4y agoIt does say on there they are training it on the Pile training data. And they have this bit comparing inference with GPT2-XL: RWKV-3 1.5B on A40 (tf32) = always 0.015 sec/token, tested using simple pytorch code (no CUDA), GPU utilization 45%, VRAM 7823M GPT2-XL 1.3B on A40 (tf32) = 0.032 sec/token (for ctxlen 1000), tested using HF, GPU utilization 45% too (interesting), VRAM 9655M So it looks about twice as fast for inference while using only about 80% as much VRAM. Obviously at such a small size, just 1.5B, you can run it even on consumer GPUs but you could do that with GPT2 as well. If it remains 80% of VRAM usage when scaled up, we’re still talking 282GB once it’s the size of BLOOM w/ 176B parameters. So yeah still 8x A100 40GB cards I guess. Not going to be the Stable Diffusion of LLMs.
- taktoa 4y agoI'm pretty sure those numbers are for training, not inference. I've run it on _CPU_ and gotten ~1 token per second.
- aljungberg 4y agoTHe RWKV model seems really cool. If you could get transformer-like performance with an RNN, the “hard coded” context length problem might go away. (That said, RNNs famously have infinite context in theory and very short context in reality.) Is there a primer for what RWKV does differently? According to the Github page it seems the key is multiple channels of state with different decaying rates, giving I assume, a combination of short and long term memory. But isn’t that what LSTMs were supposed to do too?
- swyx 4y ago> RNNs famously have infinite context in theory and very short context in reality. any sources to read more about this please? its the first ive heard of it
- georgehill 4y agoI am not sure this article will answer your question, but Karpathy has an article about RNNs. https://karpathy.github.io/2015/05/21/rnn-effectiveness https://karpathy.github.io/2015/05/21/rnn-effectiveness
- swyx 4y agoit doesnt touch on the "infinite context in theory and very short context in reality" piece which is what i was asking about
- sillysaurusx 4y agoI can confirm it from what we’re seeing on a video prediction task. Future frames end up blurry. The first frame is sharp, but by frame 3 it’s only crisp when it’s very certain of its prediction. Any kind of rare movement, it goes “I kinda know what it roughly looks like” and smears fingerpaint all over the canvas. The overall trajectory looks ok, so I’ll be more rigorously investigating whether it’s possible to squeeze more precise context out of it. For example, since the first frame is sharp, you could discard the other future frames and use that first frame as the last history entry (rolling completion window). If “the first frame is always sharp” is true, then it seems reasonable that you can generate N sharp frames with that technique, which might work better than predicting N all at once.
- rkwasny 4y agoTurns out it does not matter if you have transformer/MLP/lstm or whatever, as long as there are enough parameters and training epochs over large dataset things "just work"
- BulgarianIdiot 4y agoTo some degree, because we keep recreating the truly essential components the crude "Turing machine completeness" way. In time as we analyze the resulting models, we may find what patterns emerge and optimize for them. The result will be smaller, faster models that perform like larger slower ones.
- nl 4y agoThis isn't true - the model architecture matters a lot. In general RNNs cannot handle long term dependencies (ie, long pieces of text) because the gradient vanishes. It's unclear how this solves this problem although they do reference the "attention free transformer" paper: https://arxiv.org/abs/2105.14103 https://arxiv.org/abs/2105.14103
- cztomsik 4y agoI believe it's because you train it in GPT-mode and then only use RNN-mode for inference.
- PartiallyTyped 4y agoThe key component is the linear attention[1] and residual connections. [1] https://arxiv.org/abs/2006.16236 https://arxiv.org/abs/2006.16236 > Transformers achieve remarkable performance in several tasks but due to their quadratic complexity, with respect to the input's length, they are prohibitively slow for very long sequences. To address this limitation, we express the self-attention as a linear dot-product of kernel feature maps and make use of the associativity property of matrix products to reduce the complexity from (N2) to (N), where N is the sequence length. We show that this formulation permits an iterative implementation that dramatically accelerates autoregressive transformers and reveals their relationship to recurrent neural networks. Our linear transformers achieve similar performance to vanilla transformers and they are up to 4000x faster on autoregressive prediction of very long sequences.
- adam_arthur 4y agoSo what's the level of effort to create ChatGPT equivalent products? Is it something where we'll have 100s of competing AIs, or is it gated to only a few large companies? Not up to date on current training/querying costs. Can these models feasibly be run locally? Given the large number of competitors already announced to ChatGPT, I fail to see how the space will be easily defensible or monetizable (despite large value add, competitors can easily undercut eachother)
- numbchuckskills 4y ago[dead]
- wokwokwok 4y ago> Can these models feasibly be run locally? Bluntly, no. The models which are small enough to run locally perform so badly it’s not worth bothering. To run inference on the large models the perform decently you need the equivalent of two or three top end graphics cards. If you're serious about looking into it now, consider looking at this project that lets you run a bunch of independent machines as a cluster for inference using Bloom: https://github.com/bigscience-workshop/petals/wiki/Launch-your-own-swarm https://github.com/bigscience-workshop/petals/wiki/Launch-yo... (You'll need around 200GB of GPU memory across the machines in the swarm)
- adam_arthur 4y agoHmmm, 2-3 high end GPUs implies it's likely not very far off from mainstream. Maybe runnable on the average device within 10-20 years... perhaps even sooner if the model/software can be optimized?
- lolspace 4y ago20 years?
- adam_arthur 4y agoIn 20 years I'll still be alive and enjoying myself. Does 20 years seem long to you?
- leaving 4y ago[flagged]
- anon291 4y agoThis is a git repo, not a published paper. Hacker news is not a published journal. It's a casual space for technically oriented people. And you can say whatever you want on your own GitHub.
- VadimPR 4y agoHow does this compare to BLOOMZ's performance, if anyone knows?
- euclaise 4y agoAssuming you're referring to the largest model - BLOOM is huge, this is not, so presumably much worse
- moneywoes 4y agoName rolls of the tongue
- avmich 4y agoYeah, when the Web was young, and people told URLs to each other, pronouncing "www", which was almost always the prefix of any web server host name also sounded funny.
- klabb3 4y agoNot sure what you’re talking about. Eitch tee tee pee ess colon slash slash doubleview doubleview doubleview dot just rolls off the tongue so easily.
- golem14 4y agoit's "dub-dub-dub", isn't it?
- denton-scratch 4y agoWuh-wuh-wuh.
- joshxyz 4y agochat are-woo-kei-vee. gpt rolls out real better lol.