6 ms·
XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens
- foolfoolz 3y agowhen will the llm race peak? have we peaked already?
- nico 3y agoLong way from that
- DantesKite 3y agoIf you’re curious, you can check the progress of many open source LLM’s and how they perform on various evals here: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
- entrep 3y agoWith proprietary models: https://chat.lmsys.org/?leaderboard https://chat.lmsys.org/?leaderboard
- rasz 3y agoNobody doing xB models is participating in any AI races, at this point those are useless toys with garbage output.
- az226 3y agoWe’re still in the early innings.
- sacnoradhq 3y agoI don't think we need anymore high-flying search engines, do we? salesforce air esearch ;)
- bratao 3y agoI think we are only at the beginning. Call it first generation. LLaMA was even missing many improvements that existed before it (xPos, Multi-Query Attention, Blockwise Parallel Transformer) Many researchers are improving very fast, and I would bet that soon we will see more efficient LLMs.
- esafak 3y agoWhen it's superhuman.
- phillnom 3y agoTo quote Dennis Reynolds, it hasn't even begun to peak.
- kristianp 3y agoIn terms of Gartner hype cycle, we are at the peak of inflated expectations for llms I reckon.
- TOMDM 3y agoFrom all the experimentation I've done, 7B parameter models just don't seem to be able to produce useful output reliably enough for my use cases. What use cases do people have for these smaller LLM's?
- jimmyl02 3y agoThe main use case is that it's probably the only size consumers can run on their personal devices. If you don't want your data going into an external platform like OpenAI it's the only solution even if it's not very usuable.
- jahewson 3y agoThese are constraints but not a use case.
- verwkljdslfklj 3y agoIsn't it obvious they're referring to the use case of using a model given those constraints?
- roughly 3y agoYes, but I think the responder is wondering if there are useable use cases for that - like, what can you actually Do with that model. I’m in the same boat - I don’t want to ship my data to openai, I do want to run local, so I’d love to hear what other folks are Doing with models of that size.
- sidibe 3y ago"Use case of using" :)
- m00x 3y agoYou can run big models on the cloud yourself, or with a 3090/4090 quantized. You don't have to go to openai.
- DanAtC 3y agoI have no idea what any of these words mean, but I'd like to. Can someone point me in the direction of an "AI for Dipshits"?
- seeknotfind 3y agoWhy not ask the AIs themselves? They are pretty good at explaining this type of thing.
- dcre 3y agoSounds like a joke but it's true.
- deedree 3y agoBut if you don't know a certain amount about a subject already you won't know when it's lying to you. That would probably be the case here.
- CamperBob2 3y agoIt's basically what an interactive dialogue with Wikipedia would look like, which is still a darned useful thing.
- TeMPOraL 3y agoIf Wikipedia had the conviction and focus of a 4 year old. But yeah, while tricky, it's still usually net positive.
- esafak 3y agoIt's an open source language model with seven billion parameters (a measure of its complexity), and a longer than typical sequence length of 8K, which allows the provision of more context when querying the model. For example, it allows you to better generate text in someone else's voice by providing a longer example of their work. https://en.wikipedia.org/wiki/Foundation_models https://en.wikipedia.org/wiki/Foundation_models
- brucethemoose2 3y ago> The training recipe and model architecture follow LLaMA This is huge. MPT and Falcon are cool, but the inference runtimes and various tooling is mostly optimized for LLaMA. If this is a drop-in replacement for 7B, it's going to catch on much faster than any other small model.
- jxy 3y agoIt seems to use a different tokenizer than LLaMA, though the neural network architecture is the same.
- minimaxir 3y agoThe tokenizer appears to be the original GPT-2 tokenizer, with some curious added tokens: https://huggingface.co/Salesforce/xgen-7b-8k-base/blob/5e1ad70389cda911e995522368165a7cc4b3da21/tokenization_xgen.py#L40 https://huggingface.co/Salesforce/xgen-7b-8k-base/blob/5e1ad...
- imrehg 3y agoThose look all programming related tokens, and I think they should relate to their focus of improving this model's code generation capabilities, and adding quite a bit of data related to that (BigCode Starcoder in the Second stage in pre-training https://blog.salesforceairesearch.com/xgen/#pre-training-data https://blog.salesforceairesearch.com/xgen/#pre-training-dat... )
- imrehg 3y agoLooks like this is the limitation why it cannot just be plugged into llama.cpp (doesn't have a saved tokeniser model, and I'm not sure how would one go about creating one). Otherwise it would be cool to try it out, the metrics in the article are promising to have something running locally on an M1 Mac...
- lhl 3y agoThere's also OpenLLaMA, which has a 13B version as well and is a straight drop-in (except for code generation due to multiple space tokenization: https://github.com/openlm-research/open_llama#update-06152023 https://github.com/openlm-research/open_llama#update-0615202...). XGen-7B is probably the superior 7B model, it's trained on more tokens and a longer default sequence length (although both presumably can adopt SuperHOT (Position Interpolation) to extend context), but larger models still probably perform better on an absolute basis.
- brucethemoose2 3y agoAlso, their metric table is very interesting. It shows Falcon 7B and OpenLlama 7B much less favorably than other evaluations (including the HuggingFace leaderboard, which I am kinda suspicious of), and instruct benchmarks like that aren't seen as much.
- minimaxir 3y agoPer the validation perplexity chart shown, the 8K length model performs better than the 4K length model even at <4K length, so why are they even offering the 4K model if the 8K is strictly better?
- numeri 3y agoI'd say for research purposes. HackerNews seems to tend to mostly represent the LLM consumer viewpoint, but these waves of models being released are honestly more interesting from a research than a user point of view. As a LLM user, you're (vast generalization/simplification) really just interested in the best of N models, but as a researcher, I'm super interested in each model's performance and analyzing the reasons for any differences. With this model (and they say this in the blog post), they were testing the hypothesis that training on a longer context size would provide more performance at the same parameter count/inference FLOPs. From a quick perusal of their post, it looks like this was true, and we should train all future models with as long of a context size as we can afford.
- profsummergig 3y agoIf someone could elucidate on what these phrases signify, I'd be very grateful: 1) 7B foundational model 2) 8K length 3) 1.5T tokens
- going_ham 3y ago1. The trained model has 7B parameters or weights for each neuron. 2. It can handle upto 8k tokens. Tokens are usually some representation for a word. If your tokens are characters then, "h", "e", "y" represent 3 tokens for hey. Most of the algos use byte pair encoding. For example "hand-le" has two tokens "hand" and "le". This is a very crud example which is enough to give the gist but is not accurate. You can look into byte pair encoding for more details. 3. The token size 1.5T token means they have huge variations for input and output. Simply put, it was trained on large data corpus. I hope this simplifies it. You can research further if you are interested! Hope it helps!
- thomasahle 3y agoDon't just post ChatGPT answers as comments on hackernews. This one doesn't even make any sense. Of course it doesn't have 7B parameters _per_ neuron.
- johanvts 3y agoDon’t be too critical dr. Ahle. Maybe it’s a new single-neuron architecture.
- Sunhold 3y agoDoesn't look like ChatGPT. Grammatical errors like "on large data corpus," the poor comma usage, misspelling crude, etc. are more of a human thing.
- thomasahle 3y agoMaybe you are right. I was confused by sentences like "I hope this simplifies it. You can research further if you are interested! Hope it helps!" which seemed to be responding to a prompt other than just the previous comment.
- artemonster 3y agoPlease recommend a good tutorial/book/video on modern LLMs and NNs in general, for programmers and technical people. Where you get the idea of how it works. Tried googling with dozens of queries and it just sucks, a lot of hand-wavy articles for lay people or some paid courses.