6 ms·
InternLM2
- milliondreams 3y agoTLDR; 1. InternLM2 is an open-source Large Language Model that has shown improvements over previous models, particularly in long-context modeling. 2. The model uses a unique approach, combining traditional training with Supervised Fine-Tuning and Conditional Online Reinforcement Learning from Human Feedback. 3. It offers a variety of model sizes and training stages to the community, demonstrating significant advancements in AI research and application.
- jerpint 3y agoExcited to see how it will perform on the lmsys leaderboard
- ilaksh 3y agoDoes anyone know how the free commercial license works? Do they usually grant it? https://wj.qq.com/s2/12727483/5dba/ https://wj.qq.com/s2/12727483/5dba/ looks like a form there. Apache 2 code, free commercial license with application form for weights.
- esha_manideep 3y agoPretty amazing to see training data being discussed more openly
- WiSaGaN 3y agoIndeed. I think part of the reason when they are not discussed openly may be that much of the data used is copyrighted, which introduces some legal ambiguities.
- YetAnotherNick 3y agoIANAL but hiding something doesn't make someone legally immune. Any company could sue LLM companies and they can't hide it during the case. e.g. there is already a similar case on OpenAI.
- fl0id 3y agoYes, but it at the very least delays any findings while you rake in the cash and try to create a favorable environment. OpenAI even stated that think using copyrighted texts is necessary and should be covered by fair use.
- deleted 3y ago[deleted]
- dannyw 3y agoHow good is the base (non-instruction-tuned) model? Everyone is trying to make chat bots, but for my use cases, I find base models more suitable.
- fragmede 3y agoInteresting. What are some of those use cases?
- viraptor 3y agoThe repo is here: https://github.com/InternLM/InternLM https://github.com/InternLM/InternLM
- zone411 3y agoWe really need better long context benchmarks than needle-in-a-haystack. There is LV-Eval (https://arxiv.org/abs/2402.05136 https://arxiv.org/abs/2402.05136) with multi-hop QA that's better but still pretty basic.
- andersa 3y agoYes, I don't understand why they are using a search benchmark for these... it would be much better to have something like giving it a story up to the context length (from a book? how to find one that's not in the training data?) and have it write a new chapter/ending that is consistent with all prior text and introduces zero inconsistencies. But how can you automatically evaluate whether it did this?
- ricw 3y agoBecause that’s how people use llms. You go to ChatGPT to ask a question and get an answer, rather than searching on Google, revising your search because you didn’t know a term, and then look at 3-5 different links to find the answer to what you were searching for
- loa_in_ 3y agoThat's how people use llms because they (llms) don't seem to be good at the more sophisticated thing
- sp332 3y agoBut also because search engines have gotten even worse at answering questions.
- zone411 3y agoIt seems you might be mixing up different types of "context" in LLM benchmarking. In this case, it refers to the input text directly provided to the model during evaluation by the user (as in in-context learning). This is separate from the text an LLM is trained on or can access via RAG methods.
- barsonme 3y agoIs it normal for papers to have that many authors?
- esafak 3y agoOnly in fields like foundation models and high-energy physics, where immense resources are required. Look at GPT-4's credits: https://openai.com/contributions/gpt-4 https://openai.com/contributions/gpt-4
- barkingcat 3y agoIt's not abnormal in many fields. A lot of biology or physics papers have more than that number. In academic labs, so you're a postgrad working the overnight shift watching some petri dish to make sure the bacteria doesn't die, etc. It's super boring grunt work but you do it so you get on the paper's author list.
- exe34 3y agoParticle physics papers usually have more pages for the names than for the work.
- m3kw9 3y agoAnyone that even sniffed it can get in on the action
- pilotneko 3y agoI experimented with this model and vLLM around a month ago. The long context length is attractive, but it was incredibly slow on a g5.12xlarge (4 NVIDIA A10G GPUs). I actually could not get it to respond for single examples longer than 50K tokens.
- Kwpolska 3y agoThe name suggests this is interns posing as a chatbot, especially considering today’s date.