14 ms·

Are Open-Source Large Language Models Catching Up?

342 points by rkwz 3y ago

Tostino 3y ago
It depends on what you're doing... Just for reference, here is a small showcase of the capabilities that I've trained on a 13 billion parameter llama2 fine tune (done with qlora). https://old.reddit.com/r/LocalLLaMA/comments/186qq92/comment/kba2d6m/?utm_source=share&utm_medium=web2x&context=3 https://old.reddit.com/r/LocalLLaMA/comments/186qq92/comment... Edit: Embed some of the content instead. Inkbot can create knowledge graphs. The structure returned is proper YAML, and I got much better results with my fine-tune than using GPT4. https://huggingface.co/Tostino/Inkbot-13B-8k-0.2 https://huggingface.co/Tostino/Inkbot-13B-8k-0.2 Simple prompt: https://gist.github.com/Tostino/c3541f3a01d420e771f66c62014e6a24 https://gist.github.com/Tostino/c3541f3a01d420e771f66c62014e... Complex prompt: https://gist.github.com/Tostino/44bbc6a6321df5df23ba5b400a01e37d https://gist.github.com/Tostino/44bbc6a6321df5df23ba5b400a01... It also does chunked summarization. Here is an example of chunking: Part 1: chunked summarization - https://gist.github.com/Tostino/cacb1cecdf2eb7386baf565d157f56a0 https://gist.github.com/Tostino/cacb1cecdf2eb7386baf565d157f... Part 2: summary-of-summaries - https://gist.github.com/Tostino/81eeee9781e519044950332b4e64bef1 https://gist.github.com/Tostino/81eeee9781e519044950332b4e64... Here is an example of a single-shot document that fits entirely within context: https://gist.github.com/Tostino/4ba4e7e7988348134a7256fd1cbbf4ff https://gist.github.com/Tostino/4ba4e7e7988348134a7256fd1cbb...
- cced 3y ago
  Have any references on how you fine tuned?
  - Tostino 3y ago
    Sure thing, I used axolotl, and my training parameters were: sequence_len: 6144 lora_r: 128 lora_alpha: 48 learning_rate: 0.00006 warmup_steps: 600 lr_scheduler: cosine gradient_accumulation_steps: 4 micro_batch_size: 1 num_epochs: 4 optimizer: paged_adamw_32bit flash_attention: true sample_packing: true
    - kaycebasques 3y ago
      Thanks for sharing all these details throughout the thread, Tostino. True open source spirit.
- reactive001 3y ago
  Amazing work, I've really wanted to get into knowledge graph generation with LLM's for the last year but haven't found the time. Glad to see someone making good progress on the idea! How are you going about generating training data?
  - Tostino 3y ago
    Lots and lots of manual review of very detailed instructions to "more powerful LLMs" with 2-4 prompts to generate the training data.
- thecal 3y ago
  I really like Inkbot! Are you working on a new version? How about one from Yi 34B?
  - Tostino 3y ago
    Yeah, I will be soon. I was busy adding `chat template` support to vLLM recently, so the model (and any others that implement it properly) will work seamlessly with a clone of the OpenAI chat/completions endpoint. https://github.com/vllm-project/vllm/pull/1756 https://github.com/vllm-project/vllm/pull/1756 Now that I have that out of the way, back to model training ;).
    - thecal 3y ago
      Very cool, looking forward to it!
- anon373839 3y ago
  This looks really impressive. Any chance a 7B Inkbot is in the works?
  - Tostino 3y ago
    Yeah, i'll be training a Mistral variant soon (and some of the larger, newer models as well). I had a few dataset issues I know about that I wanted to fix first.
    - anon373839 3y ago
      That’s great to hear. Thanks!
man4 3y ago
[dead]
smy20011 3y ago
I tried to use Sakura LLM to translate some JP novels. It's really good and half the price of GPT3.5 turbo. https://github.com/SakuraLLM/Sakura-13B-Galgame/tree/dev_server https://github.com/SakuraLLM/Sakura-13B-Galgame/tree/dev_ser...
- eunos 3y ago
  It seems that it only supports Japanese to Chinese only.
whalesalad 3y ago
The innovators dilemma.
alfalfasprout 3y ago
Long term it's almost unavoidable that open source LLMs start catching up. One factor that's worth considering too is cost. The open source community is much more resource constrained and they've really accelerated the pace of development in <30B parameter models.
- YetAnotherNick 3y ago
  Google and Meta and all the funded companies also are not even close to GPT 4, so I doubt cost is the biggest factor. Claude is the only model that is decent other than OpenAI's.
  - adastra22 3y ago
    My understanding is that the models are pretty comparable, but nobody's reinforcement training set is not nearly as good as OpenAI's, so they're able to fine-tune their model to give more accurate results.
- hyperliner 3y ago
  This is an industry where cost will be an issue. It reminds me of Rackspace and others trying to win with OpenStack “because open.” AWS and Azure won. Even Google is third. The big players will win, and there will be a niche for open tools.
  - devjab 3y ago
    Google only lost because they couldn’t re-adjust their business for their paid products to not be similar to their advertising products. I can only speak for the European enterprise scene, but AWS came first and in the beginning they went a very “Googley” route of not having very great support and very little patience for local needs. Then Azure came along with their typical Microsoft approach to enterprise, which is where you get excellent support and you get contacts into Microsoft who will actually listen and make changes, well, if the changes align with what Microsoft wants. I know Microsoft isn’t necessarily a popular company amongst people who’ve never interacted with them on an Enterprise level, but they really are an excellent it-business partner because they understand that part of being an Enterprise partner is that they let CTOs tell their organisation that they know X is having issues but that Microsoft headquarters is giving them half-hourly updates by phone. Sort of useless from a technical perspective, immensely useful for the CTO when 2000 employees can’t log into Outlook. Another good example is how when Teams rolled out with being on for all users by default, basically every larger organisation in the world went through the official channels and went “nonononono” and a few hours later it was off by default. Now, when Amazon first entered the European market they were very “Googley” as I said, but once they realized Microsoft business model was losing them customers, they changed. We went from having no contacts to having an assigned AWS person and from not wanting to adopt the GDPR AWS actually became more compliant than even what Azure currently is. Google meanwhile somehow managed to make the one product they were actually selling (education) worse than it was originally, losing billions of dollars on all the European schools who could no longer use it and be GDPR compliant. The Chinese cloud options obviously had similar data privacy issues to Google and never really became valid options. At least not unless China achieves the same sort of diplomatic relationship with the EU that the US has, which is unlikely. So that’s the long story of why only two of the major cloud providers “won”. With the massive price increase, however, more and more companies are especially Azure for their own setups. This isn’t necessarily a return to having your own iron in the basement, often it’s going to smaller cloud providers and then having a third party vendor set something like Kubernetes up. Right now, Microsoft is winning the AI battle. Not so much because it’s better, but because it comes with Office365. Office365 which was already a sort of monopoly on Office products, but is now even more so. A good example is again how Teams became dominant, even though it wasn’t really the best option for a while and is now only the best option because of how it integrates directly with your Sharepoint online which is where most enterprise orgs store documents these days. So too is copilot currently winking the AI battle for organisations who can’t really use a lot of the other options because of data privacy issues. So while copilot isn’t as good as GPT, it’s still what we are using. But if it ever gets too expensive, it’s not as secure as you may think. Especially not if we start seeing more training sets, or EU and US relations worsens. I think the most likely outcome, at least here in the EU, is that anti-completion laws eventually takes a look at Office365 because of how monopolised it is. Or the EU actually follows through on their “a single vendor is a threat to national security” legislation and force half of the banking/energy/defense/andsoon industries to pick something other than Microsoft. Which will be hilariously hard, but if successful (which it probably won’t be because it’s hilariously hard) will lead to more open products.
pram 3y ago
I’ve found Mistral OpenOrca is pretty much as good as GPT4-turbo for creative writing/analysis. Actually it tends to output very similar text, which is suspicious, but whatever it saves me a lot of money. https://huggingface.co/Open-Orca/Mistral-7B-OpenOrca https://huggingface.co/Open-Orca/Mistral-7B-OpenOrca
- SushiHippie 3y ago
  Also openchat, which was trained on gpt4 conversations IIUC. https://github.com/imoneoi/openchat https://github.com/imoneoi/openchat
- bugglebeetle 3y ago
  Mistral OpenOrca is very good at task following as well. Its slightly less reliable than GPT 3.5/4, but the difference in quality for my text processing tasks is pretty much a toss-up.
calvinmorrison 3y ago
[flagged]
drakenot 3y ago
I've been somewhat disappointed with the performance of the open models. The claims of certain models outperforming GPT-3.5-Turbo and approaching GPT-4 fail to hold up to their benchmark results in real-world scenarios, potentially due to data contamination in assessments, based on my testing. As noted in the linked survey paper, some models may outperform 3.5-Turbo in specific, narrow areas, depending on the model. Yet, we still lack a general model that definitively exceeds 3.5-Turbo in all respects. I'm concerned that while we're still striving to reach 3.5-Turbo's performance level, OpenAI may unveil a new next-generation model, further widening the performance gap! Back in the summer, I had higher hopes that we would have surpassed the 3.5 threshold by now. The performance gap has been surprisingly large. It is especially noticeable in areas requiring consistent structured output or tool use from the LLM. This is where open models particularly falter.
- snowycat 3y ago
  There are tools that you can use to force the model to give structured output, such as llama.cpp GBNF grammar for example. They're a bit harder to use than ask gpt4 but they do work pretty well for what I use it for.
  - white_dragon88 3y ago
    [dead]
- raincole 3y ago
  "OpenAI has no moat" aged so badly that it's almost a satire.
  - Tostino 3y ago
    Try out my model vs gpt4 for the same tasks (I explicitly trained on) and compare. https://huggingface.co/Tostino/Inkbot-13B-8k-0.2 https://huggingface.co/Tostino/Inkbot-13B-8k-0.2 It's a 13b param model that isn't meant to be general purpose, but is meant to excel on the limited tasks I've trained on. You'll see more like this soon.
thorum 3y ago
Current ~70B models like LLAMA 2 70B are on par wih ChatGPT 3.5. The best smaller models can appear on par at first glance, but they hallucinate at a much higher rate and lack knowledge of the world. GPT 4 ‘gets’ things at a deeper level and no open source model is even close. A year is a good timeframe to evaluate things: the rest of the world seems to lag behind OpenAI by around 12-18 months, at least with LLMs and image generation. On the other hand open source tech usually has additional features for controlling output that OpenAI never bothers to implement, like llama.cpp’s grammars or ControlNet. So in that sense open source is usually ahead of OpenAI in terms of customizability.
- omeze 3y ago
  I dont think OpenAI is ever going to ahead in image generation, they were lapped very soon after dall-e and every real workflow Ive seen uses Midjourney or Stable Diffusion. The reverse (GPT 4 vision) is well ahead of open source though
  - vunderba 3y ago
    The original is leagues behind anything current, but DALL-E version 3 absolutely blows any state of the art generative model out of the water, including mid journey 5.2 and SDXL in terms of pure prompt accuracy and coherence. Midjourney still has the edge in quality, but it's a moot point if it takes you 1000 v-rolls to get to your original vision. If all you're generating is anime waifus then MJ/NovelAI/Niji will suffice, but generating prompts particularly featuring relatively complex scenes or actions are amazing on DALL-E 3. And of course unfortunately, it goes without saying that open AI DALL-E is going to be the most restrictive in terms of censorship. I generated these from DALL-E 3 instantly. Try to generate them in any other commercial offering. Go ahead. I'll wait... https://imgur.com/a/2GTRjfK https://imgur.com/a/2GTRjfK Descriptions: A 80s photograph of the Koolaid Man breaking through the Berlin Wall. Comic illustration set at a festive children's party. The main focus is on the magician who looks uncannily like a well-known fictional wizard. He's trying to say abracadabra but accidentally uses the killing curse.
    - 3y ago
yieldcrv 3y ago
Amethyst Mistral 13B q5 gguf is what I’m using most of the time now. Synthetic datasets are great to finetune with, there is no moat for having inaccessible literature data sets I’m offline now because I’ve had too many ideas and domain names registered too soon after conversing with Chat GPT4 I’m open to the idea of people reacting to similar stimuli that cause ideas to be done at the same time, but I didn't like that experience and I can run these models on my M1 with LM Studio so easily I do think some chats get flagged when the model says something seems novel, like Albert Einstein working at the patent office. Not worth making it my whole identity in wanting to prove, just the catalyst I needed to try 7B and 13B models seriously and I’m quite pleased
- deleted 3y ago
  [deleted]
- tarruda 3y ago
  Is "Amethyst Mistral 13B" a llama fine tune? I searched for it on huggingface and only found the GGUF version, the link to the original model is broken
  - Havoc 3y ago
    Mistral would be the base model
    - tarruda 3y ago
      Can a 7B base model be fine tuned into a 13B model? Mistral is 7B, this one is 13B.
      - Havoc 3y ago
        Two models can be merged from what I understand to result in different total param sizes. That’s different from a fine tune I gather the results of merges can be unpredictable though
- dkarras 3y ago
matchagaucho 3y ago
If ChatGPT were only 1 LLM, then maybe. But it's a Mixture of Experts (MoE) architecture, which I think makes open source comparisons unfair?
- zamadatix 3y ago
  Open source options lacking parts of the ChatGPT approach which make it successful doesn't make the comparisons unfair it explains why ChatGPT wins right now. There's nothing stopping open source options from using the same MoE architecture, the solutions just don't (to the same effect act least) right now.
behnamoh 3y ago
[flagged]
bumbledraven 3y ago
[flagged]
benreesman 3y ago
`shiningvaliant-1.2-Q4_K_M` is my go-to. I appreciate that it doesn't top the boards in most metrics vs e.g. GPT-4, but I'm not in some A/B group on the quantization: it's more useful to me in practice more of the time. I have it rigged up with a prompt about outputting markdown and wired up to `foo | glow -` and I get GPT-4 out when I want something to write JIRA tickets no one is going to read because it's better at that sort of thing.
refibrillator 3y ago
It's not mentioned in the paper but this month OpenChat 3.5 released the first 7b model that achieves results comparable to ChatGPT in March 2023 [1]. Only 8k context window, but personally I've been very impressed with it so far. On the chatbot arena leaderboard it ranks above Llama-2-70b-chat [2]. In many ways open source LLMs are actually leading the industry, especially in terms of parameter efficiency and shipping useful models that consumers can run on their own hardware. [1] https://huggingface.co/openchat/openchat_3.5 https://huggingface.co/openchat/openchat_3.5 [2] https://chat.lmsys.org/ https://chat.lmsys.org/
- ekianjo 3y ago
  openchat is very impressive indeed. I think it may be better than Mistral, while comparisons are not always easy.
  - tudorw 3y ago
    I'm finding Mistral good at creative literature and it is fairly adept at taking instructions, good enough for my purposes, and running locally on consumer CPU, the future of open source local models looks bright.
- LoganDark 3y ago
  > Only 8k context window Is this supposed to be low? All the chat models I've used top out at 4096.
  - Sai_ 3y ago
    GPT-4-turbo is at 128k. Claude 2.1 is 200k. But yes, among open source models 8k is roughly middle to top of the pack.
    - LoganDark 3y ago
      That's insane. The highest I've personally seen in the open-source space is RWKV being trained on (IIRC) 4k but being able to handle much longer context lengths in practice due to being an RNN (you can simply keep feeding it tokens forever and ever). It doesn't generalize infinitely by any means but it can be stretched for sure, sometimes up to 16k. It's not a transformer model though, and old context fades away much faster / is harder to recall because all the new context is layered directly on top of it. But it's quite interesting nonetheless.
czk 3y ago
I'd say they are catching up for sure, especially with how GPT4 has been regressing consistently over the past month. https://chat.openai.com/share/c91287ee-9a5e-4c99-b5df-49cc4536e471 https://chat.openai.com/share/c91287ee-9a5e-4c99-b5df-49cc45...
- tarruda 3y ago
  I suspect a lot of the "catching up" was achieved by using GPT-4 API to generate high quality fine-tune datasets.
kaycebasques 3y ago
No comment from me on the question in the title (because I don't know enough to have an opinion), but since others are discussing various open models I will mention another that I've been enjoying tonight: DeepSeek 67B https://chat.deepseek.com https://chat.deepseek.com (This chat UI has adequately replaced my ChatGPT needs so far.) https://huggingface.co/deepseek-ai/deepseek-llm-67b-base https://huggingface.co/deepseek-ai/deepseek-llm-67b-base https://twitter.com/abacaj/status/1730019229175312612 https://twitter.com/abacaj/status/1730019229175312612
lhl 3y ago
A couple big/strong open models that have just been released the past few days: * Qwen 72B (and 1.8B) - 32K context, trained on 3T tokens, <100M MAU commercial license, strong benchmark performance: https://twitter.com/huybery/status/1730127387109781932 https://twitter.com/huybery/status/1730127387109781932 * DeepSeek LLM 67B - 4K context, 2T tokens, Apache 2.0 license, strong on code (although DeepSeek Code 33B it benches better) https://twitter.com/deepseek_ai/status/1729881611234431456 https://twitter.com/deepseek_ai/status/1729881611234431456 Also recently released: Yi 34B (with a 100B rumored soon), XVERSE-65B, Aquila2-70B, and Yuan 2.0-102B, interestingly, all coming out of China. Personally, I'm also looking forward to the larger Mistral releasing soon as mistral-7b-v0.1 was already incredibly strong for its size.
- John9 3y ago
  Since it's not allowed to use ChatGPT in China, there is a huge opportunity to build a local LLM.
  - wenyuanyu 3y ago
    Anyone know what is the reason why both OpenAI and Anthropic proactively banned users from China from using their products...
    - wavemode 3y ago
      Source for this? I know China's government firewall blocks ChatGPT (for obvious reasons) but I wasn't aware that OpenAI was blocking them in return.
      - pototo666 3y ago
        Chinese IP are not allowed to use ChatGPT. Chinese credit card is not allowed for OpenAI API. Source: my own experience. What puzzles me most is the second restriction. My credit card is accepted by AWS, Google, and many other services. It is also accepted by many services which use Stripe to process payments. But OpenAI refuses to take my money.
davidkunz 3y ago
Out of personal experience, open source LLMs did not yet reach the quality of GPT 3.5, despite multiple claims with dubious benchmarks. That said, they are already useful as of today and can even run on your local machine. I regularly use them with my Neovim plugin gen.nvim [1] for simple tasks and they save me a lot of time. I'm excited about the future! [1]: https://github.com/David-Kunz/gen.nvim https://github.com/David-Kunz/gen.nvim
- tarruda 3y ago
  Very interesting. I want to give it a try, but I see that one of the dependencies is "ollama" which is a Mac App and I don't have a Mac. I'm running Llama models locally using llama-cpp-python which provides an OpenAI compatibility layer.
  - rkwz 3y ago
    Oh, can install Ollama in Linux/WSL as well
  - davidkunz 3y ago
    It should work on Linux and also with WSL. You could also try running it in Docker.
    - tarruda 3y ago
      I see. Does ollama have an http API (this the curl requirement)? If so, is it compatible with OpenAI API?
      - davidkunz 3y ago
        Yes, it works with an Ollama server and the communication is done via HTTP. I know that someone configured my plugin to talk to OpenAI.
intrepidsoldier 3y ago
Big fan of Starcoder
jurmous 3y ago
A problem I have with the open source models is that they are all not remotely good in many languages other than English compared to the OpenAI models. I specifically need Dutch and the outputs are unusable for us.
- danielbln 3y ago
  https://openchat.team/ https://openchat.team/ is quite good at German, how does it fare with Dutch?
  - jurmous 3y ago
    As far as I can see it uses the OpenAI models which are quite good. Or am I wrong?
    - danielbln 3y ago
      I believe it's using a Mistral fine-tune, not GPT.
- jug 3y ago
  Yeah, this can be an issue. Today, there are few specialized LLM's of high quality so you end up having to use a massive all-in-one model like GPT-4 to reach for the language you need. There's movement here though and it will get better. GPT-SW3 is a new model developed by AI Sweden, trained specifically on the Nordic languages only + English. And beyond this, you have TrustLLM which is a new project that aims to be a large, open, European model trained on the Germanic languages to start with: https://liu.se/en/research/trustllm https://liu.se/en/research/trustllm
  - jurmous 3y ago
    Yes there is some movement. Good to see trustllm being another initiative. The Dutch government starts with some non profit agencies to train a Dutch model too. But this will also take some time. https://www.tno.nl/en/newsroom/2023/11/netherlands-starts-realisation-gpt-nl/ https://www.tno.nl/en/newsroom/2023/11/netherlands-starts-re...
danofsteel32 3y ago
I've been using OpenHermes-2.5 [0] and NeuralHermes [1] which are both finetunes of the Mistral7B base model. The only objective test prompting I do is asking the models to generate a django timeclock/timesheets app. In this test they compare favorably to GPT-3.5. Also LMStudio [2] has a better UI than chatgpt and responses are much faster too (40tk/sec on my 2070). [0] https://huggingface.co/teknium/OpenHermes-2.5-Mistral-7B https://huggingface.co/teknium/OpenHermes-2.5-Mistral-7B [1] https://huggingface.co/TheBloke/NeuralHermes-2.5-Mistral-7B-AWQ https://huggingface.co/TheBloke/NeuralHermes-2.5-Mistral-7B-... [2] https://lmstudio.ai/ https://lmstudio.ai/
fintechie 3y ago
We're nearing a point where we'll just need a prompt router in front of several specialised models (code, chat, math, sql, health, etc)... and we'll have a local Mixture of Experts kind of thing. 1. Send request to router running a generic model. 2. Prompt/question is deconstructed, classified, and proxied to expert(s) xyz. 3. Responses come back and are assembled by generic model. Is any project working on something similar to this?
- theturtle32 3y ago
  This is how image generation works for DALL-E 3 via ChatGPT.
- ekianjo 3y ago
  it's kind of trivial today. the first layer could be a mix of nlp and zero-shot classification to clarify the nature of the request. Then using LLM deconstruct the request into several specific parts that would be sent to specialized LLMs. Then stitch it back together at the end again with LLM as the summarization machine. Problem is running so many LLMs in parallel means you need quite a bunch of resources.
  - fintechie 3y ago
    Yeah, it shouldn't be too difficult to build this with python. I wonder why none of the popular routers like https://github.com/BerriAI/litellm https://github.com/BerriAI/litellm have this feature. > Problem is running so many LLMs in parallel means you need quite a bunch of resources. Top of line MacBooks or Minis should be able to run several 7B or even 13B models without major issues. Models are also getting smaller and better. That's why we're close =)
    - generalizations 3y ago
      Could lora fine tunes be used instead of completely different models? I wonder if that would save space.
      - 3y ago
Havoc 3y ago
Briefly yes, but in near term I expect falling behind. Sounds suspiciously like the next big leap in hosted models will be more computationally expensive on the inference side rather than training. For hosted that’s much of a sameness - just moving money allocations. But consumers can’t suddenly have 5x 4090s for local.
deleted 3y ago
[deleted]
- deleted 3y ago
  [deleted]
gumballindie 3y ago
Catching up to what? Chatgpt and clones are ridiculously bad at accurate text prediction, let alone coding.
lumost 3y ago
My hunch, The history of open source has been that companies who have customers with massive customization requirements land on the open source side of the equation. Companies who don't view a component as core to their product often land in a similar state. There is almost certainly at least one major firm that wants a GPT-5 like offering, but doesn't view the model as core to their business (Meta). It's also wholly unclear if large models are necessary - or simply convenient. In a similar veign, it's unclear that data must be labeled by humans - the open source data situation is getting better by the day. I'd expect that we'll see OpenAI hold an edge for many years, maybe we'll see a number two player as well for the foundation model, but after that everybody else will base off an open source FM and maybe keep the fine tuning/model augmentation proprietary.
iamgopal 3y ago
May be experts can answer, Can SETI like distributed setup possible with large model training ?
classified 3y ago
No. The moat is being optimized, and no open-source org can or would steal as much training data as the likes of OpenAI do.

342 points by rkwz 3y ago

It depends on what you're doing... Just for reference, here is a small showcase of the capabilities that I've trained on a 13 billion parameter llama2 fine tune (done with qlora). https://old.reddit.com/r/LocalLLaMA/comments/186qq92/comment/kba2d6m/?utm_source=share&utm_medium=web2x&context=3 https://old.reddit.com/r/LocalLLaMA/comments/186qq92/comment... Edit: Embed some of the content instead. Inkbot can create knowledge graphs. The structure returned is proper YAML, and I got much better results with my fine-tune than using GPT4. https://huggingface.co/Tostino/Inkbot-13B-8k-0.2 https://huggingface.co/Tostino/Inkbot-13B-8k-0.2 Simple prompt: https://gist.github.com/Tostino/c3541f3a01d420e771f66c62014e6a24 https://gist.github.com/Tostino/c3541f3a01d420e771f66c62014e... Complex prompt: https://gist.github.com/Tostino/44bbc6a6321df5df23ba5b400a01e37d https://gist.github.com/Tostino/44bbc6a6321df5df23ba5b400a01... It also does chunked summarization. Here is an example of chunking: Part 1: chunked summarization - https://gist.github.com/Tostino/cacb1cecdf2eb7386baf565d157f56a0 https://gist.github.com/Tostino/cacb1cecdf2eb7386baf565d157f... Part 2: summary-of-summaries - https://gist.github.com/Tostino/81eeee9781e519044950332b4e64bef1 https://gist.github.com/Tostino/81eeee9781e519044950332b4e64... Here is an example of a single-shot document that fits entirely within context: https://gist.github.com/Tostino/4ba4e7e7988348134a7256fd1cbbf4ff https://gist.github.com/Tostino/4ba4e7e7988348134a7256fd1cbb...

cced 3y ago
Have any references on how you fine tuned?
- Tostino 3y ago
  Sure thing, I used axolotl, and my training parameters were: sequence_len: 6144 lora_r: 128 lora_alpha: 48 learning_rate: 0.00006 warmup_steps: 600 lr_scheduler: cosine gradient_accumulation_steps: 4 micro_batch_size: 1 num_epochs: 4 optimizer: paged_adamw_32bit flash_attention: true sample_packing: true
  - kaycebasques 3y ago
    Thanks for sharing all these details throughout the thread, Tostino. True open source spirit.
reactive001 3y ago
Amazing work, I've really wanted to get into knowledge graph generation with LLM's for the last year but haven't found the time. Glad to see someone making good progress on the idea! How are you going about generating training data?
- Tostino 3y ago
  Lots and lots of manual review of very detailed instructions to "more powerful LLMs" with 2-4 prompts to generate the training data.
thecal 3y ago
I really like Inkbot! Are you working on a new version? How about one from Yi 34B?
- Tostino 3y ago
  Yeah, I will be soon. I was busy adding `chat template` support to vLLM recently, so the model (and any others that implement it properly) will work seamlessly with a clone of the OpenAI chat/completions endpoint. https://github.com/vllm-project/vllm/pull/1756 https://github.com/vllm-project/vllm/pull/1756 Now that I have that out of the way, back to model training ;).
  - thecal 3y ago
    Very cool, looking forward to it!
anon373839 3y ago
This looks really impressive. Any chance a 7B Inkbot is in the works?
- Tostino 3y ago
  Yeah, i'll be training a Mistral variant soon (and some of the larger, newer models as well). I had a few dataset issues I know about that I wanted to fix first.
  - anon373839 3y ago
    That’s great to hear. Thanks!

man4 3y ago

[dead]

smy20011 3y ago

I tried to use Sakura LLM to translate some JP novels. It's really good and half the price of GPT3.5 turbo. https://github.com/SakuraLLM/Sakura-13B-Galgame/tree/dev_server https://github.com/SakuraLLM/Sakura-13B-Galgame/tree/dev_ser...

eunos 3y ago
It seems that it only supports Japanese to Chinese only.

whalesalad 3y ago

The innovators dilemma.

alfalfasprout 3y ago

Long term it's almost unavoidable that open source LLMs start catching up. One factor that's worth considering too is cost. The open source community is much more resource constrained and they've really accelerated the pace of development in <30B parameter models.

YetAnotherNick 3y ago
Google and Meta and all the funded companies also are not even close to GPT 4, so I doubt cost is the biggest factor. Claude is the only model that is decent other than OpenAI's.
- adastra22 3y ago
  My understanding is that the models are pretty comparable, but nobody's reinforcement training set is not nearly as good as OpenAI's, so they're able to fine-tune their model to give more accurate results.
hyperliner 3y ago
This is an industry where cost will be an issue. It reminds me of Rackspace and others trying to win with OpenStack “because open.” AWS and Azure won. Even Google is third. The big players will win, and there will be a niche for open tools.
- devjab 3y ago
  Google only lost because they couldn’t re-adjust their business for their paid products to not be similar to their advertising products. I can only speak for the European enterprise scene, but AWS came first and in the beginning they went a very “Googley” route of not having very great support and very little patience for local needs. Then Azure came along with their typical Microsoft approach to enterprise, which is where you get excellent support and you get contacts into Microsoft who will actually listen and make changes, well, if the changes align with what Microsoft wants. I know Microsoft isn’t necessarily a popular company amongst people who’ve never interacted with them on an Enterprise level, but they really are an excellent it-business partner because they understand that part of being an Enterprise partner is that they let CTOs tell their organisation that they know X is having issues but that Microsoft headquarters is giving them half-hourly updates by phone. Sort of useless from a technical perspective, immensely useful for the CTO when 2000 employees can’t log into Outlook. Another good example is how when Teams rolled out with being on for all users by default, basically every larger organisation in the world went through the official channels and went “nonononono” and a few hours later it was off by default. Now, when Amazon first entered the European market they were very “Googley” as I said, but once they realized Microsoft business model was losing them customers, they changed. We went from having no contacts to having an assigned AWS person and from not wanting to adopt the GDPR AWS actually became more compliant than even what Azure currently is. Google meanwhile somehow managed to make the one product they were actually selling (education) worse than it was originally, losing billions of dollars on all the European schools who could no longer use it and be GDPR compliant. The Chinese cloud options obviously had similar data privacy issues to Google and never really became valid options. At least not unless China achieves the same sort of diplomatic relationship with the EU that the US has, which is unlikely. So that’s the long story of why only two of the major cloud providers “won”. With the massive price increase, however, more and more companies are especially Azure for their own setups. This isn’t necessarily a return to having your own iron in the basement, often it’s going to smaller cloud providers and then having a third party vendor set something like Kubernetes up. Right now, Microsoft is winning the AI battle. Not so much because it’s better, but because it comes with Office365. Office365 which was already a sort of monopoly on Office products, but is now even more so. A good example is again how Teams became dominant, even though it wasn’t really the best option for a while and is now only the best option because of how it integrates directly with your Sharepoint online which is where most enterprise orgs store documents these days. So too is copilot currently winking the AI battle for organisations who can’t really use a lot of the other options because of data privacy issues. So while copilot isn’t as good as GPT, it’s still what we are using. But if it ever gets too expensive, it’s not as secure as you may think. Especially not if we start seeing more training sets, or EU and US relations worsens. I think the most likely outcome, at least here in the EU, is that anti-completion laws eventually takes a look at Office365 because of how monopolised it is. Or the EU actually follows through on their “a single vendor is a threat to national security” legislation and force half of the banking/energy/defense/andsoon industries to pick something other than Microsoft. Which will be hilariously hard, but if successful (which it probably won’t be because it’s hilariously hard) will lead to more open products.

pram 3y ago

I’ve found Mistral OpenOrca is pretty much as good as GPT4-turbo for creative writing/analysis. Actually it tends to output very similar text, which is suspicious, but whatever it saves me a lot of money. https://huggingface.co/Open-Orca/Mistral-7B-OpenOrca https://huggingface.co/Open-Orca/Mistral-7B-OpenOrca

SushiHippie 3y ago
Also openchat, which was trained on gpt4 conversations IIUC. https://github.com/imoneoi/openchat https://github.com/imoneoi/openchat
bugglebeetle 3y ago
Mistral OpenOrca is very good at task following as well. Its slightly less reliable than GPT 3.5/4, but the difference in quality for my text processing tasks is pretty much a toss-up.

calvinmorrison 3y ago

[flagged]

drakenot 3y ago

I've been somewhat disappointed with the performance of the open models. The claims of certain models outperforming GPT-3.5-Turbo and approaching GPT-4 fail to hold up to their benchmark results in real-world scenarios, potentially due to data contamination in assessments, based on my testing. As noted in the linked survey paper, some models may outperform 3.5-Turbo in specific, narrow areas, depending on the model. Yet, we still lack a general model that definitively exceeds 3.5-Turbo in all respects. I'm concerned that while we're still striving to reach 3.5-Turbo's performance level, OpenAI may unveil a new next-generation model, further widening the performance gap! Back in the summer, I had higher hopes that we would have surpassed the 3.5 threshold by now. The performance gap has been surprisingly large. It is especially noticeable in areas requiring consistent structured output or tool use from the LLM. This is where open models particularly falter.

snowycat 3y ago
There are tools that you can use to force the model to give structured output, such as llama.cpp GBNF grammar for example. They're a bit harder to use than ask gpt4 but they do work pretty well for what I use it for.
- white_dragon88 3y ago
  [dead]
raincole 3y ago
"OpenAI has no moat" aged so badly that it's almost a satire.
- Tostino 3y ago
  Try out my model vs gpt4 for the same tasks (I explicitly trained on) and compare. https://huggingface.co/Tostino/Inkbot-13B-8k-0.2 https://huggingface.co/Tostino/Inkbot-13B-8k-0.2 It's a 13b param model that isn't meant to be general purpose, but is meant to excel on the limited tasks I've trained on. You'll see more like this soon.

thorum 3y ago

Current ~70B models like LLAMA 2 70B are on par wih ChatGPT 3.5. The best smaller models can appear on par at first glance, but they hallucinate at a much higher rate and lack knowledge of the world. GPT 4 ‘gets’ things at a deeper level and no open source model is even close. A year is a good timeframe to evaluate things: the rest of the world seems to lag behind OpenAI by around 12-18 months, at least with LLMs and image generation. On the other hand open source tech usually has additional features for controlling output that OpenAI never bothers to implement, like llama.cpp’s grammars or ControlNet. So in that sense open source is usually ahead of OpenAI in terms of customizability.

omeze 3y ago
I dont think OpenAI is ever going to ahead in image generation, they were lapped very soon after dall-e and every real workflow Ive seen uses Midjourney or Stable Diffusion. The reverse (GPT 4 vision) is well ahead of open source though
- vunderba 3y ago
  The original is leagues behind anything current, but DALL-E version 3 absolutely blows any state of the art generative model out of the water, including mid journey 5.2 and SDXL in terms of pure prompt accuracy and coherence. Midjourney still has the edge in quality, but it's a moot point if it takes you 1000 v-rolls to get to your original vision. If all you're generating is anime waifus then MJ/NovelAI/Niji will suffice, but generating prompts particularly featuring relatively complex scenes or actions are amazing on DALL-E 3. And of course unfortunately, it goes without saying that open AI DALL-E is going to be the most restrictive in terms of censorship. I generated these from DALL-E 3 instantly. Try to generate them in any other commercial offering. Go ahead. I'll wait... https://imgur.com/a/2GTRjfK https://imgur.com/a/2GTRjfK Descriptions: A 80s photograph of the Koolaid Man breaking through the Berlin Wall. Comic illustration set at a festive children's party. The main focus is on the magician who looks uncannily like a well-known fictional wizard. He's trying to say abracadabra but accidentally uses the killing curse.
  - 3y ago

yieldcrv 3y ago

Amethyst Mistral 13B q5 gguf is what I’m using most of the time now. Synthetic datasets are great to finetune with, there is no moat for having inaccessible literature data sets I’m offline now because I’ve had too many ideas and domain names registered too soon after conversing with Chat GPT4 I’m open to the idea of people reacting to similar stimuli that cause ideas to be done at the same time, but I didn't like that experience and I can run these models on my M1 with LM Studio so easily I do think some chats get flagged when the model says something seems novel, like Albert Einstein working at the patent office. Not worth making it my whole identity in wanting to prove, just the catalyst I needed to try 7B and 13B models seriously and I’m quite pleased

deleted 3y ago
[deleted]
tarruda 3y ago
Is "Amethyst Mistral 13B" a llama fine tune? I searched for it on huggingface and only found the GGUF version, the link to the original model is broken
- Havoc 3y ago
  Mistral would be the base model
  - tarruda 3y ago
    Can a 7B base model be fine tuned into a 13B model? Mistral is 7B, this one is 13B.
    - Havoc 3y ago
      Two models can be merged from what I understand to result in different total param sizes. That’s different from a fine tune I gather the results of merges can be unpredictable though
dkarras 3y ago

matchagaucho 3y ago

If ChatGPT were only 1 LLM, then maybe. But it's a Mixture of Experts (MoE) architecture, which I think makes open source comparisons unfair?

zamadatix 3y ago
Open source options lacking parts of the ChatGPT approach which make it successful doesn't make the comparisons unfair it explains why ChatGPT wins right now. There's nothing stopping open source options from using the same MoE architecture, the solutions just don't (to the same effect act least) right now.

behnamoh 3y ago

[flagged]

bumbledraven 3y ago

[flagged]

benreesman 3y ago

`shiningvaliant-1.2-Q4_K_M` is my go-to. I appreciate that it doesn't top the boards in most metrics vs e.g. GPT-4, but I'm not in some A/B group on the quantization: it's more useful to me in practice more of the time. I have it rigged up with a prompt about outputting markdown and wired up to `foo | glow -` and I get GPT-4 out when I want something to write JIRA tickets no one is going to read because it's better at that sort of thing.

refibrillator 3y ago

It's not mentioned in the paper but this month OpenChat 3.5 released the first 7b model that achieves results comparable to ChatGPT in March 2023 [1]. Only 8k context window, but personally I've been very impressed with it so far. On the chatbot arena leaderboard it ranks above Llama-2-70b-chat [2]. In many ways open source LLMs are actually leading the industry, especially in terms of parameter efficiency and shipping useful models that consumers can run on their own hardware. [1] https://huggingface.co/openchat/openchat_3.5 https://huggingface.co/openchat/openchat_3.5 [2] https://chat.lmsys.org/ https://chat.lmsys.org/

ekianjo 3y ago
openchat is very impressive indeed. I think it may be better than Mistral, while comparisons are not always easy.
- tudorw 3y ago
  I'm finding Mistral good at creative literature and it is fairly adept at taking instructions, good enough for my purposes, and running locally on consumer CPU, the future of open source local models looks bright.
LoganDark 3y ago
> Only 8k context window Is this supposed to be low? All the chat models I've used top out at 4096.
- Sai_ 3y ago
  GPT-4-turbo is at 128k. Claude 2.1 is 200k. But yes, among open source models 8k is roughly middle to top of the pack.
  - LoganDark 3y ago
    That's insane. The highest I've personally seen in the open-source space is RWKV being trained on (IIRC) 4k but being able to handle much longer context lengths in practice due to being an RNN (you can simply keep feeding it tokens forever and ever). It doesn't generalize infinitely by any means but it can be stretched for sure, sometimes up to 16k. It's not a transformer model though, and old context fades away much faster / is harder to recall because all the new context is layered directly on top of it. But it's quite interesting nonetheless.

czk 3y ago

I'd say they are catching up for sure, especially with how GPT4 has been regressing consistently over the past month. https://chat.openai.com/share/c91287ee-9a5e-4c99-b5df-49cc4536e471 https://chat.openai.com/share/c91287ee-9a5e-4c99-b5df-49cc45...

tarruda 3y ago
I suspect a lot of the "catching up" was achieved by using GPT-4 API to generate high quality fine-tune datasets.

kaycebasques 3y ago

No comment from me on the question in the title (because I don't know enough to have an opinion), but since others are discussing various open models I will mention another that I've been enjoying tonight: DeepSeek 67B https://chat.deepseek.com https://chat.deepseek.com (This chat UI has adequately replaced my ChatGPT needs so far.) https://huggingface.co/deepseek-ai/deepseek-llm-67b-base https://huggingface.co/deepseek-ai/deepseek-llm-67b-base https://twitter.com/abacaj/status/1730019229175312612 https://twitter.com/abacaj/status/1730019229175312612

lhl 3y ago

A couple big/strong open models that have just been released the past few days: * Qwen 72B (and 1.8B) - 32K context, trained on 3T tokens, <100M MAU commercial license, strong benchmark performance: https://twitter.com/huybery/status/1730127387109781932 https://twitter.com/huybery/status/1730127387109781932 * DeepSeek LLM 67B - 4K context, 2T tokens, Apache 2.0 license, strong on code (although DeepSeek Code 33B it benches better) https://twitter.com/deepseek_ai/status/1729881611234431456 https://twitter.com/deepseek_ai/status/1729881611234431456 Also recently released: Yi 34B (with a 100B rumored soon), XVERSE-65B, Aquila2-70B, and Yuan 2.0-102B, interestingly, all coming out of China. Personally, I'm also looking forward to the larger Mistral releasing soon as mistral-7b-v0.1 was already incredibly strong for its size.

John9 3y ago
Since it's not allowed to use ChatGPT in China, there is a huge opportunity to build a local LLM.
- wenyuanyu 3y ago
  Anyone know what is the reason why both OpenAI and Anthropic proactively banned users from China from using their products...
  - wavemode 3y ago
    Source for this? I know China's government firewall blocks ChatGPT (for obvious reasons) but I wasn't aware that OpenAI was blocking them in return.
    - pototo666 3y ago
      Chinese IP are not allowed to use ChatGPT. Chinese credit card is not allowed for OpenAI API. Source: my own experience. What puzzles me most is the second restriction. My credit card is accepted by AWS, Google, and many other services. It is also accepted by many services which use Stripe to process payments. But OpenAI refuses to take my money.

davidkunz 3y ago

Out of personal experience, open source LLMs did not yet reach the quality of GPT 3.5, despite multiple claims with dubious benchmarks. That said, they are already useful as of today and can even run on your local machine. I regularly use them with my Neovim plugin gen.nvim [1] for simple tasks and they save me a lot of time. I'm excited about the future! [1]: https://github.com/David-Kunz/gen.nvim https://github.com/David-Kunz/gen.nvim

tarruda 3y ago
Very interesting. I want to give it a try, but I see that one of the dependencies is "ollama" which is a Mac App and I don't have a Mac. I'm running Llama models locally using llama-cpp-python which provides an OpenAI compatibility layer.
- rkwz 3y ago
  Oh, can install Ollama in Linux/WSL as well
- davidkunz 3y ago
  It should work on Linux and also with WSL. You could also try running it in Docker.
  - tarruda 3y ago
    I see. Does ollama have an http API (this the curl requirement)? If so, is it compatible with OpenAI API?
    - davidkunz 3y ago
      Yes, it works with an Ollama server and the communication is done via HTTP. I know that someone configured my plugin to talk to OpenAI.

intrepidsoldier 3y ago

Big fan of Starcoder

jurmous 3y ago

A problem I have with the open source models is that they are all not remotely good in many languages other than English compared to the OpenAI models. I specifically need Dutch and the outputs are unusable for us.

danielbln 3y ago
https://openchat.team/ https://openchat.team/ is quite good at German, how does it fare with Dutch?
- jurmous 3y ago
  As far as I can see it uses the OpenAI models which are quite good. Or am I wrong?
  - danielbln 3y ago
    I believe it's using a Mistral fine-tune, not GPT.
jug 3y ago
Yeah, this can be an issue. Today, there are few specialized LLM's of high quality so you end up having to use a massive all-in-one model like GPT-4 to reach for the language you need. There's movement here though and it will get better. GPT-SW3 is a new model developed by AI Sweden, trained specifically on the Nordic languages only + English. And beyond this, you have TrustLLM which is a new project that aims to be a large, open, European model trained on the Germanic languages to start with: https://liu.se/en/research/trustllm https://liu.se/en/research/trustllm
- jurmous 3y ago
  Yes there is some movement. Good to see trustllm being another initiative. The Dutch government starts with some non profit agencies to train a Dutch model too. But this will also take some time. https://www.tno.nl/en/newsroom/2023/11/netherlands-starts-realisation-gpt-nl/ https://www.tno.nl/en/newsroom/2023/11/netherlands-starts-re...

danofsteel32 3y ago

I've been using OpenHermes-2.5 [0] and NeuralHermes [1] which are both finetunes of the Mistral7B base model. The only objective test prompting I do is asking the models to generate a django timeclock/timesheets app. In this test they compare favorably to GPT-3.5. Also LMStudio [2] has a better UI than chatgpt and responses are much faster too (40tk/sec on my 2070). [0] https://huggingface.co/teknium/OpenHermes-2.5-Mistral-7B https://huggingface.co/teknium/OpenHermes-2.5-Mistral-7B [1] https://huggingface.co/TheBloke/NeuralHermes-2.5-Mistral-7B-AWQ https://huggingface.co/TheBloke/NeuralHermes-2.5-Mistral-7B-... [2] https://lmstudio.ai/ https://lmstudio.ai/

fintechie 3y ago

We're nearing a point where we'll just need a prompt router in front of several specialised models (code, chat, math, sql, health, etc)... and we'll have a local Mixture of Experts kind of thing. 1. Send request to router running a generic model. 2. Prompt/question is deconstructed, classified, and proxied to expert(s) xyz. 3. Responses come back and are assembled by generic model. Is any project working on something similar to this?

theturtle32 3y ago
This is how image generation works for DALL-E 3 via ChatGPT.
ekianjo 3y ago
it's kind of trivial today. the first layer could be a mix of nlp and zero-shot classification to clarify the nature of the request. Then using LLM deconstruct the request into several specific parts that would be sent to specialized LLMs. Then stitch it back together at the end again with LLM as the summarization machine. Problem is running so many LLMs in parallel means you need quite a bunch of resources.
- fintechie 3y ago
  Yeah, it shouldn't be too difficult to build this with python. I wonder why none of the popular routers like https://github.com/BerriAI/litellm https://github.com/BerriAI/litellm have this feature. > Problem is running so many LLMs in parallel means you need quite a bunch of resources. Top of line MacBooks or Minis should be able to run several 7B or even 13B models without major issues. Models are also getting smaller and better. That's why we're close =)
  - generalizations 3y ago
    Could lora fine tunes be used instead of completely different models? I wonder if that would save space.
    - 3y ago

Havoc 3y ago

Briefly yes, but in near term I expect falling behind. Sounds suspiciously like the next big leap in hosted models will be more computationally expensive on the inference side rather than training. For hosted that’s much of a sameness - just moving money allocations. But consumers can’t suddenly have 5x 4090s for local.

deleted 3y ago

[deleted]

deleted 3y ago
[deleted]

gumballindie 3y ago

Catching up to what? Chatgpt and clones are ridiculously bad at accurate text prediction, let alone coding.

lumost 3y ago

My hunch, The history of open source has been that companies who have customers with massive customization requirements land on the open source side of the equation. Companies who don't view a component as core to their product often land in a similar state. There is almost certainly at least one major firm that wants a GPT-5 like offering, but doesn't view the model as core to their business (Meta). It's also wholly unclear if large models are necessary - or simply convenient. In a similar veign, it's unclear that data must be labeled by humans - the open source data situation is getting better by the day. I'd expect that we'll see OpenAI hold an edge for many years, maybe we'll see a number two player as well for the foundation model, but after that everybody else will base off an open source FM and maybe keep the fine tuning/model augmentation proprietary.

iamgopal 3y ago

May be experts can answer, Can SETI like distributed setup possible with large model training ?

classified 3y ago

No. The moat is being optimized, and no open-source org can or would steal as much training data as the likes of OpenAI do.