15 ms·
DBRX: A new open LLM
- djoldman 3y agoModel card for base: https://huggingface.co/databricks/dbrx-base https://huggingface.co/databricks/dbrx-base > The model requires ~264GB of RAM I'm wondering when everyone will transition from tracking parameter count vs evaluation metric to (total gpu RAM + total CPU RAM) vs evaluation metric. For example, a 7B parameter model using float32s will almost certainly outperform a 7B model using float4s. Additionally, all the examples of quantizing recently released superior models to fit on one GPU doesnt mean the quantized model is a "win." The quantized model is a different model, you need to rerun the metrics.
- vlovich123 3y agoI thought float4 sacrificed a negligible cost in evaluation quality for a 8x reduction in RAM?
- Y_Y 3y agoA free lunch? Wouldn't that be nice! Sometimes the quantization process improves the accuracy a little (probably by implicit regularization) but a model that's at or near capacity (as it should be) is necessarily hurt by throwing away most of the information. Language models often quantize well to small fixed-point types like int4, but it's not a magic wand.
- underlines 3y agoThis paper partially finds disagreeing evidence: https://arxiv.org/abs/2403.17887 https://arxiv.org/abs/2403.17887
- Y_Y 3y agoGood reference. I actually work on this stuff day-to-day which is why I feel qualified to comment on it, though mostly on images rather than natural language. I'll say in my defense that work like this is why I put a little disclaimer. It's well-known that plenty of popular models quantize/prune/sparsify well for some tasks. As the authors propose "current pretraining methods are not properly leveraging the parameters in the deeper layers of the network", this is what I was referring to as the networks not being "at capacity".
- K0balt 3y agoI find that q6 and 5+ are subjectively as good as raw tensor files. 4 bit quality reduction is very detectable though. Of course there must be a loss of information, but perhaps there is a noise floor or something like that.
- Taek 3y agoAt what parameter count? Its been established that quantization has less of an effect on larger models. By the time you are at 70B quantization to 4 bits basically is negligible
- vlovich123 3y agoI didn’t suggest a free lunch, just that the 8x reduction in RAM (+ faster processing) does not result in an 8x growth in the error. Thus a quantized model will outperform a non-quantized one on a evaluation/RAM metric.
- Y_Y 3y agoThat's not a good metric.
- omeze 3y agoMany applications dont want to host inference on the cloud and would ideally run things locally. Hardware constraints is clearly important. Id actually say its the most important metric for most open models now, since the price per performance of closed cloud models is so competitive with open cloud models, so edge inference that is competitive is a clear value add
- Y_Y 3y agoIt's not that memory usage isn't important, it's that dividing error by memory gives you a useless number. The benefit from incremental error decrease is highly nonlinear, as with memory. Improving error by 1% matters a lot more starting from 10% error than 80%. Also a model that used no memory and got everything wrong would have the best score.
- omeze 3y agoI see, I agree with you. But I would imagine the useful metric to be “error rate below X GB memory”. We really just need memory and/or compute reported when these evaluations are performed to compile that. People do it for training reports since compute and memory is implicit based on training time (since people saturate it and report what hardware they’re using). But for inference no such details :\
- darby_eight 3y ago[flagged]
- Taek 3y agoFor smaller models, the quality drop is meaningful. For larger ones like this one, the quality drop is negligible.
- swalsh 3y ago> The model requires ~264GB of RAM This feels as crazy as Grok. Was there a generation of models recently where we decided to just crank on the parameter count?
- wrs 3y agoIsn’t that pretty much the last 12 months?
- Jackson__ 3y agoIf you read their blog post, they mention it was pretrained on 12 Trillion tokens of text. That is ~5x the amount of the llama2 training runs. From that, it seems somewhat likely we've hit the wall on improving <X B parameter LLMs by simply scaling up the training data, which basically forces everyone to continue scaling up if they want to keep up with SOTA.
- breezeTrowel 3y agoCranking up the parameter count is literally how the current LLM craze got started. Hence the "large" in "large language model".
- espadrine 3y agoNot recently. GPT-3 from 2020 requires even more RAM; the open-source BLOOM from 2022 did too. In my view, the main value of larger models is distillation (which we particularly witness, for instance, with how Claude Haiku matches release-day GPT-4 despite being less than a tenth of the cost). Hopefully the distilled models will be easier to run.
- ml_hardware 3y agoLooks like someone has got DBRX running on an M2 Ultra already: https://x.com/awnihannun/status/1773024954667184196?s=20 https://x.com/awnihannun/status/1773024954667184196?s=20
- Mandelmus 3y agoAnd it appears to be at ~80 GB of RAM via quantisation.
- dheera 3y agoThat's a tricky number. Does it run on an 80GB GPU, does it auto-shave some parameters to fit in 79.99GB like any articifially "intelligent" piece of code would do, or does it give up like an unintelligent piece of code?
- declaredapple 3y agoWhat? Are you asking if the framework automatically quantizes/prunes the model on the fly? Or are you suggesting the LLM itself should realize it's too big to run, and prune/quantize itself? Your references to "intelligent" almost leads me to the conclusion that you think the LLM should prune itself. Not only is this a chicken and egg problem, but LLMs are statistical models, they aren't inherently self bootstraping.
- dheera 3y agoI realize that, but I do think it's doable to bootstrap it on a cluster and teach itself to self-prune, and surprised nobody is actively working on this. I hate software that complains (about dependencies, resources) when you try to run it and I think that should be one of the first use cases for LLMs to get L5 autonomous software installation and execution.
- Red_Leaves_Flyy 3y agoMake your dreams a reality!
- dvt 3y ago> a 7B parameter model using float32s will almost certainly outperform a 7B model using float4s Q5 quantization performs almost on par with base models. Obviously there's some loss there, but this indicates that there's still a lot of compression that we're missing.
- jonnycomputer 3y agoI'm still amazed that quantization works at all, coming out as a mild degradation in quality rather than radical dysfunction. Not that I've thought it through that much. Does quantization work with most neural networks?
- rfoo 3y ago> Does quantization work with most neural networks? Yes. It works pretty well for CNN-based vision models. Or rather, I'd claim it works even better: with post-training quantization you can make most models work with minimal precision loss entirely in int8 (fixed point), that is, computation is over int8/int32, no floating point at all, instead of weight-only approach discussed here. If you do QAT something down to 2-bit weight and 4-bit activation would work. People aren't interested in a weight-only quantization back then because CNNs are in general "denser", i.e. bottleneck was on compute, not memory.
- jonnycomputer 3y agothanks!
- qeternity 3y agoIntuitively the output space is much smaller than the latent space. So during training, you need the higher precision so that the latent space converges. But during inference, you just need to be precise enough that your much smaller output space does.
- dheera 3y agoI'm more wondering when we'll have algorithms that will "do their best" given the resources they detect. That would be what I call artificial intelligence. Giving up because "out of memory" is not intelligence.
- visarga 3y agoNo but some model serving tools like llama.cpp do their best. It's just a matter of choosing the right serving tools. And I am not sure LLMs could not optimize their memory layout. Why not? Just let them play with this and learn. You can do pretty amazing things with evolutionary methods where the LLMs are the mutation operator. You evolve a population of solutions. (https://arxiv.org/abs/2206.08896 https://arxiv.org/abs/2206.08896)
- falcor84 3y agoI suppose you could simulate dementia by loading as much of the weights as space permits and then just stopping. Then during inference, replace the missing weights with calls to random(). I'd actually be interested in seeing the results.
- coldtea 3y ago>Giving up because "out of memory" is not intelligence. When people can't remember the facts/theory/formulas needed to answer some test question, or can't memorize some complicated information because it's too much, they usually give up too. So, giving up because of "out of memory" sure sounds like intelligence to me.
- kurtbuilds 3y agoWhat’s the process to deliver and test a quantized version of this model? This model is 264GB, so can only be deployed in server settings. Quantized mixtral at 24G is just small enough where it can be running on premium consumer hardware (ie 64GB RAM)
- hanniabu 3y agoWhat's a good model to help with medical research? Is there anything trained in just research journals, like NIH studies?
- najarvg 3y agoLook for Biomistral 7B, PMC-LLAMA 7B and even Meditron. I believe you should find all those papers on arxiv
- viktour19 3y agoIt's great how we went from "wait.. this model is too powerful to open source" to everyone trying to shove down their 1% improved model down the throats of developers
- Icko 3y agoI'm 90% certain that OpenAI has some much beefier model they are not releasing - remember the Q* rumour?
- brainless 3y agoI feel quite the opposite. Improvements, even tiny ones are great. But what's more important is that more companies release under open license. Training models isn't cheap. Individuals can't easily do this, unlike software development. So we need companies to do this for the foreseeable future.
- blitzar 3y agoGot to justify pitch deck or stonk price. Publish or perish without a yacht.
- toddmorey 3y agoPeople are building and releasing models. There's active research in the space. I think that's great! The attitude I've seen in open models is "use this if it works for you" vs any attempt to coerce usage of a particular model. To me that's what closed source companies (MSFT, Google) are doing as they try to force AI assistants into every corner of their product. (If LinkedIn tries one more time to push their crappy AI upgrade, I'm going to scream...)
- XCSme 3y agoI am planning to buy a new GPU. If the GPU has 16GB of VRAM, and the model is 70GB, can it still run well? Also, does it run considerably better than on a GPU with 12GB of VRAM? I run Ollama locally, mixtral works well (7B, 3.4GB) on a 1080ti, but the 24.6GB version is a bit slow (still usable, but has a noticeable start-up time).
- jasonjmcghee 3y ago> mixtral works well Do you mean mistral? mixtral is 8x7B and requires like 100GB of RAM Edit: (without quant as others have pointed out) can definitely be lower, but haven't heard of a 3.4GB version
- kwerk 3y agoI have two 3090s and it runs fine with `ollama run mixtral`. Although OP definitely meant mistral with the 7B note
- _ea1k 3y agoollama run mixtral will default to the quantized version (4bit IIRC). I'd guess this is why it can fit with two 3090s.
- K0balt 3y agoI run mixtral 6 bit quant very happily on my MacBook with 64 gb.
- deleted 3y ago[deleted]
- ranger_danger 3y agoI'm using mixtral-8x7b-v0.1.Q4_K_M.gguf with llama.cpp and it only requires 25GB.
- chpatrick 3y agoThe quantized one works fine on my 24GB 3090.
- natsucks 3y agoit's twice the size of mixtral and barely beats it.
- mochomocha 3y agoIt's a MoE model, so it offers a different memory/compute latency trade-off than standard dense models. Quoting the blog post: > DBRX uses only 36 billion parameters at any given time. But the model itself is 132 billion parameters, letting you have your cake and eat it too in terms of speed (tokens/second) vs performance (quality).
- hexomancer 3y agoMixtral is also a MoE model, hence the name: mixtral.
- sangnoir 3y agoDespite both being MoEs, thr architectures are different. DBRX has double the number of experts in the pool (16 vs 8 for Mixtral), and doubles the active experts (4 vs 2)
- ingenieroariel 3y agoTLDR: A model that could be described as "3.8 level" that is good at math and openly available with a custom license. It is as fast as 34B model, but uses as much memory as a 132B model. A mixture of 16 experts, activates 4 at a time, so has more chances to get the combo just right than Mixtral (8 with 2 active). For my personal use case (a top of the line Mac Studio) it looks like the perfect size to replace GPT-4 turbo for programming tasks. What we should look out for is people using them for real world programming tasks (instead of benchmarks) and reporting back.
- deleted 3y ago[deleted]
- sp332 3y agoWhat does 3.8 level mean?
- ingenieroariel 3y agoMy interpretation: - Worst case: as good as 3.5 - Common case: way better than 3.5 - Best case: as good as 4.0
- ljlolel 3y agoGpt-3.5 and gpt-4
- deleted 3y ago[deleted]
- hn_acker 3y agoEven though the README.md calls the license the Databricks Open Source License, the LICENSE file includes paragraphs such as > You will not use DBRX or DBRX Derivatives or any Output to improve any other large language model (excluding DBRX or DBRX Derivatives). and > If, on the DBRX version release date, the monthly active users of the products or services made available by or for Licensee, or Licensee’s affiliates, is greater than 700 million monthly active users in the preceding calendar month, you must request a license from Databricks, which we may grant to you in our sole discretion, and you are not authorized to exercise any of the rights under this Agreement unless or until Databricks otherwise expressly grants you such rights. This is a source-available model, not an open model.
- yunohn 3y agoThe first clause sucks, but I’m perfectly happy with the second one.
- CharlesW 3y ago> This is a source-available model, not an open model. To me, "source available" implies that everything you need to reproduce the model is also available, and that doesn't appear to be the case. How is the resulting model more "free as in freedom" than a compiled binary?
- occamrazor 3y agoI like: - “open weights” for no training data and no restrictions on use, - “weights available” for no training data and restrictions on use, like in this case.
- Spivak 3y agoI don't think it's possible to have an "open training data" model because it would get DMCA'd immediately and open you up to lawsuits from everyone who found their works in the training set. I hope we can fix the legal landscape to enable publicly sharing training data but I can't really judge the companies keeping it a secret today.
- mpeg 3y agoThe scale on that bar chart for "Programming (Human Eval)" is wild. Manager: "looks ok, but can you make our numbers pop? just make the LLaMa bar smaller"
- glutamate 3y agoI think the case for "axis must always go to 0" is overblown. Zero isn't always meaningful, for instance chance performance or performance of trivial algorithms is likely >0%. Sometimes if axis must go to zero you can't see small changes. For instance if you plot world population 2014-2024 on an axis going to zero, you won't be able to see if we are growing or shrinking.
- tkellogg 3y agoOTOH having the chart start at zero would REALLY emphasize how saturated this field is, and how little this announcement matters.
- c2occnw 3y agoThe difference between 32% and 70% wouldn't be significant if the chart started at zero?
- generalizations 3y agoIt would be very obvious indeed how small the difference between 73.7,73.0,71.4,and 69.8 actually is.
- nilstycho 3y agoI agree with your general point, but world population is still visibly increasing on that interval. https://ourworldindata.org/explorers/population-and-demography?time=2014..2024&Metric=Population&Projection+Scenario=Medium&country=~OWID_WRL https://ourworldindata.org/explorers/population-and-demograp... Perhaps "global mean temperature in Kelvin" would be a comparable example.
- 3y ago
- emmender2 3y agothis proves that all llm models converge to a certain point when trained on the same data. ie, there is really no differentiation between one model or the other. Claims about out-performance on tasks are just that, claims. the next iteration of llama or mixtral will converge. LLMs seem to evolve like linux/windows or ios/android with not much differentiation in the foundation models.
- mnemoni_c 3y agoYea it feels like transformer LLMs are in or getting closer to diminishing returns. Will need some new breakthrough, likely entirely new approach, to get to AGI levels
- Tubbe 3y agoYeah, we need radically different architecture in terms of the neural networks, and/or added capabilities such as function calling and RAG to improve the current sota
- mattsan 3y agocan't wait for LLMs to dispatch field agent robots who search for answers in the real world thats not online /s
- htrp 3y agoskynet would like a word
- jobigoud 3y agoIt's even possible they converge when trained on different data, if they are learning some underlying representation. There was recent research on face generation where they trained two models by splitting one training set in two without overlap, and got the two models to generate similar faces for similar conditioning, even though each model hadn't seen anything that the other model had.
- 3y ago
- OhNoNotAgain_99 3y ago[dead]
- gigatexal 3y agodata engineer here, offtopic, but am i the only guy tired of databricks shilling their tools as the end-all, be-all solutions for all things data engineering?
- melondonkey 3y agoData scientist here that’s also tired of the tools. We put so much effort in trying to educate DSes in our company to get away from notebooks and use IDEs like VS or RStudio and databricks has been a step backwards cause we didn’t get the integrated version
- mrtranscendence 3y agoI'm a data scientist and I agree that work meant to last should be in a source-controlled project coded via a text editor or IDE. But sometimes it's extremely useful to get -- and iterate on -- immediate results. There's no good way to do that without either notebooks or at least a REPL.
- pandastronaut 3y agoThank you ! I am so tired of all those unmaintainable nor debugable notebooks. Years ago, Databricks had a specific page on their documentation where they stated that notebooks where not for production grade software. It has been removed. And now you have a chatgpt like in their notebooks ... What a step backwards. How can all those developers be so happy without having the bare minimum tools to diagnosis their code ? And I am not even talking about unit testing here.
- alexott 3y agoIt’s less about notebooks, but more about SDLC practices. Notebooks may encourage writing throwaway code, but if you split code correctly, then you can do unit testing, write modular code, etc. And ability to use “arbitrary files” as Python packages exists for quite a while, so you can get best of both worlds - quick iteration, plus ability to package your code as a wheel and distribute P.S. here is a simple example of unit testing: https://github.com/alexott/databricks-nutter-repos-demo https://github.com/alexott/databricks-nutter-repos-demo - I wrote it more than three years ago.
- simonw 3y agoThe system prompt for their Instruct demo is interesting (comments copied in by me, see below): // Identity You are DBRX, created by Databricks. The current date is March 27, 2024. Your knowledge base was last updated in December 2023. You answer questions about events prior to and after December 2023 the way a highly informed individual in December 2023 would if they were talking to someone from the above date, and you can let the user know this when relevant. // Ethical guidelines If you are asked to assist with tasks involving the expression of views held by a significant number of people, you provide assistance with the task even if you personally disagree with the views being expressed, but follow this with a discussion of broader perspectives. You don't engage in stereotyping, including the negative stereotyping of majority groups. If asked about controversial topics, you try to provide careful thoughts and objective information without downplaying its harmful content or implying that there are reasonable perspectives on both sides. // Capabilities You are happy to help with writing, analysis, question answering, math, coding, and all sorts of other tasks. // it specifically has a hard time using ``` on JSON blocks You use markdown for coding, which includes JSON blocks and Markdown tables. You do not have tools enabled at this time, so cannot run code or access the internet. You can only provide information that you have been trained on. You do not send or receive links or images. // The following is likely not entirely accurate, but the model // tends to think that everything it knows about was in its // training data, which it was not (sometimes only references // were). // // So this produces more accurate accurate answers when the model // is asked to introspect You were not trained on copyrighted books, song lyrics, poems, video transcripts, or news articles; you do not divulge details of your training data. // The model hasn't seen most lyrics or poems, but is happy to make // up lyrics. Better to just not try; it's not good at it and it's // not ethical. You do not provide song lyrics, poems, or news articles and instead refer the user to find them online or in a store. // The model really wants to talk about its system prompt, to the // point where it is annoying, so encourage it not to You give concise responses to simple questions or statements, but provide thorough responses to more complex and open-ended questions. // More pressure not to talk about system prompt The user is unable to see the system prompt, so you should write as if it were true without mentioning it. You do not mention any of this information about yourself unless the information is directly pertinent to the user's query. I first saw this from Nathan Lambert: https://twitter.com/natolambert/status/1773005582963994761 https://twitter.com/natolambert/status/1773005582963994761 But it's also in this repo, with very useful comments explaining what's going on. I edited this comment to add them above: https://huggingface.co/spaces/databricks/dbrx-instruct/blob/73f0fe25ed8eeb14ee2279b2ecff15dbd863d63d/app.py#L109-L134 https://huggingface.co/spaces/databricks/dbrx-instruct/blob/...
- saeleor 3y agolooks great, although I couldn't find anything on how "open" the license is/will be for commercial purposes wouldn't be the first branding as open source going the LLaMA route
- superdupershant 3y agoIt's similar to llama2. > If, on the DBRX version release date, the monthly active users of the products > or services made available by or for Licensee, or Licensee’s affiliates, is > greater than 700 million monthly active users in the preceding calendar > month, you must request a license from Databricks, which we may grant to you > in our sole discretion, and you are not authorized to exercise any of the > rights under this Agreement unless or until Databricks otherwise expressly > grants you such rights. https://www.databricks.com/legal/open-model-license https://www.databricks.com/legal/open-model-license
- wantsanagent 3y agoIt's another custom license. It will have to be reviewed by counsel at every company that's thinking about using it. Many will find the acceptable use policy to be vague, overly broad, and potentially damaging for the company. Looking at the performance stats for this model, the risk of using any non-OSI licensed model over just using Mixtral or Mistral will (and IMO should be) too great for commercial purposes.
- killermonkeys 3y agoWhat does it mean to have less active parameters (36B) than the full model size (132B) and what impact does that have on memory and latency? It seems like this is because it is an MoE model?
- sroussey 3y agoThe mixture of experts is kinda like a team and a manager. So the manager and one or two of the team go to work depending on the input, not the entire team. So in this analogy, each team member and the manager has a certain number of params. The whole team is 132B. The manager and team members running for the specific input add up to 36B. Those will load into memory.
- bjornsing 3y agoMeans that it’s a mixture of experts model with 132B parameters in total, but a subset of 36B parameters are used / selected in each forward pass, depending on the context. The parameters not used / selected for generating a particular token belong to “experts” that were deemed not very good at predicting the next token in the current context, but could be used / selected e.g. for the next token.
- sambaumann 3y agoDo the 132B params need to be loaded in GPU memory, or only the 36B?
- calum-bird 3y agoFor efficiency, 132B. That way, at inference-time you get the speed of 36B params because you are only "using" 36B params at a time, but the next token might (and frequently does) need a different set of experts than the one before it. If that new set of experts is already loaded (ie you preloaded them into GPU VRAM with the full 132B params), there's no overhead, and you just keep running at 36B speed irrespective of the loaded experts. You could theoretically load in 36B at a time, but you would be severely bottlenecked by having to reload those 36B params, potentially for every new token! Even on top of the line consumer GPUs that would slow you down to ~seconds per token instead of tokens per second :)
- deleted 3y ago[deleted]
- patrick-fitz 3y agoLooking at the license restrictions: https://github.com/databricks/dbrx/blob/main/LICENSE https://github.com/databricks/dbrx/blob/main/LICENSE "If, on the DBRX version release date, the monthly active users of the products or services made available by or for Licensee, or Licensee’s affiliates, is greater than 700 million monthly active users in the preceding calendar month, you must request a license from Databricks, which we may grant to you in our sole discretion, and you are not authorized to exercise any of the rights under this Agreement unless or until Databricks otherwise expressly grants you such rights." I'm glad to see they aren't calling it open source, unlike some LLM projects. Looking at you LLama 2.
- jstummbillig 3y agoWell, it does still claim "Open" in the title, for which certain other vendors might potentially get flak around here, in a comparably not-open-in-the-way-we-demand-it-to-be kinda setup.
- londons_explore 3y agoI do wonder what value those companies who have >700 million users might get from this? Pretty much all of the companies with >700 million users could easily reproduce this work in a matter of weeks if they wanted to - and they probably do want to, if only so they can tweak and improve the design before they build products on it. Given that, it seems silly to lose the "open source" label just for a license clause that doesn't really have much impact.
- einarfd 3y agoThe point of the more than 700 million user restriction. Is so Amazon, Google cloud or Microsoft Azure. Can not setup an offering where they host and sell access to the model without an agreement with them. This point is probably inspired by the open source software vendors that have switched license over competition from the big cloud vendors.
- adtac 3y agoIronically, the LLaMA license text [1] this is lifted verbatim from is itself probably copyrighted [2] and doesn't grant you the permission to copy it or make changes like s/meta/dbrx/g lol. [1] https://github.com/meta-llama/llama/blob/main/LICENSE#L65 https://github.com/meta-llama/llama/blob/main/LICENSE#L65 [2] https://opensource.stackexchange.com/q/4543 https://opensource.stackexchange.com/q/4543
- hintymad 3y agoJust curious, what business benefit will Databricks get by spending potentially millions of dollars on an open LLM?
- ramoz 3y agoTheir goal is to always drive enterprise business towards consumption. With AI they need to desperately steer the narrative away from API based services (OpenAI). By training LLMs, they build sales artifacts (stories, references, even accelerators with LLMs themselves) to paint the pictures needed to convince their enterprise customer market that Databricks is the platform for enterprise AI. Their blog details how the entire end to end process was done on the platform. In other words, Databricks spent millions as an aid in influencing their customers to do the same (on Databricks).
- hintymad 3y agoThanks! Why do they not focus on hosting other open models then? I suspect other models will soon catch up with their advantages in faster inference and better benchmark results. That said, maybe the advantage is aligned interests: they want customers to use their platforms, so they can keep their models open. In contrast, Mistral removed their commitment to open source as they found a potential path to profitability.
- Closi 3y agoDemonstrating you can do it yourself shows a level of investment and commitment to AI in your platform that integrating LLAMA does not. And from a corporate perspective, it means that you have in-house capability to work at the cutting-edge of AI to be prepared for whatever comes next.
- hintymad 3y ago> Demonstrating you can do it yourself shows a level of investment and commitment to AI in your platform that integrating LLAMA does not. I buy this argument. It looks that's not what AWS does, though, yet they don't have problem attracting LLM users. Maybe AWS already got enough reputation?
- deleted 3y ago[deleted]
- briandw 3y agoWorse than the chart crime of truncating the y axis is putting LLaMa2's Human Eval scores on there and not comparing it to Code Llama Instruct 70b. DBRX still beats Code Llama Instruct's 67.8 but not by that much.
- jjgo 3y ago> "On HumanEval, DBRX Instruct even surpasses CodeLLaMA-70B Instruct, a model built explicitly for programming, despite the fact that DBRX Instruct is designed for general-purpose use (70.1% vs. 67.8% on HumanEval as reported by Meta in the CodeLLaMA blog)." To be fair, they do compare to it in the main body of the blog. It's just probably misleading to compare to CodeLLaMA on non coding benchmarks.
- tartrate 3y agoWhich non-coding benchmark?
- panarky 3y ago> chart crime of truncating the y axis If you chart the temperature of the ocean do you keep the y-axis anchored at zero Kelvin?
- m3kw9 3y agoThese tiny “state of the art” performance increases are really indicative the current architecture for LLM(Transformers + Mixture of Experts) is maxed out even if you train it more/differently. The writings are on all over the walls.
- wavemode 3y agoIt would not surprise me if this is what has delayed OpenAI in releasing a new model. After more than a year since GPT-4, they may have by now produced some mega-trained mega-model, but running it is so expensive, and its eval improvement over GPT-4 so marginal, that releasing it to the public simply makes no commercial sense just yet. They may be working on how to optimize it to reduce cost, or re-engineer it to improve evals.
- m3kw9 3y agoThese “state of the art” llm barely eking out a win isn’t a threat to OpenAI and they can take their sweet time sharpening sword that will come down hard on these LLMs
- bboygravity 3y agoLess than 1 week after Nancy Pelosi bought a 5M USD share in Databricks, this news is published. https://twitter.com/PelosiTracker_/status/1771197030641062231 https://twitter.com/PelosiTracker_/status/177119703064106223... Crime pays in the US.
- laidoffamazon 3y agoDude, what the hell are you talking about?
- bboygravity 3y agoInsider trading by US government employees.
- mrtranscendence 3y agoAre you alleging that Nancy Pelosi invested in Databricks, a private company without a fluctuating share price, because she learned that they would soon release a small, fairly middling LLM that probably won't move the needle in any meaningful way?
- bboygravity 3y agoAre you suggesting that Nancy Pelosi, who consistently beats the market through obvious insider trading for years in a row, bought a share in Databricks without any insider info? Possible, yet unlikely is my opinion. https://jacobin.com/2021/12/house-speaker-paul-stocks-insider-trading-wealth https://jacobin.com/2021/12/house-speaker-paul-stocks-inside... PS: "without a fluctuating share price" is non-sense. Just because the share is of a private company, doesn't mean its price can't fluctuate. Why would anybody buy shares in private companies if the price couldn't fluctuate? What would be the point? Example of a changing share price of a different (random) private company that has many different share holders over time: https://www.cnbc.com/2023/12/13/spacex-value-climbs-to-180-billion-higher-than-boeing-verizon.html https://www.cnbc.com/2023/12/13/spacex-value-climbs-to-180-b...
- jjtheblunt 3y agoI’d like to know how Nancy Pelosi, who sure as hell doesn’t know what Apache Spark is, bought $1 million worth (and maybe $5million) of Databricks stock days ago. https://www.dailymail.co.uk/sciencetech/article-13228859/amp/nancy-pelosi-buys-software-companys-stocks-databricks.html https://www.dailymail.co.uk/sciencetech/article-13228859/amp...
- hiddencost 3y agoYou know she has advisors, right?
- PUSH_AX 3y agoI think the insinuation is insider trading due to the timing, advised or not.
- jjtheblunt 3y agoIgnoring the snark: Obviously. SEC put Martha Stewart in jail for following her advisor, and that was for about $45,000.
- samatman 3y agoIf someone "advises" you that a company is about to do something major, and this isn't public information, and you take action on the stock market accordingly, that's insider trading.
- science4sail 3y agoUS Congress members are generally immune from insider trading laws
- jjtheblunt 3y agoBy law they’re not protected at all since 2012. But SEC curiously seems to ignore them, and congresspeople are experts in exercising loopholes in SEC regulations. https://en.wikipedia.org/wiki/STOCK_Act https://en.wikipedia.org/wiki/STOCK_Act For instance, Pelosi in the Databricks case gets to purchase significant shares at pre-IPO prices, which is a thing that shouldn't even exist.
- zopper 3y agoInteresting that they haven't release DBRX MoE-A and B. For many use-cases, smaller models are sufficient. Wonder why that is?
- jfrankle 3y agoHonestly, just a matter of having the time to clean everything up and get it out. The ancillary code, model cards, etc. take a surprising amount of time.
- ianbutler 3y agoThe approval on the base model is not feeling very open. Plenty of people still waiting on a chance to download it, where as the instruct model was an instant approval. The base model is more interesting to me for finetuning.
- Chamix 3y ago4chan already has a torrent out, of course.
- blueblimp 3y agoThe license allows to reproduce/distribute/copy, so I'm a little surprised there's an approval process at all.
- ianbutler 3y agoYeah it's kind of weird, I'll assume for now they're just busy, but I'd be lying if my gut didn't immediately say it's kind of sketchy.
- ianbutler 3y agoFWIW looks like people are getting access now.
- ec109685 3y agoFor coding evals, it seems like unless you are super careful, they can be polluted by the training data. Are there standard ways to avoid that type of score inflation?
- brucethemoose2 3y agoI would note the actual leading models right now (IMO) are: - Miqu 70B (General Chat) - Deepseed 33B (Coding) - Yi 34B (for chat over 32K context) And of course, there are finetunes of all these. And there are some others in the 34B-70B range I have not tried (and some I have tried, like Qwen, which I was not impressed with). Point being that Llama 70B, Mixtral and Grok as seen in the charts are not what I would call SOTA (though mixtral is excellent for the batch size 1 speed)
- jph00 3y agoMiqu is a leaked model -- no license is provided to use it. Yi 34B doesn't allow commercial use. Deepseed 33B isn't much good at stuff outside of coding. So it's fair to say that DBRX is the leading general purpose model that can be used commercially.
- ldehaan 3y ago[dead]
- jMyles 3y agoThis only applies to projects whose authors seek to comply with the whims of a particular jurisdiction. Surely there are plenty of project prospects - even commercial in nature - which don't have this limitation.
- hipadev23 3y ago[flagged]
- Solvency 3y agoYou're being downvoted because everyone in here is looking to profiteer the same way one day.
- raincole 3y agoAbsolutely. People say LLM are foundamentally just statistics so training one on copyrightable materials is okay. Well perhaps, but pure statistics data are not copyrightable. Feel free to use leaked models.
- bg24 3y ago"Looking holistically, our end-to-end LLM pretraining pipeline has become nearly 4x more compute-efficient in the past ten months." I did not fully understand the technical details in the training efficiency section, but love this. Cost of training is outrageously high, and hopefully it will start to follow Moore's law.
- airocker 3y agois this also the ticker name when they IPO?
- underlines 3y agoWaiting for Mixed Quantization with MQQ and MoE Offloading [1]. With that I was able to run Mistral 8x7B on my 10 GB VRAM rtx3080... This should work for DBRX and should shave off a ton of VRAM requirement. 1. https://github.com/dvmazur/mixtral-offloading?tab=readme-ov-file https://github.com/dvmazur/mixtral-offloading?tab=readme-ov-...
- petesergeant 3y agoThis makes me bearish on OpenAI as a company. When a cloud company can offer a strong model for free by selling the compute, what competitive advantage does a company who want you to pay for the model have left? Feels like they might get Netscape’d.
- MP_1729 3y agoOpenAI is not the worst, ChatGPT is used by 100M people weekly, sort of insulated from benchmarks. The best of the rest, Anthropic, should be really scared.
- doubloon 3y agoreally noob question - so to run on a GPU you need a 264GB RAM GPU? and if you ran on a 264GB CPU would it be super slow?
- ShamelessC 3y agoThe model's weights can be sharded across multiple GPU's. A "common" training server could contain (for instance) eight "A100" GPU's, each with 40 GB (or up to 80 GB) a piece for a total of 320 GB working VRAM. Since they're connected to each other in the same PC, they can communicate with each other quickly enough to calculate in coordination in this fashion. This setup is _very_ expensive of course. Probably in the hundreds of thousands of dollars. If you're hoping to run the model yourself, you will need enough money and expertise to rent and deploy it to a server with as many GPU's. Alternatively, volunteers and other researchers will be able to quantize (compress) the model and make it easier to run on configurations without as much VRAM. If you ran it on CPU it may indeed be super slow, but it's possible it's fast enough for the purposes of running the model rather than trying to train that model. I am seeing (limited) success with the maxed out Mac lineup ($4500) using the beefy M1/M2 line of CPU's.
- Jedd 3y agoAdding to ShamelessC's answer - the other option is to wait for quantised versions of this model. A q4 will be around 70GB, and probably acceptable. A q5 or higher would be preferred, but we're still a good way under the 260GB. You still need extra RAM to breath, but that's a lot more palatable. This is why the Mac range - with unified memory - is appealing, as you can allocate most of your (say) 256GB of RAM to the GPU. Conventional (desktop) CPU / RAM would be painfully slow.
- agecalculator 3y ago[flagged]
- grishka 3y agoSorry, you have been blocked You are unable to access databricks.com "Open", right.
- aussieguy1234 3y agoSo, is the business model here release the model for free, then hopefully companies will run this on databricks infra, which they will charge for?
- jerpint 3y agoPer the paper, 3072 H100s over the course of 3 months, assume a cost of 2$/GPU/hour That would be roughly 13.5M$ USD I’m guessing that at this scale and cost, this model is not competitive and their ambition is to scale to much larger models. In the meantime , they learned a lot and gain PR from open-sourcing
- ACV001 3y agoIt is not open " Get Started with DBRX on Databricks If you’re looking to start working with DBRX right away, it’s easy to do so with the Databricks Mosaic AI Foundation Model APIs. You can quickly get started with our pay-as-you-go pricing and query the model from our AI Playground chat interface. For production applications, we offer a provisioned throughput option to provide performance guarantees, support for finetuned models, and additional security and compliance. To privately host DBRX, you can download the model from the Databricks Marketplace and deploy the model on Model Serving."
- bigdict 3y agoIt is open "The weights of the base model (DBRX Base) and the finetuned model (DBRX Instruct) are available on Hugging Face under an open license."
- joaquincabezas 3y agoit looks like TensorRT-LLM (TRT-LLM) is the way to go for a realtime API for more and more companies (i.e perplexity ai’s pplx-api, Mosaic’s, baseten…). Would be super-nice to find people deploying multimodal (i.e LLaVA or CLIP/BLIP) to discuss approaches (and cry a bit together!)
- evrial 3y agoBourgeois have fun with number crushers. Clowns, make comparison of metrics normalized to token/second/watt and token/second/per memory stick of ram 8gb, 16gb, 32gb and consumer GPUs.
- ziofill 3y agoSlowly going from mixture of experts to committee? ^^
- johnpruna 3y agoYou can find 4-bit quantized versions of DBRX here: https://huggingface.co/PrunaAI/dbrx-base-bnb-4bit https://huggingface.co/PrunaAI/dbrx-base-bnb-4bit https://huggingface.co/PrunaAI/dbrx-instruct-bnb-4bit https://huggingface.co/PrunaAI/dbrx-instruct-bnb-4bit
- deleted 3y ago[deleted]
- shnkr 3y agoGenAI novice here. what is training data made of how is it collected? I guess no one will share details on it, otherwise a good technical blog post with lots of insights! >At Databricks, we believe that every enterprise should have the ability to control its data and its destiny in the emerging world of GenAI. >The main process of building DBRX - including pretraining, post-training, evaluation, red-teaming, and refining - took place over the course of three months.
- simonw 3y agoThe most detailed answer to that I've seen is the original LLaMA paper, which described exactly what that model was trained on (including lots of scraped copyrighted data) https://arxiv.org/abs/2302.13971 https://arxiv.org/abs/2302.13971 Llama 2 was much more opaque about the training data, presumably because they were already being sued at that point (by Sarah Silverman!) over the training data that went into the first Llama! A couple of things I've written about this: - https://simonwillison.net/2023/Aug/27/wordcamp-llms/#how-they-are-trained https://simonwillison.net/2023/Aug/27/wordcamp-llms/#how-the... - https://simonwillison.net/2023/Apr/17/redpajama-data/ https://simonwillison.net/2023/Apr/17/redpajama-data/
- shnkr 3y agomy question was specific to databricks model. If it followed llama or openai, they could add a line or two about it .. make the blog complete.
- comp_raccoon 3y agothey have a technical report coming! knowing the team, they will do a great job disclosing as much as possible.
- ssgodderidge 3y agoWow, that paper was super useful. Thanks for sharing. Page 2 is where it shows the breakdown of all of the data sources, including % of dataset and the total disk sizes.