8 ms·

I’ve got my stuff rigged to hit mixtral-8x7, and dolphin locally, and 3.5-turbo, and the 4-series preview all with easy comparison in emacs and stuff, and in fa

I’ve got my stuff rigged to hit mixtral-8x7, and dolphin locally, and 3.5-turbo, and the 4-series preview all with easy comparison in emacs and stuff, and in fairness the 4.5-preview is starting to show some edge on 8x7 that had been a toss-up even two weeks ago. I’m still on the mistral-medium waiting list. Until I realized Perplexity will give you a decent amount of Mistral Medium for free through their partnership. Who is sama kidding they’re still leading here? Mistral Medium destroys the 4.5 preview. And Perplexity wouldn’t be giving it away in any quantity if it had a cost structure like 4.5, Mistral hasn’t raised enough. Speculation is risky but fuck it: Mistral is the new “RenTech of AI”, DPO and Alibi and sliding window and modern mixtures are well-understood so the money is in the lag between some new edge and TheBloke having it quantized for a Mac Mini or 4070 Super, and the enterprise didn’t love the weird structure, remembers how much fun it was to be over a barrel to MSFT, and can afford to dabble until it’s affordable and operable on-premise. “Hate to see you go, love to watch you leave”.

icelancer 3y ago
> Mistral Medium destroys the 4.5 preview. On what metrics? LMSys shows it does well but 4-Turbo is still leading the field by a wide margin. I am using 8x-7b internally for a lot of things and Mistral-7b fine-tunes for other specific applications. They're both excellent. But neither can touch GPT-4-turbo (preview) for wide-ranging needs or the strongest reasoning requirements. https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar... EDIT: Neither does mistral-medium, which I didn't discuss, but is in the leaderboard link.
- benreesman 3y ago
  Keep in mind that modern quantitative approaches to LLM evaluation have been effectively co-designed with the rise of OpenAI, and folks like Ravenwolf routinely disagree with the leaderboards. There's also very little if any credible literature on what constitutes statistically significant on MMLU or whatever. There's such a massive vested interest from so many parties (the YC ecosystem is invested in Sam, MSFT is invested in OpenAI, the US is invested in not-France, a bunch of academics are invested in GPT-is-borderline-AGI, Yud is either a Time Magazine cover author or a Harry Potter fanfic guy, etc.) in seeing GPT-4.5 at the top of those rankings and taking the bold one at < 10% lift as state of the art that I think everyone should just use a bunch of them and optimize per use case. I have my own biases as well and freely admit that I love to see OpenAI stumble (no I didn't apply to work there, yes I know knuckleheads who go on about the fact they do). And once you factor in "mixtral is aligned to the demands of the user and GPT balks at using profanity while happily taking sides on things Ilya has double-spoken on", even e.g. MMLU is nowhere near the whole picture. It's easy and cheap to just try both these days, don't take my word for which one is better.
  - icelancer 3y ago
    > It's easy and cheap to just try both these days, don't take my word for which one is better. I literally use 8x-7b on my on-prem GPU cluster and have several fine tunes of 7b (which I said in the previous post). I've used mistral-medium. GPT-4-turbo is better than them all on all benchmarks, human preference, and anything that isn't biased vibes. My opinion - such that it is - is that GPT-4-turbo is by far the best. I have no vested interest in it being the best. I'd actually prefer if it wasn't. But all objective data points to it being the best and most lived experiences that are unbiased agree (assuming broad model use and not hyperfocused fine-tunes; I have Mistral-7b fine-tunes beating 4-turbo in very limited domains, but that hardly counts). The rest of your post I really have no idea what's going on, so good luck with all that I guess.
    - MacsHeadroom 3y ago
      Mistral Medium beats 4.5 on the censorship benchmark. It doesn't refuse to help with anything that could be vaguely non-PC or could potentially be used to hurt anyone in the wrong hands, including dangerously hot salsa recipes.
      - wokwokwok 3y ago
        That's not a metric. That's a use case. Certainly, no one here is arguing that there are things openai refuses to allow, and given that the effectiveness of using GPT4 on them is literally zero, a sweet potato connected to a spring and keyboard will "beat" GPT-4, if that's your scoring metric. If you want a meaningful comparison you need tasks that both tools are capable of doing, and then see how effective they are. Claiming that mistral medium beats it is like me claiming the RenderMan beats DALLE2 at rendering 3d models; yes, technically they both generate images, but since it's not possible to use DALLE2 to render a 3d model, it's not really a meaningful comparison is it?
        theshackleford 3y ago
        > If you want a meaningful comparison you need tasks that both tools are capable of doing, and then see how effective they are. The fact it’s incapable of simple requests that an alternative can is absolutely part of a worthwhile comparison.
- leo150 3y ago
  It just feels like “what LLM is better” becomes new “what GPU is better” type of talk. It’s great to find a clear winner, but at the end the gap between the leaders isn’t an order of magnitude.
  - speedgoose 3y ago
    These days the question is more about which LLM is second best. It’s very tight while ChatGPT 4 is in its own league.
  - seydor 3y ago
    I think people are missing the context that the prices of even the largest LLMs trend towards $0 in the medium term. Mistral-medium is almost open source, and we are still early days
hermiod 3y ago
Any chance you could post some comparisons between Mistral medium and gpt-4 turbo? I'm curious where you think it's more impressive, I hadn't spent the time to evaluate it yet.
- icelancer 3y ago
  Go to the Arena (side-by-side) tab on LMsys and you can try it yourself! https://chat.lmsys.org/ https://chat.lmsys.org/ It's a great tool they make available.
pyinstallwoes 3y ago
Can you share some examples of how you are using it? Mixtral that is? What's your setup? What's your flow/workflow?
- benreesman 3y ago
  I screenshotted my emacs session upthread in a bit of a cheeky "AI-talking-about-AI" joke: https://imgur.com/WDrqxsz https://imgur.com/WDrqxsz. While I heavily rely on `emacs` as my primary interface to all this stuff, I'm slowly-but-surely working on a curated and opinionated collection of bindings and tools and themes and shit for all the major hacker tools (VSCode, `nvim`, even to a degree the JetBrains ecosystem). This is all broadly part of a project I'm calling `hyper-modern` which will be MIT if I get to a release candidate at all. I have a `gRPC` service that wraps the outstanding work by the "`ggeranov` crew" loosely patterned on the sharded model-server architectures we used at FB/IG and mercilessly exploiting the really generous free-plan offered by the `buf.build` people (seriously, check out the `buf.build` people) in an effort to give hackers the best tools in a truly modern workflow. It's also an opportunity to surface some of the outstanding models that seem to have sunk without a trace (top of mind would be Segment Anything out of Meta and StyleTTS which obsoletes a bunch of well-funded companies) in a curated collection of hacker-oriented capabilities that aren't clumsy bullshit like co-pilot. Right now it's a name and a few thousand lines of code too rough to publish, but if I get it to a credible state the domain is `https://hyper-modern.ai https://hyper-modern.ai` and the code will be MIT at `https://github.com/hyper-modern-ai/ https://github.com/hyper-modern-ai/`.
  - rightbyte 3y ago
    Feel free to push the code. Elisp should be hacky and embarrassing otherwise you are doing it wrong :)
Demiurge 3y ago
I have 20 years of software development experience, and I couldn’t understand anything you said. Is there a dictionary for this new lingo, or am I just too mid?
- walteweiss 3y ago
  Oh thank you, I was reading and none of that made any sense to me. I thought it could be a presentation of some dumb AI output. Now I see I’m not alone.
  - Demiurge 3y ago
    That gave me an idea, here is what I got from Copilot: You have set up your system to run different AI models and compare their performance using a text editor. You are using Mixtral-8x7, a high-quality open-source model developed by Mistral AI, Dolphin, an emulator for Nintendo video games, 3.5-Turbo, a customized version of GPT-3.5, a powerful natural language model, and 4-Series Preview, a new version of the BMW sports coupe. You have noticed that the 4.5-Preview, an upcoming update of GPT-3.5, is slightly better than Mixtral-8x7, which used to be a close match. You are still waiting to access Mistral-Medium, a prototype model that is even better than Mixtral-8x7, but only available to a limited number of users. You have discovered that Perplexity, an AI company that provides information discovery and sharing services, offers free access to Mistral-Medium through their partnership with Mistral AI. You think that Perplexity is making a mistake by giving away such a valuable model, and that they are underestimating the superiority of Mistral-Medium over the 4.5-Preview. You also think that Mistral AI is the new leader in the AI industry, and that their techniques, such as DPO (Data Processing Optimization), Alibi (a library for algorithmic accountability), sliding window (a method for analyzing time series data), and modern mixtures (a way of combining different models), are well-known and effective. You believe that the advantage of Mistral AI lies in the gap between their innovation and the ability of other developers to replicate it on cheaper and more accessible hardware. You also think that the enterprise market is not fond of the complex structure of GPT-3.5 and its variants, and that they prefer to use Mistral AI's models, which are more affordable and operable on their own premises. You end your text with a quote from the movie Armageddon, which implies that you are leaving a situation that you dislike, but also admire.
rvba 3y ago
Was this generated by some AI? It it a parody?
- benreesman 3y ago
  I've made similar apologies upthread but I'm passionate about this being an inclusive conversation and so I'm trying to respond to everyone who I confused with all the jargon. The trouble with the jargon is that it obfuscates to a high degree even by the standards of the software space, and in a field where the impact on people's daily lives is at the high end of the range, even by the standards of the software space. HN routinely front-pages stuff where the math and CS involved is much less accessible, but for understandable reasons a somewhat tone-deaf comment like mine is disproportionately disruptive: people know this stuff matters to them either now or soon, and it's moving as quickly as anything does, and it's graduate-level material. If you have concrete questions about what probably looks like word salad I'll do my best to clarify (without the aid of an LLM).
nopinsight 3y ago
Curious that you mentioned "4.5-preview". What do you mean there? To my knowledge, and I searched to confirm, GPT-4.5 is not yet released. There were some rumors and a link to ChatGPT's answer about GPT-4.5 (could also be a hallucination) but Sam tweeted it was not true.
- callalex 3y ago
  They literally made it up.
  - benreesman 3y ago
    That seems a little harsh. There was clearly what amounted to an internal point release in Q4, there was a big announcement and the historical convention on versioning is `.5` increments. It's "unofficial" but "literally made it up" seems a bit unfair, it's not like I called it `GPT-4-Ti Founders Edition` and tried to list it on eBay.
- EmilStenstrom 3y ago
  They meant GPT-4 Turbo, which is an improvement over GPT-4.
dudeinjapan 3y ago
Speculative musings beckon, and we dare to embrace them. The crux of the matter appears to be the chasm that separates novel advancements from the moment they are quantified for mainstream consumption. Retaining vivid memories of past entanglements with industry titans, circumspectly explore and exploit these innovations until they become both affordable and practicable for on-premise utilization, finally unveiling competitive prowess. The overarching question looms large. Perhaps, Mistral has not yet amassed the financial resources commensurate with such largesse. "My hips don't lie."
- benreesman 3y ago
  https://gist.github.com/b7r6/fde6fb3be9a752a989054e62905307fb https://gist.github.com/b7r6/fde6fb3be9a752a989054e62905307f...
  - dudeinjapan 3y ago
    Not bad! It's having trouble with syllables tho.
logicchains 3y ago
>Alibi Do you have a source on Mistral/Mixtral using that?
- benreesman 3y ago
  No, they could be using any of the variants of pointwise scalar trig-style embedding, one imagines it's at least a little custom to their particular training setup. It was just an example of a modern positional encoding. I regret that I implied inside knowledge about that level of detail. They're doing something clever on scalar pointwise positional encoding but as for what who knows.
Prcmaker 3y ago
I'm still waiting for the AI encabulator.
- LeonM 3y ago
  Had a good laugh about your comment, then realized that this is _exactly_ what AI would be really good at... Basically let an AI hallucinate on some technical subject. It would make a great script for a new encabulator video.
asenna 3y ago
Dolphin-mixtral is incredible for the size that it is. But I'm curious, have you tried Goliath-120b or the new `Mixtral_34Bx2_MoE_60B` (it's named Mixtral but the base is actually Yi). Goliath is too big for my system but Mixtral_34Bx2_MoE_60B[1] is giving me some really good results. PSA to anyone that does not understand what we're talkign about: I was new to all of this until two weeks ago as well. If you want to get up to speed with the incredible innovation and home-tinkering happening with LLMs, you have to checkout - https://www.reddit.com/r/LocalLLaMA/ https://www.reddit.com/r/LocalLLaMA/ I believe we should be at GPT4 levels of intelligence locally sometime later this year (Possibly with the release of Llama3 or Mistral Medium open-model). [1] - https://huggingface.co/TheBloke/Mixtral_34Bx2_MoE_60B-GGUF https://huggingface.co/TheBloke/Mixtral_34Bx2_MoE_60B-GGUF
EmilStenstrom 3y ago
Here's a glossary to understand this post: - mixtral-8x7 or 8x7: Open source model by Mistral AI. - Dolphin: An uncensored version of the mistral model - 3.5-turbo: GPT-3.5 Turbo, the cheapest API from OpenAI - 4-series preview OR "4.5 preview": GPT-4 Turbo, the most capable API from OpenAI - mistral-medium: A new model by Mistral AI that they are only serving through AI. It's in private beta and there's a waiting list to access it. - Perplexity: A new search engine that is challenging Google by applying LLM to search - Sama: Sam Altman, CEO of OpenAI - RenTech: Renaissance Technologies, a secretive hedge fund known for delivering impressive returns improving on the work of others - DPO: Direct Preference Optimization. It is a technique that leverages AI feedback to optimize the performance of smaller, open-source models like Zephyr-7B1. - Alibi: a Python library that provides tools for machine learning model inspection and interpretation2. It can be used to explain the predictions of any black-box model, including LLMs. - Sliding window: a type of attention mechanism introduced by Mistral-7B3. It is used to support longer sequences in LLMs. - Modern mixtures: The process of using multiple models together, like "mixtral" is a mixture of several mistral models. - TheBloke: Open source developer that is very quick at quantizing all new models that come out - Quantize: Decreasing memory requirements of a new model by decreasing the precision of weights, typically with just minor performance degradation. - 4070 Super: NVIDIA 4070 Super, new graphics card announced just a week ago - MSFT: Microsoft
- azeirah 3y ago
  That's an impressive list of jargon whaha Love how deep the rabbithole has gone in just a year. I am unfortunately in the camp of understanding the post without needing a glossary. I should go outside more :|
- rrr_oh_man 3y ago
  I love you, Emil
- benreesman 3y ago
  I'm clearly spending far too much time tuning/training/using these things if a glossary to make my post comprehensible to HN is longer than my remark: thank you for correcting my error in dragging this sub-sub-sub-field into a thread of general interest.
kidsil 3y ago
I understand some of these words. In all seriousness, are self hosted GPT alternatives really viable?
eurekin 3y ago
Care to share, what are you using it for? I'm curious, because I'm gathering some usecases; so that I could share that internally in the company to provide better education on, what LLMs do and how they work.
coldtea 3y ago
Not sure what all the fuss is about about the incomprehensibility of this. It's a densely packed comment, information wise, and expects familiarity with the field, but there's nothing really that obscure about it. I might not know half of the references like "sama" or "TheBloke", but I could understand the context of them all. Like: "the lag between some new edge and TheBloke having it quantized for a Mac Mini or 4070 Super," Not sure who TheBloke is, but he obviously means "between some new (cutting) edge AI model, and some person scaling it to run on smaller computers with less memory". Similarly, not sure who Perplexity is, but "Until I realized Perplexity will give you a decent amount of Mistral Medium for free through their partnership" basically spells out that they're a service provider of some kind, that they have partnered with Mistral AI, and you get to use the Mistral Medium model through opening a free account on Perplexity. I mean, duh!
megablast 3y ago
What a non sensical statement
pama 3y ago
Thanks for the insights. What is your typical Emacs workflow for using and comparing the models?
- benreesman 3y ago
  I'm running custom stuff that I plan/hope to MIT soon, but `gptel` is killer and I've substantially plagiarized feature-wise it in my own dots. (I don't intend to release anything under a more permissive license that it was published under, merely that it sets the bar on a good interface and I plan to launch nothing less capable).
no_streams 3y ago
I'm curious about your workflow including all of these, is it only for your curiousity? Do you switch between them for specific tasks, or even run them in parallel for some purpose? Also, is anyone aware of a service that supplies API endpoints for dolphin? I'd love to experiment with it, but running locally exceeds my budget.
deleted 3y ago
[deleted]
brcmthrowaway 3y ago
If anyone understands this post you are worth a million dollars. Get that bag!

icelancer 3y ago

> Mistral Medium destroys the 4.5 preview. On what metrics? LMSys shows it does well but 4-Turbo is still leading the field by a wide margin. I am using 8x-7b internally for a lot of things and Mistral-7b fine-tunes for other specific applications. They're both excellent. But neither can touch GPT-4-turbo (preview) for wide-ranging needs or the strongest reasoning requirements. https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar... EDIT: Neither does mistral-medium, which I didn't discuss, but is in the leaderboard link.

benreesman 3y ago
Keep in mind that modern quantitative approaches to LLM evaluation have been effectively co-designed with the rise of OpenAI, and folks like Ravenwolf routinely disagree with the leaderboards. There's also very little if any credible literature on what constitutes statistically significant on MMLU or whatever. There's such a massive vested interest from so many parties (the YC ecosystem is invested in Sam, MSFT is invested in OpenAI, the US is invested in not-France, a bunch of academics are invested in GPT-is-borderline-AGI, Yud is either a Time Magazine cover author or a Harry Potter fanfic guy, etc.) in seeing GPT-4.5 at the top of those rankings and taking the bold one at < 10% lift as state of the art that I think everyone should just use a bunch of them and optimize per use case. I have my own biases as well and freely admit that I love to see OpenAI stumble (no I didn't apply to work there, yes I know knuckleheads who go on about the fact they do). And once you factor in "mixtral is aligned to the demands of the user and GPT balks at using profanity while happily taking sides on things Ilya has double-spoken on", even e.g. MMLU is nowhere near the whole picture. It's easy and cheap to just try both these days, don't take my word for which one is better.
- icelancer 3y ago
  > It's easy and cheap to just try both these days, don't take my word for which one is better. I literally use 8x-7b on my on-prem GPU cluster and have several fine tunes of 7b (which I said in the previous post). I've used mistral-medium. GPT-4-turbo is better than them all on all benchmarks, human preference, and anything that isn't biased vibes. My opinion - such that it is - is that GPT-4-turbo is by far the best. I have no vested interest in it being the best. I'd actually prefer if it wasn't. But all objective data points to it being the best and most lived experiences that are unbiased agree (assuming broad model use and not hyperfocused fine-tunes; I have Mistral-7b fine-tunes beating 4-turbo in very limited domains, but that hardly counts). The rest of your post I really have no idea what's going on, so good luck with all that I guess.
  - MacsHeadroom 3y ago
    Mistral Medium beats 4.5 on the censorship benchmark. It doesn't refuse to help with anything that could be vaguely non-PC or could potentially be used to hurt anyone in the wrong hands, including dangerously hot salsa recipes.
    - wokwokwok 3y ago
      That's not a metric. That's a use case. Certainly, no one here is arguing that there are things openai refuses to allow, and given that the effectiveness of using GPT4 on them is literally zero, a sweet potato connected to a spring and keyboard will "beat" GPT-4, if that's your scoring metric. If you want a meaningful comparison you need tasks that both tools are capable of doing, and then see how effective they are. Claiming that mistral medium beats it is like me claiming the RenderMan beats DALLE2 at rendering 3d models; yes, technically they both generate images, but since it's not possible to use DALLE2 to render a 3d model, it's not really a meaningful comparison is it?
      - theshackleford 3y ago
        > If you want a meaningful comparison you need tasks that both tools are capable of doing, and then see how effective they are. The fact it’s incapable of simple requests that an alternative can is absolutely part of a worthwhile comparison.
leo150 3y ago
It just feels like “what LLM is better” becomes new “what GPU is better” type of talk. It’s great to find a clear winner, but at the end the gap between the leaders isn’t an order of magnitude.
- speedgoose 3y ago
  These days the question is more about which LLM is second best. It’s very tight while ChatGPT 4 is in its own league.
- seydor 3y ago
  I think people are missing the context that the prices of even the largest LLMs trend towards $0 in the medium term. Mistral-medium is almost open source, and we are still early days

hermiod 3y ago

Any chance you could post some comparisons between Mistral medium and gpt-4 turbo? I'm curious where you think it's more impressive, I hadn't spent the time to evaluate it yet.

icelancer 3y ago
Go to the Arena (side-by-side) tab on LMsys and you can try it yourself! https://chat.lmsys.org/ https://chat.lmsys.org/ It's a great tool they make available.

pyinstallwoes 3y ago

Can you share some examples of how you are using it? Mixtral that is? What's your setup? What's your flow/workflow?

benreesman 3y ago
I screenshotted my emacs session upthread in a bit of a cheeky "AI-talking-about-AI" joke: https://imgur.com/WDrqxsz https://imgur.com/WDrqxsz. While I heavily rely on `emacs` as my primary interface to all this stuff, I'm slowly-but-surely working on a curated and opinionated collection of bindings and tools and themes and shit for all the major hacker tools (VSCode, `nvim`, even to a degree the JetBrains ecosystem). This is all broadly part of a project I'm calling `hyper-modern` which will be MIT if I get to a release candidate at all. I have a `gRPC` service that wraps the outstanding work by the "`ggeranov` crew" loosely patterned on the sharded model-server architectures we used at FB/IG and mercilessly exploiting the really generous free-plan offered by the `buf.build` people (seriously, check out the `buf.build` people) in an effort to give hackers the best tools in a truly modern workflow. It's also an opportunity to surface some of the outstanding models that seem to have sunk without a trace (top of mind would be Segment Anything out of Meta and StyleTTS which obsoletes a bunch of well-funded companies) in a curated collection of hacker-oriented capabilities that aren't clumsy bullshit like co-pilot. Right now it's a name and a few thousand lines of code too rough to publish, but if I get it to a credible state the domain is `https://hyper-modern.ai https://hyper-modern.ai` and the code will be MIT at `https://github.com/hyper-modern-ai/ https://github.com/hyper-modern-ai/`.
- rightbyte 3y ago
  Feel free to push the code. Elisp should be hacky and embarrassing otherwise you are doing it wrong :)

Demiurge 3y ago

I have 20 years of software development experience, and I couldn’t understand anything you said. Is there a dictionary for this new lingo, or am I just too mid?

walteweiss 3y ago
Oh thank you, I was reading and none of that made any sense to me. I thought it could be a presentation of some dumb AI output. Now I see I’m not alone.
- Demiurge 3y ago
  That gave me an idea, here is what I got from Copilot: You have set up your system to run different AI models and compare their performance using a text editor. You are using Mixtral-8x7, a high-quality open-source model developed by Mistral AI, Dolphin, an emulator for Nintendo video games, 3.5-Turbo, a customized version of GPT-3.5, a powerful natural language model, and 4-Series Preview, a new version of the BMW sports coupe. You have noticed that the 4.5-Preview, an upcoming update of GPT-3.5, is slightly better than Mixtral-8x7, which used to be a close match. You are still waiting to access Mistral-Medium, a prototype model that is even better than Mixtral-8x7, but only available to a limited number of users. You have discovered that Perplexity, an AI company that provides information discovery and sharing services, offers free access to Mistral-Medium through their partnership with Mistral AI. You think that Perplexity is making a mistake by giving away such a valuable model, and that they are underestimating the superiority of Mistral-Medium over the 4.5-Preview. You also think that Mistral AI is the new leader in the AI industry, and that their techniques, such as DPO (Data Processing Optimization), Alibi (a library for algorithmic accountability), sliding window (a method for analyzing time series data), and modern mixtures (a way of combining different models), are well-known and effective. You believe that the advantage of Mistral AI lies in the gap between their innovation and the ability of other developers to replicate it on cheaper and more accessible hardware. You also think that the enterprise market is not fond of the complex structure of GPT-3.5 and its variants, and that they prefer to use Mistral AI's models, which are more affordable and operable on their own premises. You end your text with a quote from the movie Armageddon, which implies that you are leaving a situation that you dislike, but also admire.

rvba 3y ago

Was this generated by some AI? It it a parody?

benreesman 3y ago
I've made similar apologies upthread but I'm passionate about this being an inclusive conversation and so I'm trying to respond to everyone who I confused with all the jargon. The trouble with the jargon is that it obfuscates to a high degree even by the standards of the software space, and in a field where the impact on people's daily lives is at the high end of the range, even by the standards of the software space. HN routinely front-pages stuff where the math and CS involved is much less accessible, but for understandable reasons a somewhat tone-deaf comment like mine is disproportionately disruptive: people know this stuff matters to them either now or soon, and it's moving as quickly as anything does, and it's graduate-level material. If you have concrete questions about what probably looks like word salad I'll do my best to clarify (without the aid of an LLM).

nopinsight 3y ago

Curious that you mentioned "4.5-preview". What do you mean there? To my knowledge, and I searched to confirm, GPT-4.5 is not yet released. There were some rumors and a link to ChatGPT's answer about GPT-4.5 (could also be a hallucination) but Sam tweeted it was not true.

callalex 3y ago
They literally made it up.
- benreesman 3y ago
  That seems a little harsh. There was clearly what amounted to an internal point release in Q4, there was a big announcement and the historical convention on versioning is `.5` increments. It's "unofficial" but "literally made it up" seems a bit unfair, it's not like I called it `GPT-4-Ti Founders Edition` and tried to list it on eBay.
EmilStenstrom 3y ago
They meant GPT-4 Turbo, which is an improvement over GPT-4.

dudeinjapan 3y ago

Speculative musings beckon, and we dare to embrace them. The crux of the matter appears to be the chasm that separates novel advancements from the moment they are quantified for mainstream consumption. Retaining vivid memories of past entanglements with industry titans, circumspectly explore and exploit these innovations until they become both affordable and practicable for on-premise utilization, finally unveiling competitive prowess. The overarching question looms large. Perhaps, Mistral has not yet amassed the financial resources commensurate with such largesse. "My hips don't lie."

benreesman 3y ago
https://gist.github.com/b7r6/fde6fb3be9a752a989054e62905307fb https://gist.github.com/b7r6/fde6fb3be9a752a989054e62905307f...
- dudeinjapan 3y ago
  Not bad! It's having trouble with syllables tho.

logicchains 3y ago

>Alibi Do you have a source on Mistral/Mixtral using that?

benreesman 3y ago
No, they could be using any of the variants of pointwise scalar trig-style embedding, one imagines it's at least a little custom to their particular training setup. It was just an example of a modern positional encoding. I regret that I implied inside knowledge about that level of detail. They're doing something clever on scalar pointwise positional encoding but as for what who knows.

Prcmaker 3y ago

I'm still waiting for the AI encabulator.

LeonM 3y ago
Had a good laugh about your comment, then realized that this is _exactly_ what AI would be really good at... Basically let an AI hallucinate on some technical subject. It would make a great script for a new encabulator video.

asenna 3y ago

Dolphin-mixtral is incredible for the size that it is. But I'm curious, have you tried Goliath-120b or the new `Mixtral_34Bx2_MoE_60B` (it's named Mixtral but the base is actually Yi). Goliath is too big for my system but Mixtral_34Bx2_MoE_60B[1] is giving me some really good results. PSA to anyone that does not understand what we're talkign about: I was new to all of this until two weeks ago as well. If you want to get up to speed with the incredible innovation and home-tinkering happening with LLMs, you have to checkout - https://www.reddit.com/r/LocalLLaMA/ https://www.reddit.com/r/LocalLLaMA/ I believe we should be at GPT4 levels of intelligence locally sometime later this year (Possibly with the release of Llama3 or Mistral Medium open-model). [1] - https://huggingface.co/TheBloke/Mixtral_34Bx2_MoE_60B-GGUF https://huggingface.co/TheBloke/Mixtral_34Bx2_MoE_60B-GGUF

EmilStenstrom 3y ago

Here's a glossary to understand this post: - mixtral-8x7 or 8x7: Open source model by Mistral AI. - Dolphin: An uncensored version of the mistral model - 3.5-turbo: GPT-3.5 Turbo, the cheapest API from OpenAI - 4-series preview OR "4.5 preview": GPT-4 Turbo, the most capable API from OpenAI - mistral-medium: A new model by Mistral AI that they are only serving through AI. It's in private beta and there's a waiting list to access it. - Perplexity: A new search engine that is challenging Google by applying LLM to search - Sama: Sam Altman, CEO of OpenAI - RenTech: Renaissance Technologies, a secretive hedge fund known for delivering impressive returns improving on the work of others - DPO: Direct Preference Optimization. It is a technique that leverages AI feedback to optimize the performance of smaller, open-source models like Zephyr-7B1. - Alibi: a Python library that provides tools for machine learning model inspection and interpretation2. It can be used to explain the predictions of any black-box model, including LLMs. - Sliding window: a type of attention mechanism introduced by Mistral-7B3. It is used to support longer sequences in LLMs. - Modern mixtures: The process of using multiple models together, like "mixtral" is a mixture of several mistral models. - TheBloke: Open source developer that is very quick at quantizing all new models that come out - Quantize: Decreasing memory requirements of a new model by decreasing the precision of weights, typically with just minor performance degradation. - 4070 Super: NVIDIA 4070 Super, new graphics card announced just a week ago - MSFT: Microsoft

azeirah 3y ago
That's an impressive list of jargon whaha Love how deep the rabbithole has gone in just a year. I am unfortunately in the camp of understanding the post without needing a glossary. I should go outside more :|
rrr_oh_man 3y ago
I love you, Emil
benreesman 3y ago
I'm clearly spending far too much time tuning/training/using these things if a glossary to make my post comprehensible to HN is longer than my remark: thank you for correcting my error in dragging this sub-sub-sub-field into a thread of general interest.

kidsil 3y ago

I understand some of these words. In all seriousness, are self hosted GPT alternatives really viable?

eurekin 3y ago

Care to share, what are you using it for? I'm curious, because I'm gathering some usecases; so that I could share that internally in the company to provide better education on, what LLMs do and how they work.

coldtea 3y ago

Not sure what all the fuss is about about the incomprehensibility of this. It's a densely packed comment, information wise, and expects familiarity with the field, but there's nothing really that obscure about it. I might not know half of the references like "sama" or "TheBloke", but I could understand the context of them all. Like: "the lag between some new edge and TheBloke having it quantized for a Mac Mini or 4070 Super," Not sure who TheBloke is, but he obviously means "between some new (cutting) edge AI model, and some person scaling it to run on smaller computers with less memory". Similarly, not sure who Perplexity is, but "Until I realized Perplexity will give you a decent amount of Mistral Medium for free through their partnership" basically spells out that they're a service provider of some kind, that they have partnered with Mistral AI, and you get to use the Mistral Medium model through opening a free account on Perplexity. I mean, duh!

megablast 3y ago

What a non sensical statement

pama 3y ago

Thanks for the insights. What is your typical Emacs workflow for using and comparing the models?

benreesman 3y ago
I'm running custom stuff that I plan/hope to MIT soon, but `gptel` is killer and I've substantially plagiarized feature-wise it in my own dots. (I don't intend to release anything under a more permissive license that it was published under, merely that it sets the bar on a good interface and I plan to launch nothing less capable).

no_streams 3y ago

I'm curious about your workflow including all of these, is it only for your curiousity? Do you switch between them for specific tasks, or even run them in parallel for some purpose? Also, is anyone aware of a service that supplies API endpoints for dolphin? I'd love to experiment with it, but running locally exceeds my budget.

deleted 3y ago

[deleted]

brcmthrowaway 3y ago

If anyone understands this post you are worth a million dollars. Get that bag!