5 ms·

This is my hello world test for coding assistant for LLMs. CodeLlama and GPT4 crush the answer. I love seeing innovation in the space but Mistral kind of failed

This is my hello world test for coding assistant for LLMs. CodeLlama and GPT4 crush the answer. I love seeing innovation in the space but Mistral kind of failed me on my first test: --------------------- >>> Write code that will track swipes in Typescript . Swipe should be detected when the user moves a finger over 30% of the screen, and then drags it all the way to the end of the screen. ## Getting started You need to have yarn or npm installed before you can run this app ### Running in browser ```bash yarn install yarn serve ``` ### Running with docker Use the docker-compose file located at ./docker/docker-compose.yml ```bash cd <path-to-project> docker-compose up ``` ## Screenshot ![Screenshot](./images/screenshot.png)

capableweb 3y ago
CodeLlama is obviously trained on code specifically, so maybe not a useful comparison. GPT4 is also way different than Mistral, with a huge step up in parameters and seems to be using multi-agent approach too. Since Mistral is just a 7B parameter model, it's obvious that you won't be able to have it straight up write accurate code, it's simply too small for being able to accomplish something like that, unless you train the model specifically for writing code up front. I guess if all you're looking for is a model to write code for you, that makes sense as a "hello world" test, but then you're looking at the wrong model here. What you really want to do if you're looking for a good generalized model, is to run a bunch of different tests against it, from different authors, average/aggregate a score based on those and then rank all the models based on this score. Luckily, huggingface already put this all in place, and can be seen here: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... This Mistral 7B model seems to earn itself a 3rd place compared to the rest of the 7B models added to the leaderboard. Edit: As mentioned by another commentator, this also seems to be a base model, not trained specifically for request<>reply/chat/instructions. They're (or someone) is meant to fine-tune this model for that, if they want to.
- qeternity 3y ago
  > and seems to be using multi-agent approach too. What do you mean by this? MoE?
pclmulqdq 3y ago
I'm pretty sure this model is not instruct tuned, so it's not exactly apples-to-apples.
amilios 3y ago
Not a fair comparison as Mistral is not instruction tuned. You need to write a "continuation-style" prompt or few-shot it, rather than just giving it a direct order. Honestly this is one of the reasons imo that Mistral won't see wide acceptance, LLaMA-2-7b-chat is RLHF and instruction tuned and is just so much easier to use as a result.
- code51 3y ago
  What about this one? https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1 https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1
  - amilios 3y ago
    Huh! Nevermind then! I take it back. Would be interesting to see what kind of tuning they did/pit the model head-to-head with LLaMA-2-7B-chat. Seems like they did just instruction tuning but not RLHF? So I assume Mistral won't be refusing to answer etc., probably doesn't have many safety guardrails (I guess that's desirable for some!)
black_puppydog 3y ago
You didn't enjoy the screenshot? :)
code51 3y ago
Here is the response from Instruct model: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1 https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1 To track swipes in TypeScript, we can use a library like `react-native-gesture-handler`. Here's an example of how to use it to track horizontal and vertical swipes: ```typescript import React from 'react'; import { GestureHandlerEvent, GestureHandlerProps } from 'react-native-gesture-handler'; const SwipeableContent: React.FC<GestureHandlerProps> = ({ onStart, onEnd, onMove }) => { const handleHorizontalSwipe = (event: GestureHandlerEvent) => { if (event.type === 'horizontal') { onStart(event); } else { onEnd(event); } }; const handleVerticalSwipe = (event: GestureHandlerEvent) => { if (event.type === 'vertical') { onStart(event); } else { onEnd(event); } }; return ( <View> <PanGestureHandler onStart={handleHorizontalSwipe} onMove={onMove}> <Text>Drag me horizontally</Text> </PanGestureHandler> <PanGestureHandler onStart={handleVerticalSwipe} onMove={onMove}> <Text>Drag me vertically</Text> </PanGestureHandler> </View> ); }; ``` In this example, we use the `PanGestureHandler` component from `react-native-gesture-handler` to track horizontal and vertical swipes. The `onStart`, `onEnd`, and `onMove` props are passed to the `PanGestureHandler` component to handle the different types of gestures. You can then use this component in your app to track swipes on any view: ```typescript const App = () => { const [isHorizontalSwiping, setIsHorizontalSwiping] = useState(false); const [isVerticalSwiping, setIsVerticalSwiping] = useState(false); const handleStart = (event: GestureHandlerEvent) => { if (event.type === 'horizontal') { setIsHorizontalSwiping(true); } else { setIsVerticalSwiping(true); } }; const handleEnd = (event: GestureHandlerEvent) => { if (event.type === 'horizontal') { setIsHorizontalSwiping(false); } else { setIsVerticalSwiping(false); } }; const handleMove = (event: GestureHandlerEvent) => { console.log('Gesture moved'); }; return ( <View> <SwipeableContent onStart={handleStart} onEnd={handleEnd} onMove={handleMove} /> <Text>{isHorizontalSwiping ? 'Horizontal swipe is in progress' : ''}</Text> <Text>{isVerticalSwiping ? 'Vertical swipe is in progress' : ''}</Text> </View> ); }; ``` In this example, we use the `SwipeableContent` component to track horizontal and vertical swipes. We also track the status of the swipe using state variables to show a message when a swipe is in progress.

capableweb 3y ago

CodeLlama is obviously trained on code specifically, so maybe not a useful comparison. GPT4 is also way different than Mistral, with a huge step up in parameters and seems to be using multi-agent approach too. Since Mistral is just a 7B parameter model, it's obvious that you won't be able to have it straight up write accurate code, it's simply too small for being able to accomplish something like that, unless you train the model specifically for writing code up front. I guess if all you're looking for is a model to write code for you, that makes sense as a "hello world" test, but then you're looking at the wrong model here. What you really want to do if you're looking for a good generalized model, is to run a bunch of different tests against it, from different authors, average/aggregate a score based on those and then rank all the models based on this score. Luckily, huggingface already put this all in place, and can be seen here: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... This Mistral 7B model seems to earn itself a 3rd place compared to the rest of the 7B models added to the leaderboard. Edit: As mentioned by another commentator, this also seems to be a base model, not trained specifically for request<>reply/chat/instructions. They're (or someone) is meant to fine-tune this model for that, if they want to.

qeternity 3y ago
> and seems to be using multi-agent approach too. What do you mean by this? MoE?

pclmulqdq 3y ago

I'm pretty sure this model is not instruct tuned, so it's not exactly apples-to-apples.

amilios 3y ago

Not a fair comparison as Mistral is not instruction tuned. You need to write a "continuation-style" prompt or few-shot it, rather than just giving it a direct order. Honestly this is one of the reasons imo that Mistral won't see wide acceptance, LLaMA-2-7b-chat is RLHF and instruction tuned and is just so much easier to use as a result.

code51 3y ago
What about this one? https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1 https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1
- amilios 3y ago
  Huh! Nevermind then! I take it back. Would be interesting to see what kind of tuning they did/pit the model head-to-head with LLaMA-2-7B-chat. Seems like they did just instruction tuning but not RLHF? So I assume Mistral won't be refusing to answer etc., probably doesn't have many safety guardrails (I guess that's desirable for some!)

black_puppydog 3y ago

You didn't enjoy the screenshot? :)

code51 3y ago

Here is the response from Instruct model: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1 https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1 To track swipes in TypeScript, we can use a library like `react-native-gesture-handler`. Here's an example of how to use it to track horizontal and vertical swipes: ```typescript import React from 'react'; import { GestureHandlerEvent, GestureHandlerProps } from 'react-native-gesture-handler'; const SwipeableContent: React.FC<GestureHandlerProps> = ({ onStart, onEnd, onMove }) => { const handleHorizontalSwipe = (event: GestureHandlerEvent) => { if (event.type === 'horizontal') { onStart(event); } else { onEnd(event); } }; const handleVerticalSwipe = (event: GestureHandlerEvent) => { if (event.type === 'vertical') { onStart(event); } else { onEnd(event); } }; return ( <View> <PanGestureHandler onStart={handleHorizontalSwipe} onMove={onMove}> <Text>Drag me horizontally</Text> </PanGestureHandler> <PanGestureHandler onStart={handleVerticalSwipe} onMove={onMove}> <Text>Drag me vertically</Text> </PanGestureHandler> </View> ); }; ``` In this example, we use the `PanGestureHandler` component from `react-native-gesture-handler` to track horizontal and vertical swipes. The `onStart`, `onEnd`, and `onMove` props are passed to the `PanGestureHandler` component to handle the different types of gestures. You can then use this component in your app to track swipes on any view: ```typescript const App = () => { const [isHorizontalSwiping, setIsHorizontalSwiping] = useState(false); const [isVerticalSwiping, setIsVerticalSwiping] = useState(false); const handleStart = (event: GestureHandlerEvent) => { if (event.type === 'horizontal') { setIsHorizontalSwiping(true); } else { setIsVerticalSwiping(true); } }; const handleEnd = (event: GestureHandlerEvent) => { if (event.type === 'horizontal') { setIsHorizontalSwiping(false); } else { setIsVerticalSwiping(false); } }; const handleMove = (event: GestureHandlerEvent) => { console.log('Gesture moved'); }; return ( <View> <SwipeableContent onStart={handleStart} onEnd={handleEnd} onMove={handleMove} /> <Text>{isHorizontalSwiping ? 'Horizontal swipe is in progress' : ''}</Text> <Text>{isVerticalSwiping ? 'Vertical swipe is in progress' : ''}</Text> </View> ); }; ``` In this example, we use the `SwipeableContent` component to track horizontal and vertical swipes. We also track the status of the swipe using state variables to show a message when a swipe is in progress.