5 ms·

I am using 8b-instruct and I am honestly blown away, this almost feels like early GPT4 levels of logic I am absolutely pumped for whats in store for small mode

I am using 8b-instruct and I am honestly blown away, this almost feels like early GPT4 levels of logic I am absolutely pumped for whats in store for small models!

kromem 2y ago
I dunno, I'm less than dazzled. One of my gotos is a variation of a classic logic puzzle, as models naturally tend to try to provide the original form of the solution because of token similarities and by seeing how well they can identify where steps are wrong or seeing if they can solve the variation when breaking token similarities with representative symbols or other techniques like mapping out object relationships in the initial answer before providing a solution it tends to reveal the underlying capacity for logic outside of simply similarities to the training data. Mistral, GPT-4, Claude, Gemini can all solve it with various degrees of competency and hand holding. Llama 3 just cannot handle it. I think what we're seeing is that the massive training data set is leading to a much more capable stochastic parrot but not necessarily as capable world modeling. Or else there's an advantage to MoE for this kind of task that Llama 3 lacks. But it's tracking with my subjective sense of being somewhere between GPT-3 and 4 in terms of "critical thinking" ability, but not in the same class as more modern models, even though it seems to have a better breadth of training to draw on than most other medium sized models.
- MyFirstSass 2y ago
  Both GPT-4 and llama 3 8b fail actual reasoning still: jack has 15 sisters, each sister has 3 brothers. How many brothers does jack have? (real answer is 2) ------------------- Llama 3 8b Let's break it down step by step: Jack has 15 sisters. Each of his sisters has 3 brothers. To find the total number of brothers, we need to multiply the number of sisters (15) by the number of brothers each sister has (3). 15 sisters x 3 brothers/sister = 45 brothers So, Jack has 45 brothers! ------------------- GPT-4 Jack has 3 brothers. Here's why: If each of Jack's sisters has 3 brothers, these brothers are the same for all sisters, including Jack himself. Therefore, Jack and his 3 brothers make up the total count of brothers in the family. Since the problem states that each sister has 3 brothers, we know that Jack himself must be counted as one of these brothers. So, Jack has 3 brothers. ------------------- Lol. You can tweak the questions numbers and names to avoid training data. Can Opus solve this?
  - rvnx 2y ago
    > User: jack has 15 sisters, each sister has 3 brothers. How many brothers does jack have? > Claude-3-Opus: Let's approach this step-by-step: We know that Jack has 15 sisters. Each of Jack's sisters has 3 brothers. Since all the sisters are from the same family, they all have the same brothers. Therefore, the 3 brothers that each sister has are the same 3 brothers. Since Jack is a brother to his sisters, he is one of these 3 brothers. So, Jack has 2 brothers. Therefore, Jack has 2 brothers.
  - joelthelion 2y ago
    Llama 3 70B seems to handle it just fine: A classic lateral thinking puzzle! Let's break it down: * Jack has 15 sisters. * Each sister has 3 brothers. Now, here's the key insight: if each sister has 3 brothers, that means each sister is counting Jack as one of their 3 brothers! So, how many brothers does Jack have? Only 2 The other 2 brothers that each sister is counting are actually Jack himself. Answer: Jack has 2 brothers.
    - MyFirstSass 2y ago
      Wow, that's extremely impressive in light of GPT being 1800b so a model 25 times smaller is beating it logic wise? I don't have enough ram to run 70b but i'm very curious about performance on other logic puzzles!
      - MyFirstSass 2y ago
        I just tried variations and weirdly it fails many of them like: "bob has 7 sisters, each sister has 4 brothers. How many brothers does bob have?" ----------------- Llama 70b instruct (wrong) A classic lateral thinking puzzle! At first glance, it seems like a straightforward math problem, but wait... If each of Bob's 7 sisters has 4 brothers, that would mean each sister has Bob and 3 other brothers, right? But that's not possible, because Bob is the only brother! So, the correct answer is: Bob has no brothers. He has 7 sisters, and each of them has him as their only brother. Did I get it right? ----------------- GPT-4 (correct) To solve this, let's consider the family structure: Bob has 7 sisters. Each of these sisters has 4 brothers. Since all the sisters have the same number of brothers, and these brothers include Bob, this implies Bob has 3 other brothers. Therefore, including Bob, there are 4 brothers in total. Hence, Bob has 3 brothers.
- dotsam 2y ago
  You can change the LLama 3 system prompt to coax the right answer. Shame it doesn't work out of the box though.
- pennomi 2y ago
  Man, if I was training an AI, I’d just create a dataset of Hacker News pet problems just to con everyone into thinking it’s somehow better at reasoning.
  - BoorishBears 2y ago
    I'm honestly amazed people still think riddles that are essentially asking a fish how well it climbs a tree are relevant to anything. Ask it to write code for you, ask it pop culture questions to test recall and hallucinations, literally a million more interesting things you can do.
pikseladam 2y ago
how did you tried it

I am using 8b-instruct and I am honestly blown away, this almost feels like early GPT4 levels of logic I am absolutely pumped for whats in store for small models!

kromem 2y ago

I dunno, I'm less than dazzled. One of my gotos is a variation of a classic logic puzzle, as models naturally tend to try to provide the original form of the solution because of token similarities and by seeing how well they can identify where steps are wrong or seeing if they can solve the variation when breaking token similarities with representative symbols or other techniques like mapping out object relationships in the initial answer before providing a solution it tends to reveal the underlying capacity for logic outside of simply similarities to the training data. Mistral, GPT-4, Claude, Gemini can all solve it with various degrees of competency and hand holding. Llama 3 just cannot handle it. I think what we're seeing is that the massive training data set is leading to a much more capable stochastic parrot but not necessarily as capable world modeling. Or else there's an advantage to MoE for this kind of task that Llama 3 lacks. But it's tracking with my subjective sense of being somewhere between GPT-3 and 4 in terms of "critical thinking" ability, but not in the same class as more modern models, even though it seems to have a better breadth of training to draw on than most other medium sized models.

MyFirstSass 2y ago
Both GPT-4 and llama 3 8b fail actual reasoning still: jack has 15 sisters, each sister has 3 brothers. How many brothers does jack have? (real answer is 2) ------------------- Llama 3 8b Let's break it down step by step: Jack has 15 sisters. Each of his sisters has 3 brothers. To find the total number of brothers, we need to multiply the number of sisters (15) by the number of brothers each sister has (3). 15 sisters x 3 brothers/sister = 45 brothers So, Jack has 45 brothers! ------------------- GPT-4 Jack has 3 brothers. Here's why: If each of Jack's sisters has 3 brothers, these brothers are the same for all sisters, including Jack himself. Therefore, Jack and his 3 brothers make up the total count of brothers in the family. Since the problem states that each sister has 3 brothers, we know that Jack himself must be counted as one of these brothers. So, Jack has 3 brothers. ------------------- Lol. You can tweak the questions numbers and names to avoid training data. Can Opus solve this?
- rvnx 2y ago
  > User: jack has 15 sisters, each sister has 3 brothers. How many brothers does jack have? > Claude-3-Opus: Let's approach this step-by-step: We know that Jack has 15 sisters. Each of Jack's sisters has 3 brothers. Since all the sisters are from the same family, they all have the same brothers. Therefore, the 3 brothers that each sister has are the same 3 brothers. Since Jack is a brother to his sisters, he is one of these 3 brothers. So, Jack has 2 brothers. Therefore, Jack has 2 brothers.
- joelthelion 2y ago
  Llama 3 70B seems to handle it just fine: A classic lateral thinking puzzle! Let's break it down: * Jack has 15 sisters. * Each sister has 3 brothers. Now, here's the key insight: if each sister has 3 brothers, that means each sister is counting Jack as one of their 3 brothers! So, how many brothers does Jack have? Only 2 The other 2 brothers that each sister is counting are actually Jack himself. Answer: Jack has 2 brothers.
  - MyFirstSass 2y ago
    Wow, that's extremely impressive in light of GPT being 1800b so a model 25 times smaller is beating it logic wise? I don't have enough ram to run 70b but i'm very curious about performance on other logic puzzles!
    - MyFirstSass 2y ago
      I just tried variations and weirdly it fails many of them like: "bob has 7 sisters, each sister has 4 brothers. How many brothers does bob have?" ----------------- Llama 70b instruct (wrong) A classic lateral thinking puzzle! At first glance, it seems like a straightforward math problem, but wait... If each of Bob's 7 sisters has 4 brothers, that would mean each sister has Bob and 3 other brothers, right? But that's not possible, because Bob is the only brother! So, the correct answer is: Bob has no brothers. He has 7 sisters, and each of them has him as their only brother. Did I get it right? ----------------- GPT-4 (correct) To solve this, let's consider the family structure: Bob has 7 sisters. Each of these sisters has 4 brothers. Since all the sisters have the same number of brothers, and these brothers include Bob, this implies Bob has 3 other brothers. Therefore, including Bob, there are 4 brothers in total. Hence, Bob has 3 brothers.
dotsam 2y ago
You can change the LLama 3 system prompt to coax the right answer. Shame it doesn't work out of the box though.
pennomi 2y ago
Man, if I was training an AI, I’d just create a dataset of Hacker News pet problems just to con everyone into thinking it’s somehow better at reasoning.
- BoorishBears 2y ago
  I'm honestly amazed people still think riddles that are essentially asking a fish how well it climbs a tree are relevant to anything. Ask it to write code for you, ask it pop culture questions to test recall and hallucinations, literally a million more interesting things you can do.

pikseladam 2y ago

how did you tried it