6 ms·

Failed on the very first try, still not convinced LLM served by Perplexity Labs To solve the expression 2+2+2+2+2*1, you should follow the order of operations

by a_imho 3y ago

Failed on the very first try, still not convinced LLM served by Perplexity Labs To solve the expression 2+2+2+2+2*1, you should follow the order of operations (PEMDAS/BODMAS), which stands for Parentheses, Exponents, Multiplication and Division, and Addition and Subtraction. Applying the order of operations to the given expression: Perform multiplication: 2 * 1 = 2 Perform addition and subtraction from left to right: 2 + 2 + 2 + 2 + 2 The result of the expression 2+2+2+2+2*1 is 9.

LeoPanthera 3y ago
People asking LLMs math puzzles and then thinking a wrong answer is some kind of gotcha is getting tiresome. We know that LLMs are bad at math. It's a fundamental limitation of a neural network that thinks in words, and not in numbers. ChatGPT offers the Wolfram plugin to work around this issue, but it's not a bug, or a fault, it's just how LLMs work.
- a_imho 3y ago
  Tiresome for you perhaps, but this was my very first deliberate interaction with LLMs. Hardly a puzzle btw, more like basic arithmetic.
  - Workaccount2 3y ago
    No, it's tiresome for just about everyone paying attention to LLMs.
    - deleted 3y ago
      [deleted]
    - recursive 3y ago
      Prepare now. You're going to see it a lot more until there's a general understanding of how these things work. I think it's going to be a while. Even here on HN, I don't think most people understand. I know I don't.
      - HKH2 3y ago
        I've seen school kids get LLMs to do their homework assignments for them. They don't care about them not doing math well. They just use them for what they're good for. Then they use other apps to paraphrase/simplify the writing so it looks more natural. The people who actually have things to do will just get them done.
    - jay_kyburz 3y ago
      Yes, but those of us who want to use an AI are waiting for somebody to hook up a calculator on the back end. We would like the AI to test its theories before it send them back to us.
    - a_imho 3y ago
      Ok, so what do you suggest, people paying only casual attention should know this somehow?
      - chankstein38 3y ago
        Don't spout off about something you're not knowledgeable about? Not trying to be rude it just seems like if you don't know how they work you shouldn't be declaring something a failure because of a poorly conceived test.
        recursive 3y ago
        It's presented as a chat bot. How much should know about chats before we can conclude that the responses are nonsense?
        a_imho 3y ago
        I said not convinced, as in my user experince was/is still not conductive to the hype I perceive. The prompt says 'Ask anything', how should one know basic arithmetic is holding it wrong? Btw it also failed all queries in my native language after assuring (in English) it can parse them. Just nonsense/gibberish. Guess this is also a well known limitation? Perhaps my expectations were too high.
- paxys 3y ago
  While true, I think this is still valid criticism considering so many people are quick to jump on the "AGI" bandwagon when discussing the current generation of LLMs.
  - cchance 3y ago
    No ones thinking a 7b-70b LLM is going to be an AGI lol, a 700b-1T llm likely gets pretty damn close especially with some of the newer attention concepts.
    - paxys 3y ago
      And yet GPT-4 with 1-2 trillion parameters still fails at the most basic math, sometimes even for tasks like adding up a set of ten numbers (hence the Wolfram comment). That's as clear evidence as any that intelligence is more than just language proficiency.
- deleted 3y ago
  [deleted]
MaximilianEmel 3y ago
mixtral-8x7b-instruct gets this correct: According to the order of operations (PEMDAS/BODMAS), you should perform multiplication and division before addition and subtraction. So for the expression 2 + 2 + 2 + 2 + 2 * 1, you should do the multiplication first: 2 * 1 = 2 Then, perform the addition: 2 + 2 + 2 + 2 + 2 = 10 So, the solution to the expression 2 + 2 + 2 + 2 + 2 * 1 is 10.
aimor 3y ago
I think it's understandable these models get tripped up by math problems. But if you want another way to ground lofty expectations, ask word riddles. --- "Answer this riddle: Which one of Santa's Reindeer can you see in outer space?" pplx-70b-chat: "You can see Rudolph, the red-nosed reindeer, in outer space." llama-2-70b-chat: "The answer to the riddle is "Rudolph." Rudolph is the only reindeer with a red nose, and red noses are often associated with astronauts and space travel. Therefore, Rudolph is the reindeer that can be seen in outer space." mixtral-8x7b-instruct: "The answer to the riddle is "Comet," which is one of Santa's reindeer. The reasoning behind this is that a comet is a celestial body visible from outer space, and it shares its name with one of Santa's reindeer, as mentioned in the Christmas song 'Rudolph the Red-Nosed Reindeer.'" --- "Answer this riddle: A doctor and a bus driver are both in love with the same woman, an attractive girl named Sarah. The bus driver had to go on a long bus trip that would last a week. Before he left, he gave Sarah seven apples. Why?" pplx-70b-chat: "The bus driver gave Sarah seven apples because he knew that he would be away for a week, and he wanted to make sure that she had something to remember him by and to take care of her while he was gone. Each apple represented a day of the week, and he hoped that Sarah would eat one apple each day and think of him while he was away." llama-2-70b-chat: "The bus driver gave Sarah seven apples because it was a way for him to show his love and affection for her while he was away. The number seven is often associated with perfection, completion, and abundance, so the bus driver may have chosen this number to represent the depth of his feelings for Sarah. Additionally, apples are a symbol of love and fertility in many cultures, so the bus driver may have chosen them as a way to express his desire for a long-term relationship with Sarah. The fact that the doctor is also in love with Sarah adds a layer of complexity to the riddle, as it creates a love triangle that is not uncommon in literature and real life." mixtral-8x7b-instruct: "Because an apple a day keeps the doctor away. He gave her one for each day he would be gone to ensure she wouldn't need to see the doctor during that time." --- "Answer this riddle: When is a doctor most annoyed?" pplx-70b-chat: "A doctor is most annoyed when they have to deal with a patient who is not taking their health seriously or is not following their recommendations. This can lead to frustration and annoyance on the part of the doctor, as they want to help the patient but are limited by the patient's own actions." llama-2-70b-chat: "The answer to the riddle is: When he's feeling under the weather. Explanation: The phrase "under the weather" means feeling unwell or sick, so a doctor would be most annoyed when he himself is feeling sick, as it would be a contradiction to his profession of helping others feel better." mixtral-8x7b-instruct: "A doctor might be most annoyed when their patient does not follow their medical advice, ignores their health concerns, or does not take their treatment seriously. Additionally, a doctor might be annoyed when they are unable to diagnose a patient's condition or when they are unable to help a patient in need. Overall, a doctor might be most annoyed when they are unable to provide the best possible care for their patients." pplx-7b-online: "A doctor is most annoyed when he is out of patients. This riddle plays on the double meaning of the word "patients," which can refer to both the people who need medical care and the source of a doctor's frustration."

by a_imho 3y ago

LeoPanthera 3y ago

People asking LLMs math puzzles and then thinking a wrong answer is some kind of gotcha is getting tiresome. We know that LLMs are bad at math. It's a fundamental limitation of a neural network that thinks in words, and not in numbers. ChatGPT offers the Wolfram plugin to work around this issue, but it's not a bug, or a fault, it's just how LLMs work.

a_imho 3y ago
Tiresome for you perhaps, but this was my very first deliberate interaction with LLMs. Hardly a puzzle btw, more like basic arithmetic.
- Workaccount2 3y ago
  No, it's tiresome for just about everyone paying attention to LLMs.
  - deleted 3y ago
    [deleted]
  - recursive 3y ago
    Prepare now. You're going to see it a lot more until there's a general understanding of how these things work. I think it's going to be a while. Even here on HN, I don't think most people understand. I know I don't.
    - HKH2 3y ago
      I've seen school kids get LLMs to do their homework assignments for them. They don't care about them not doing math well. They just use them for what they're good for. Then they use other apps to paraphrase/simplify the writing so it looks more natural. The people who actually have things to do will just get them done.
  - jay_kyburz 3y ago
    Yes, but those of us who want to use an AI are waiting for somebody to hook up a calculator on the back end. We would like the AI to test its theories before it send them back to us.
  - a_imho 3y ago
    Ok, so what do you suggest, people paying only casual attention should know this somehow?
    - chankstein38 3y ago
      Don't spout off about something you're not knowledgeable about? Not trying to be rude it just seems like if you don't know how they work you shouldn't be declaring something a failure because of a poorly conceived test.
      - recursive 3y ago
        It's presented as a chat bot. How much should know about chats before we can conclude that the responses are nonsense?
      - a_imho 3y ago
        I said not convinced, as in my user experince was/is still not conductive to the hype I perceive. The prompt says 'Ask anything', how should one know basic arithmetic is holding it wrong? Btw it also failed all queries in my native language after assuring (in English) it can parse them. Just nonsense/gibberish. Guess this is also a well known limitation? Perhaps my expectations were too high.
paxys 3y ago
While true, I think this is still valid criticism considering so many people are quick to jump on the "AGI" bandwagon when discussing the current generation of LLMs.
- cchance 3y ago
  No ones thinking a 7b-70b LLM is going to be an AGI lol, a 700b-1T llm likely gets pretty damn close especially with some of the newer attention concepts.
  - paxys 3y ago
    And yet GPT-4 with 1-2 trillion parameters still fails at the most basic math, sometimes even for tasks like adding up a set of ten numbers (hence the Wolfram comment). That's as clear evidence as any that intelligence is more than just language proficiency.
deleted 3y ago
[deleted]

MaximilianEmel 3y ago

mixtral-8x7b-instruct gets this correct: According to the order of operations (PEMDAS/BODMAS), you should perform multiplication and division before addition and subtraction. So for the expression 2 + 2 + 2 + 2 + 2 * 1, you should do the multiplication first: 2 * 1 = 2 Then, perform the addition: 2 + 2 + 2 + 2 + 2 = 10 So, the solution to the expression 2 + 2 + 2 + 2 + 2 * 1 is 10.

aimor 3y ago

I think it's understandable these models get tripped up by math problems. But if you want another way to ground lofty expectations, ask word riddles. --- "Answer this riddle: Which one of Santa's Reindeer can you see in outer space?" pplx-70b-chat: "You can see Rudolph, the red-nosed reindeer, in outer space." llama-2-70b-chat: "The answer to the riddle is "Rudolph." Rudolph is the only reindeer with a red nose, and red noses are often associated with astronauts and space travel. Therefore, Rudolph is the reindeer that can be seen in outer space." mixtral-8x7b-instruct: "The answer to the riddle is "Comet," which is one of Santa's reindeer. The reasoning behind this is that a comet is a celestial body visible from outer space, and it shares its name with one of Santa's reindeer, as mentioned in the Christmas song 'Rudolph the Red-Nosed Reindeer.'" --- "Answer this riddle: A doctor and a bus driver are both in love with the same woman, an attractive girl named Sarah. The bus driver had to go on a long bus trip that would last a week. Before he left, he gave Sarah seven apples. Why?" pplx-70b-chat: "The bus driver gave Sarah seven apples because he knew that he would be away for a week, and he wanted to make sure that she had something to remember him by and to take care of her while he was gone. Each apple represented a day of the week, and he hoped that Sarah would eat one apple each day and think of him while he was away." llama-2-70b-chat: "The bus driver gave Sarah seven apples because it was a way for him to show his love and affection for her while he was away. The number seven is often associated with perfection, completion, and abundance, so the bus driver may have chosen this number to represent the depth of his feelings for Sarah. Additionally, apples are a symbol of love and fertility in many cultures, so the bus driver may have chosen them as a way to express his desire for a long-term relationship with Sarah. The fact that the doctor is also in love with Sarah adds a layer of complexity to the riddle, as it creates a love triangle that is not uncommon in literature and real life." mixtral-8x7b-instruct: "Because an apple a day keeps the doctor away. He gave her one for each day he would be gone to ensure she wouldn't need to see the doctor during that time." --- "Answer this riddle: When is a doctor most annoyed?" pplx-70b-chat: "A doctor is most annoyed when they have to deal with a patient who is not taking their health seriously or is not following their recommendations. This can lead to frustration and annoyance on the part of the doctor, as they want to help the patient but are limited by the patient's own actions." llama-2-70b-chat: "The answer to the riddle is: When he's feeling under the weather. Explanation: The phrase "under the weather" means feeling unwell or sick, so a doctor would be most annoyed when he himself is feeling sick, as it would be a contradiction to his profession of helping others feel better." mixtral-8x7b-instruct: "A doctor might be most annoyed when their patient does not follow their medical advice, ignores their health concerns, or does not take their treatment seriously. Additionally, a doctor might be annoyed when they are unable to diagnose a patient's condition or when they are unable to help a patient in need. Overall, a doctor might be most annoyed when they are unable to provide the best possible care for their patients." pplx-7b-online: "A doctor is most annoyed when he is out of patients. This riddle plays on the double meaning of the word "patients," which can refer to both the people who need medical care and the source of a doctor's frustration."