3 ms·

However big it may be, it still hallucinates very, very badly. I just asked it an economics question and asked it to cite its sources. All the links provided

However big it may be, it still hallucinates very, very badly. I just asked it an economics question and asked it to cite its sources. All the links provided as sources were complete BS. Color me unimpressed.

mike_hearn 3y ago
It's intended for SQL generation and similar with cheap fine tuning and inference, not answering general knowledge questions. Their blog post is pretty clear about that. If you just want a chatbot this isn't the model for you. If you want to let non-SQL trained people ask questions of your data, it might be really useful.
- mritchie712 3y ago
  It's worse at SQL generation than llama3 according to their own post. https://www.snowflake.com/blog/arctic-open-efficient-foundation-language-models-snowflake/#:~:text=on%20enterprise%20intelligence.-,Table%203,-.%20Full%20Metrics%20Table https://www.snowflake.com/blog/arctic-open-efficient-foundat...
  - CharlesW 3y ago
    To be fair, that's comparing their 17B model with the 70B Llama 3 model.
    - ru552 3y ago
      To stay fair, their "17B" model sits at 964GB on your disk and the 70B Llama 3 model sits at 141GB. unquantized GB numbers for both
      - CharlesW 3y ago
        Sorry, it sounds like you know a lot more than I do about this, and I'd appreciate it if you'd connect the dots. Is your comment a dig at either Snowflake or Llama? Where are you finding the unquantized size of Llama 3 70B? Isn't it extremely rare to do inference with large unquantized models?
      - fsiefken 3y ago
        to stay fairer, the required extra disk space for snowflake-arctic is cheaper then the required extra ram memory for llama3
        Manabu-eo 3y ago
        For decent performance, you need to keep all the parameters on memory for both. Well, with a raid-0 of two PCIe 5 SSDs (or 4 PCIe 4) you might get 1 t/s loading experts from disk on snowflake-artic... but that is slooow.
sp332 3y ago
It's a statistical model of language. If it wasn't trained on text that says "I don't know that", then it's not going to produce that text. You need to use tools that can look at the logits produced and see if you're getting a confident answer or noise.
cqqxo4zV46cp 3y ago
Please read the post before commenting.
claar 3y ago
To me, your complaint is equivalent to "I tried your new screwdriver and it couldn't even hammer in this simple nail!" You're using it wrong. Expecting an auto-complete engine to not make up words is an exercise in frustration.

mike_hearn 3y ago

It's intended for SQL generation and similar with cheap fine tuning and inference, not answering general knowledge questions. Their blog post is pretty clear about that. If you just want a chatbot this isn't the model for you. If you want to let non-SQL trained people ask questions of your data, it might be really useful.

mritchie712 3y ago
It's worse at SQL generation than llama3 according to their own post. https://www.snowflake.com/blog/arctic-open-efficient-foundation-language-models-snowflake/#:~:text=on%20enterprise%20intelligence.-,Table%203,-.%20Full%20Metrics%20Table https://www.snowflake.com/blog/arctic-open-efficient-foundat...
- CharlesW 3y ago
  To be fair, that's comparing their 17B model with the 70B Llama 3 model.
  - ru552 3y ago
    To stay fair, their "17B" model sits at 964GB on your disk and the 70B Llama 3 model sits at 141GB. unquantized GB numbers for both
    - CharlesW 3y ago
      Sorry, it sounds like you know a lot more than I do about this, and I'd appreciate it if you'd connect the dots. Is your comment a dig at either Snowflake or Llama? Where are you finding the unquantized size of Llama 3 70B? Isn't it extremely rare to do inference with large unquantized models?
    - fsiefken 3y ago
      to stay fairer, the required extra disk space for snowflake-arctic is cheaper then the required extra ram memory for llama3
      - Manabu-eo 3y ago
        For decent performance, you need to keep all the parameters on memory for both. Well, with a raid-0 of two PCIe 5 SSDs (or 4 PCIe 4) you might get 1 t/s loading experts from disk on snowflake-artic... but that is slooow.

sp332 3y ago

It's a statistical model of language. If it wasn't trained on text that says "I don't know that", then it's not going to produce that text. You need to use tools that can look at the logits produced and see if you're getting a confident answer or noise.

cqqxo4zV46cp 3y ago

Please read the post before commenting.

claar 3y ago

To me, your complaint is equivalent to "I tried your new screwdriver and it couldn't even hammer in this simple nail!" You're using it wrong. Expecting an auto-complete engine to not make up words is an exercise in frustration.