5 ms·

Quantized to 8 bits it needs only 178GB to run. With llama.cpp on a dual 7763 inference runs at 0.72 tokens per second (maybe about 32 words per minute), witho

by nullc 3y ago

Quantized to 8 bits it needs only 178GB to run. With llama.cpp on a dual 7763 inference runs at 0.72 tokens per second (maybe about 32 words per minute), without any tuning effort.

nullc 3y ago
[A chat between a curious user and an artificial intelligence assistant named Falcon. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: Please give a short explination of 'clickbait'.\nFalcon:] Certainly! Clickbait is a term used to describe online content that is designed to attract attention and encourage users to click on a link to a particular webpage. It often uses sensational or provocative headlines, images, or videos to entice users to click through to the content. The goal of clickbait is typically to generate more traffic to a website and increase advertising revenue. However, it can sometimes be misleading or deceptive, as the actual content may not live up to the expectations set by the clickbait headline or image.
nullc 3y ago
Seriously, my self reply showing output from Falcon 180B is flagged??
- KirillPanov 3y ago
  I think 100% LLM-generated comments are fairly unpopular. Suggestion: if you feel the need to share some LLM output with HN, prefix it with a human-written sentence or two explaining why it's worth reading.
jamesblonde 3y ago
Llama-2-70b with 16-bit precision = 2 bytes * 70 billion = 140 GB of memory Assuming zero copy, it should be loadable on around 170GB. Reference: https://www.hopsworks.ai/dictionary/llms-large-language-models https://www.hopsworks.ai/dictionary/llms-large-language-mode...
- stevenhuang 3y ago
  No one runs inference at full FP16 precision. 6 bit quant gives ~90% of original quality at more than half memory savings.
- nullc 3y ago
  Sigh. My comment gives a measurement of the memory usage of the model in question, Falcon 180B, (in mostly-q8 form, which is how someone would likely use it if not q4). Perhaps it would have been more obvious that I actually ran the model and measured it if people hadn't flagged the self-reply with the model output.

by nullc 3y ago

Quantized to 8 bits it needs only 178GB to run. With llama.cpp on a dual 7763 inference runs at 0.72 tokens per second (maybe about 32 words per minute), without any tuning effort.

nullc 3y ago

[A chat between a curious user and an artificial intelligence assistant named Falcon. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: Please give a short explination of 'clickbait'.\nFalcon:] Certainly! Clickbait is a term used to describe online content that is designed to attract attention and encourage users to click on a link to a particular webpage. It often uses sensational or provocative headlines, images, or videos to entice users to click through to the content. The goal of clickbait is typically to generate more traffic to a website and increase advertising revenue. However, it can sometimes be misleading or deceptive, as the actual content may not live up to the expectations set by the clickbait headline or image.

nullc 3y ago

Seriously, my self reply showing output from Falcon 180B is flagged??

KirillPanov 3y ago
I think 100% LLM-generated comments are fairly unpopular. Suggestion: if you feel the need to share some LLM output with HN, prefix it with a human-written sentence or two explaining why it's worth reading.

jamesblonde 3y ago

Llama-2-70b with 16-bit precision = 2 bytes * 70 billion = 140 GB of memory Assuming zero copy, it should be loadable on around 170GB. Reference: https://www.hopsworks.ai/dictionary/llms-large-language-models https://www.hopsworks.ai/dictionary/llms-large-language-mode...

stevenhuang 3y ago
No one runs inference at full FP16 precision. 6 bit quant gives ~90% of original quality at more than half memory savings.
nullc 3y ago
Sigh. My comment gives a measurement of the memory usage of the model in question, Falcon 180B, (in mostly-q8 form, which is how someone would likely use it if not q4). Perhaps it would have been more obvious that I actually ran the model and measured it if people hadn't flagged the self-reply with the model output.