4 ms·

If I want to put a Gemma model in a minimalist command line interface, build it to a standalone exe file that runs offline, what is the size of my final executa

by a1o 3y ago

If I want to put a Gemma model in a minimalist command line interface, build it to a standalone exe file that runs offline, what is the size of my final executable? I am interested in how small can the size of something like this be and it still be functional.

replete 3y ago
I used gemm:2b with ollama last night and the model was around 1.3gb IIRC
samus 3y ago
Depends how much you quantize the model. For most general-purpose LLMs, the model completely dwarfs the size of the binary code.
coder543 3y ago
https://ollama.com/library/gemma/tags https://ollama.com/library/gemma/tags You can see the various quantizations here, both for the 2B model and the 7B model. The smallest you can go is the q2_K quantization of the 2B model, which is 1.3GB, but I wouldn't really call that "functional". The q4_0 quantization is 1.7GB, and that would probably be functional. The size of anything but the model is going to be rounding error compared to how large the models are, in this context.
- sorenjan 3y ago
  What's the use case of models this small? Can you use the "knowledge" encoded in them and ask them questions and get relevant answers, or are they used as text processors to summarize documents etc?
  - trisfromgoogle 3y ago
    Gemma 2B generation quality is excellent in my own very-biased opinion. I asked it to write a response to your comment: -- Large language models (LLMs) have achieved significant progress in recent years, with models like GPT-3 and LaMDA demonstrating remarkable abilities in various tasks such as language generation, translation, and question answering. However, 2b parameter models are a much smaller and simpler type of LLM compared to GPT-3. While they are still capable of impressive performance, they have a limited capacity for knowledge representation and reasoning. Despite their size, 2b parameter models can be useful in certain scenarios where the specific knowledge encoded in the model is relevant to the task at hand. For example: - Question answering: 2b parameter models can be used to answer questions by leveraging their ability to generate text that is similar to the question. - Text summarization: 2b parameter models can be used to generate concise summaries of documents by extracting the most important information. - Code generation: While not as common, 2b parameter models can be used to generate code snippets based on the knowledge they have learned. Overall, 2b parameter models are a valuable tool for tasks that require specific knowledge or reasoning capabilities. However, for tasks that involve general language understanding and information retrieval, larger LLMs like GPT-3 may be more suitable. -- Generated in under 1s from query to full response on together.ai https://api.together.xyz/playground/chat/google/gemma-2b-it https://api.together.xyz/playground/chat/google/gemma-2b-it
brucethemoose2 3y ago
The code is a basically irrelevant fraction of the model weights. The raw FP16 is like 17GB. In practice your priority would be fancy quantization, and just any library that compiles down to an executable (like this, MLC-LLM or llama.cpp)
- a1o 3y ago
  17GB looks like a lot. Thanks, I will wait until people figure how to make these smaller before trying to use to make something standalone.
  - wg0 3y ago
    These won't be smaller I guess. Given we keep the number of parameters same. Pre LLM era (let's say 2020), the hardware used to look decently powerful for most use cases (disks in hundreds of GBs, dozen or two of RAM and quad or hex core processors) but with the advent of LLMs, even disk drives start to look pretty small let alone compute and memory.
    - brucethemoose2 3y ago
      And cache! The talk of AI hardware is now "how do we fit these darn things inside SRAM?"
  - sillysaurusx 3y ago
    The average PS5 game seems to be around 45GB. Cyberpunk was 250GB. Distributing 17GB isn’t a big deal if you shove it into Cloudflare R2.
  - brucethemoose2 3y ago
    In theory quantized weights of smaller models are under a gigabyte. If you are looking for megabytes, yeah, those "chat" llms are pretty unusable at that size.
  - swatcoder 3y ago
    It's always going to be a huge quantity of data. Even as efficiency improves, storage and bandwidth are so cheap now that the incentive will be to convert that efficiency towards performance (models with more parameters, ensembles of models, etc) rather than chasing some micro-model that doesn't do as well. It might not always be 17GB, but don't expect some lesser order of magnitude for anything competitive. As maturity arrives, we'll likely see a handful of competing local models shipped as part of the OS or as redistributable third-party bundles (a la the .NET or Java runtimes) so that individual applications don't all need to be massive. You'll either need to wait for that or bite the bullet and make something chonky. It's never going to get that small.
superkuh 3y ago
*EDIT*: Nevermind, llamafile hasn't been updated in a full month and gemma support was only added to llama.cpp on the 21st of this month. Disregard this post for now and come back when mozilla updates llamafile. --- llama.cpp has integrated gemma support. So you can use llamafile for this. It is a standalone executable that is portable across most popular OSes. https://github.com/Mozilla-Ocho/llamafile/releases https://github.com/Mozilla-Ocho/llamafile/releases So, download the executable from the releases page under assets. You want either just main and server and llava. Don't get the huge ones with the model inlined in the file. The executable is about 30MB in size, https://github.com/Mozilla-Ocho/llamafile/releases/download/0.6.2/llamafile-0.6.2 https://github.com/Mozilla-Ocho/llamafile/releases/download/...

by a1o 3y ago

replete 3y ago

I used gemm:2b with ollama last night and the model was around 1.3gb IIRC

samus 3y ago

Depends how much you quantize the model. For most general-purpose LLMs, the model completely dwarfs the size of the binary code.

coder543 3y ago

https://ollama.com/library/gemma/tags https://ollama.com/library/gemma/tags You can see the various quantizations here, both for the 2B model and the 7B model. The smallest you can go is the q2_K quantization of the 2B model, which is 1.3GB, but I wouldn't really call that "functional". The q4_0 quantization is 1.7GB, and that would probably be functional. The size of anything but the model is going to be rounding error compared to how large the models are, in this context.

sorenjan 3y ago
What's the use case of models this small? Can you use the "knowledge" encoded in them and ask them questions and get relevant answers, or are they used as text processors to summarize documents etc?
- trisfromgoogle 3y ago
  Gemma 2B generation quality is excellent in my own very-biased opinion. I asked it to write a response to your comment: -- Large language models (LLMs) have achieved significant progress in recent years, with models like GPT-3 and LaMDA demonstrating remarkable abilities in various tasks such as language generation, translation, and question answering. However, 2b parameter models are a much smaller and simpler type of LLM compared to GPT-3. While they are still capable of impressive performance, they have a limited capacity for knowledge representation and reasoning. Despite their size, 2b parameter models can be useful in certain scenarios where the specific knowledge encoded in the model is relevant to the task at hand. For example: - Question answering: 2b parameter models can be used to answer questions by leveraging their ability to generate text that is similar to the question. - Text summarization: 2b parameter models can be used to generate concise summaries of documents by extracting the most important information. - Code generation: While not as common, 2b parameter models can be used to generate code snippets based on the knowledge they have learned. Overall, 2b parameter models are a valuable tool for tasks that require specific knowledge or reasoning capabilities. However, for tasks that involve general language understanding and information retrieval, larger LLMs like GPT-3 may be more suitable. -- Generated in under 1s from query to full response on together.ai https://api.together.xyz/playground/chat/google/gemma-2b-it https://api.together.xyz/playground/chat/google/gemma-2b-it

brucethemoose2 3y ago

The code is a basically irrelevant fraction of the model weights. The raw FP16 is like 17GB. In practice your priority would be fancy quantization, and just any library that compiles down to an executable (like this, MLC-LLM or llama.cpp)

a1o 3y ago
17GB looks like a lot. Thanks, I will wait until people figure how to make these smaller before trying to use to make something standalone.
- wg0 3y ago
  These won't be smaller I guess. Given we keep the number of parameters same. Pre LLM era (let's say 2020), the hardware used to look decently powerful for most use cases (disks in hundreds of GBs, dozen or two of RAM and quad or hex core processors) but with the advent of LLMs, even disk drives start to look pretty small let alone compute and memory.
  - brucethemoose2 3y ago
    And cache! The talk of AI hardware is now "how do we fit these darn things inside SRAM?"
- sillysaurusx 3y ago
  The average PS5 game seems to be around 45GB. Cyberpunk was 250GB. Distributing 17GB isn’t a big deal if you shove it into Cloudflare R2.
- brucethemoose2 3y ago
  In theory quantized weights of smaller models are under a gigabyte. If you are looking for megabytes, yeah, those "chat" llms are pretty unusable at that size.
- swatcoder 3y ago
  It's always going to be a huge quantity of data. Even as efficiency improves, storage and bandwidth are so cheap now that the incentive will be to convert that efficiency towards performance (models with more parameters, ensembles of models, etc) rather than chasing some micro-model that doesn't do as well. It might not always be 17GB, but don't expect some lesser order of magnitude for anything competitive. As maturity arrives, we'll likely see a handful of competing local models shipped as part of the OS or as redistributable third-party bundles (a la the .NET or Java runtimes) so that individual applications don't all need to be massive. You'll either need to wait for that or bite the bullet and make something chonky. It's never going to get that small.

superkuh 3y ago

*EDIT*: Nevermind, llamafile hasn't been updated in a full month and gemma support was only added to llama.cpp on the 21st of this month. Disregard this post for now and come back when mozilla updates llamafile. --- llama.cpp has integrated gemma support. So you can use llamafile for this. It is a standalone executable that is portable across most popular OSes. https://github.com/Mozilla-Ocho/llamafile/releases https://github.com/Mozilla-Ocho/llamafile/releases So, download the executable from the releases page under assets. You want either just main and server and llava. Don't get the huge ones with the model inlined in the file. The executable is about 30MB in size, https://github.com/Mozilla-Ocho/llamafile/releases/download/0.6.2/llamafile-0.6.2 https://github.com/Mozilla-Ocho/llamafile/releases/download/...