austinvhuang

5 ms·

1. 31.▲
   by austinvhuang 3y ago

   A nice side effect of implementing cpu simd is you just need enough regular RAM, which tends to be far less scarce than VRAM. Nonetheless, I get your point that more aggressive quantization is valuable + will share with the modeling team.
2. 32.▲
   by austinvhuang 3y ago

   Gemma support has been added to llama.cpp, in fact it was added almost immediately after the release: https://twitter.com/ggerganov/status/1760293079313973408 However, be aware that there were some quality issues
3. 33.▲
   by austinvhuang 3y ago

   any particular hardware folks are most interested in?
4. 34.▲
   by austinvhuang 3y ago

   lol
5. 35.▲
   by austinvhuang 3y ago

   I think it's mostly an organic process arising from the ecosystem. My personal way of understanding it is this - the original sin of model weight format complexity is that NNs are both data and computation. Representing the computation
6. 36.▲
   by austinvhuang 3y ago

   There's a custom 8-bit quantization (SFP), it's what we recommend. At 16 bit, we do bfloat16 instead of fp16 thanks to https://github.com/google/highway , even on CPU. Other quants - stay tuned. python wrapper
7. 37.▲
   by austinvhuang 3y ago

   Don't be discouraged, you don't have to follow everything. In fact it's probably better to dive deep into one hobby project like you're doing than constantly context switch with every little news item that comes up. Whil
8. 38.▲
   by austinvhuang 3y ago

   Thanks so much! Everyone working on this self-selected into contributing, so I think of it less as my team than ... a team? Specifically want to call out: Jan Wassenberg (author of https://github.com/google/highway ) an
9. 39.▲
   by austinvhuang 3y ago

   llama.cpp is great, if it fit your needs you can use it. I think at this point llama.cpp is effectively a platform that's hardened for production. In its current form, I think of gemma.cpp is more of a direct model implementation (some
10. 40.▲
    by austinvhuang 3y ago

    gguf files are out there, so anyone should be able to do this! are people looking for an "official" version? ps i'm a fan of cosmopolitan as well.
11. 41.▲
    by austinvhuang 3y ago

    Hi, one of the authors austin here. Happy to answer any questions the best I can. To get a few common questions out of the way: - This is separate / independent of llama.cpp / ggml. I'm a big fan of that project and it was an
12. 42.▲
    by austinvhuang 3y ago

    You can download the model checkpoints from kaggle https://www.kaggle.com/models/google/gemma and huggingface https://huggingface.co/blog/gemma Besides the python implementations, we also imp
13. 43.▲
    by austinvhuang 3y ago

    Yes models can be downloaded locally. In addition to the python NN frameworks and ggml as options, we also implemented a standalone C++ implementation that you can run locally at https://github.com/google/gemma.cpp
14. 44.▲
    by austinvhuang 5y ago

    I'm working on local-first OSS for this sort of second brain thing called OpenMemex. The data store is sqlite (wasm/rust frontend) with programmable integrations in mind: https://github.com/austinvhuang/openme
15. 45.▲
    by austinvhuang 5y ago

    I recently released this which is open source and based on sqlite https://github.com/austinvhuang/openmemex
16. 46.▲
    by austinvhuang 5y ago

    Not the OP, but I recently published this which others may find interesting https://github.com/austinvhuang/openmemex