brucethemoose2

8 ms·

61.
by brucethemoose2 3y ago
And cache! The talk of AI hardware is now "how do we fit these darn things inside SRAM?"
62.
by brucethemoose2 3y ago
In theory quantized weights of smaller models are under a gigabyte. If you are looking for megabytes, yeah, those "chat" llms are pretty unusable at that size.
63.
by brucethemoose2 3y ago
Yeah, especially with how much Google is hyping it. It could have been long context? Or a little bigger, to fill the relative gap in the 13B-30B area? Even if the model itself was mediocre (which you can't know until after training), i
64.
by brucethemoose2 3y ago
This is not necessarily a production backend, as it mentions in the readme. There are some very interesting efforts in JAX/TPU land like https://github.com/erfanzar/EasyDeL
65.
by brucethemoose2 3y ago
If you are implying we are at the mercy of a few CUDA experts because we can't efficiently write it ourselves, you would be correct. Triton does seem way more approachable as something I'd like to pick up.
66.
by brucethemoose2 3y ago
The code is a basically irrelevant fraction of the model weights. The raw FP16 is like 17GB. In practice your priority would be fancy quantization, and just any library that compiles down to an executable (like this, MLC-LLM or llama.cpp)
67.
by brucethemoose2 3y ago
...Also, we have eval'd Gemma 7B internally in a deterministic, zero temperature test, and its error rate is like double Mistral Instruct 0.2. Well below most other 7Bs. Was not very impressed with the chat either. So maybe this is nea
68.
by brucethemoose2 3y ago
Not to be confused with llama.cpp and the GGML library, which is a seperate project (and almost immediately worked with Gemma).
69.
by brucethemoose2 3y ago
> "Basically nobody writes CUDA," Quite a thing to say as head of an AI hardware startup. Most of the machine learning libraries we use/test in production (and that I use personally on my desktop) have hand written, extens
70.
by brucethemoose2 3y ago
Being accesible is huge. I still see post of people running ollama on H100s or whatever, and that's just because its so easy to set up.
71.
by brucethemoose2 3y ago
Yeah I will have to test it out, though TBH I am more inclined to run models locally. As I mentioned, being such an extensive continuation train can (sometimes) totally change the capabilities of a model.
72.
by brucethemoose2 3y ago
Yeah, and I'd go so far as to call it a continued pretrain with that many tokens. More like a whole new model than a traditional finetune.
73.
by brucethemoose2 3y ago
Your options are: - Hybrid offloading with llama.cpp, but with slow inference. - Squeezing it in with extreme quantization (exllamav2 ~2.6bpw, or llama.cpp IQ3XS), but reduced quality and a relatively short context. 30B-34B is more of a s
74.
by brucethemoose2 3y ago
I have not had luck with codellama 70B models for coding, nor have I had it with the mistral leak. If I were Phind, I'd be looking at Deepseek 33B instead. While obviously dumber for anything else, it feels much better at coding. Its j
75.
by brucethemoose2 3y ago
I agree... Except for the leaderboard. Its all but useless, not just because of the data contamination/cheating but because the benchmarks themselves are flawed. They are full of ambiguity/errors, and they dont even use instruct f
76.
by brucethemoose2 3y ago
I think so. But Intel pretty much bet everything AI on Falcon Shores (2025). If its crumby or late, I think it will be difficult for them to catch up. They'll have inference hardware in laptops, sure, but still...
77.
by brucethemoose2 3y ago
That is awesome! I hope y'all consider longer context models as well. Also, are ya'll looking alternative architectures like Mamba? Being "first" with a large Mamba model would cement your architectural choices/fram
78.
by brucethemoose2 3y ago
There are some exceptions, like Mistral 0.1 (which is technically 32K according to the config but practically 8K because the sliding window is awful) and InternLM (which (at least initially) used auto rope scaling to extend the context as p
79.
by brucethemoose2 3y ago
You should be looking at Deepseek's coding models, and finetunes of those. I run 33B on my desktop, and find it to be sufficient for many tasks.
80.
by brucethemoose2 3y ago
> Also the inference doesn't always stop generating resulting in garbage being added to the response. That sounds like a chat format misconfiguration. This could partially be Google's fault, as they used yet another novel pro
81.
by brucethemoose2 3y ago
Alignment is all but a non issue with open weight base model releases, as they can be finetuned to "de align" them if prompt engineering is not enough.
82.
by brucethemoose2 3y ago
This is a good point actually, and an underappreciated fact. I think so many people (including me) effectively ignored Mistral 0.1's sliding window that few realized 0.2 instruct is native 32K.
83.
by brucethemoose2 3y ago
Miqu was (allegedly) an internal continued pretrain Mistral did as a test, that was leaked as a GGUF. Maybe its just semantics, it is technically a finetune... But to me theres a big difference between expensive "continuation training&
84.
by brucethemoose2 3y ago
TBH the community has largely outrun Mistral's own finetuning. The 7B model in particular is such a popular target because its so practical to train.
85.
by brucethemoose2 3y ago
Will there be "extended context" releases like 01.ai did for Yi? Also, is the model GQA?
86.
by brucethemoose2 3y ago
Only 8K context as well, like Mistral. Also, as always, take these benchmarks with a huge grain of salt. Even base model releases are frequently (seemingly) contaminated these days.
87.
by brucethemoose2 3y ago
Are you going to be training constantly? TBH renting a big GPU (A100 or better) and getting the run over with quick is usually best. VRAM is everything.
88.
by brucethemoose2 3y ago
Yeah this was my impression. I am but an end user, but I noticed that disk IO for a certain app was glacial compared to a local test deployment, and I chalked it up to networking/VM overhead
89.
by brucethemoose2 3y ago
But people need to use it. Critical internet/app browsing should be taught in school, like critical reading. I feel lucky to have been a nerd in the 2000s where people picked up this skill, but honestly I have no idea how kids, older f
90.
by brucethemoose2 3y ago
And yet Discord has become a de-facto forum replacement for niche topics/communities. ...Which is awful. As is often said here, Discord is where information goes to die.

More ›