mlgoatherder

5 ms·

1. 1.▲
   by mlgoatherder 4y ago

   Ah but LLM.int8 (eg. as in huggingface transformers) isn't actually int8, it's a mixed precision encoding scheme that is nominally eight bits per parameter. This means custom cuda kernels etc, these kernels could be improved but w
2. 2.▲
   by mlgoatherder 4y ago

   I've done some experiments here with Llama 13B, in my subjective experience the original fp16 model is significantly better (particularly on coding tasks). There are a bunch of synthetic benchmarks such a wikitext2 PPL and all the whiz
3. 3.▲
   by mlgoatherder 4y ago

   Great opportunity to pitch my recent work on a similar solution for gRPC: https://github.com/sillystack/api/blob/main/transport/transp... I'm not going to make any security claims as of now (it
4. 4.▲
   by mlgoatherder 4y ago

   I recently implemented a very similar service to service gRPC authentication mechanism for Go using EC25519 and the noise protocol framework. https://github.com/sillystack/api/blob/main/transport/tra