2 ms·

Generating either a 0 or 1 in response to something? Maybe useful in some testing scenarios? Honestly don't know either. New lorem ipsum?

• visarga 3y ago
  You can use it as the assistant model to a large model, it's called speculative sampling. You generate text with the small model and validate with the large one, ensuring no deviation occurs. Speedup of 2.5x https://arxiv.org/abs/2302.01318 https://arxiv.org/abs/2302.01318 (DeepMind)
• MacsHeadroom 3y ago
  That would be 1bit quantization. In reality quantization under 8bits is done in smart ways which result in higher effective output quantization and lower effective memory size quantization. For example, bucketing identical groups of weights and then reusing one bucket for all the identical groups lowers the effective bit quantization at the memory level while retaining the uncompressed quantization quality. There is literature on effective quantization levels below 1 ("So called 0bit quantization). But even then the actual weights are typically 2-4 bits and there is just a lot of reuse of weights going on. Another neat trick is to line bins of weights up in order of lowest to highest weights and compute a function to produce a zero offset throughout the bins of weights; such that 0010 in one bin and and 0010 in another bin are unsampled to 16bit and then have different offsets added, maintaining uncompressed 16bit performance without the memory overhead. There are many more tricks like this and many still to be found!