3 ms·

Extraordinary claims require extraordinary evidence. That's not to say that a 70B model is necessary, but surely something larger than 3B is doable, especially

• GaggiX 3y ago
  For sure bigger models are needed to compete with transformer LLM, same thing for Mamba, I was just bothered by the distrust about something very reasonable like not being able to fully train a 70B model.
• edflsafoiewq 3y ago
  > results of the paper directly imply a significant reduction in memory requirements for training such a model Isn't memory use in training higher, since they maintain high precision latent weights in addition to the binarized weights used in the forward pass?
  • pclmulqdq 3y ago
    Most research universities have the resources to train a ~10B parameter model, at least.
  • danielmarkbruce 3y ago
    Yes. The optimizer is keeping a higher precision copy. It's likely slower and requires more memory than an equivalent full precision model when it comes to training. I'd also imagine it requires a multiple of epochs to get one epoch equivalent because the forward pass will need several goes to get the right choice between three states, rather than just moving a little bit in the right direction.