4 ms·

Update - I'm still cautious about this paper, but I had the table numbers inverted in my head while thinking about it. The paper shows better perplexity results

Update - I'm still cautious about this paper, but I had the table numbers inverted in my head while thinking about it. The paper shows better perplexity results than competing models at larger parameter sizes, so I was wrong.

pclmulqdq 3y ago
I was pretty unhappy and suspicious for the same reason. Not reporting perplexity for a 70B network while reporting its efficiency means that someone did something and the result wasn't good enough to put in the paper.
- GaggiX 3y ago
  According to the author, the 70B model is not fully trained.
  - pclmulqdq 3y ago
    "Is not fully trained" can also mean "we did not figure out how to reach an acceptable loss" or "training was unstable," both of which are common for ML systems.
    - GaggiX 3y ago
      It probably means that the model is not fully trained, because it is very expensive to train a 70B model, not even Mamba or RWKV have a model that comes close to that size, the leeriness is just kinda silly honestly.
      - bick_nyers 3y ago
        Extraordinary claims require extraordinary evidence. That's not to say that a 70B model is necessary, but surely something larger than 3B is doable, especially given that the results of the paper directly imply a significant reduction in memory requirements for training such a model.
        GaggiX 3y ago
        For sure bigger models are needed to compete with transformer LLM, same thing for Mamba, I was just bothered by the distrust about something very reasonable like not being able to fully train a 70B model.
        edflsafoiewq 3y ago
        > results of the paper directly imply a significant reduction in memory requirements for training such a model Isn't memory use in training higher, since they maintain high precision latent weights in addition to the binarized weights used in the forward pass?
        pclmulqdq 3y ago
        Most research universities have the resources to train a ~10B parameter model, at least.
        danielmarkbruce 3y ago
        Yes. The optimizer is keeping a higher precision copy. It's likely slower and requires more memory than an equivalent full precision model when it comes to training. I'd also imagine it requires a multiple of epochs to get one epoch equivalent because the forward pass will need several goes to get the right choice between three states, rather than just moving a little bit in the right direction.
- kristjansson 3y ago
  One can forgive the lack of quality results for the 70B model, but apparently they trained 7B and 13B versions of their model, and don't report those either.

pclmulqdq 3y ago

I was pretty unhappy and suspicious for the same reason. Not reporting perplexity for a 70B network while reporting its efficiency means that someone did something and the result wasn't good enough to put in the paper.

GaggiX 3y ago
According to the author, the 70B model is not fully trained.
- pclmulqdq 3y ago
  "Is not fully trained" can also mean "we did not figure out how to reach an acceptable loss" or "training was unstable," both of which are common for ML systems.
  - GaggiX 3y ago
    It probably means that the model is not fully trained, because it is very expensive to train a 70B model, not even Mamba or RWKV have a model that comes close to that size, the leeriness is just kinda silly honestly.
    - bick_nyers 3y ago
      Extraordinary claims require extraordinary evidence. That's not to say that a 70B model is necessary, but surely something larger than 3B is doable, especially given that the results of the paper directly imply a significant reduction in memory requirements for training such a model.
      - GaggiX 3y ago
        For sure bigger models are needed to compete with transformer LLM, same thing for Mamba, I was just bothered by the distrust about something very reasonable like not being able to fully train a 70B model.
      - edflsafoiewq 3y ago
        > results of the paper directly imply a significant reduction in memory requirements for training such a model Isn't memory use in training higher, since they maintain high precision latent weights in addition to the binarized weights used in the forward pass?
        pclmulqdq 3y ago
        Most research universities have the resources to train a ~10B parameter model, at least.
        danielmarkbruce 3y ago
        Yes. The optimizer is keeping a higher precision copy. It's likely slower and requires more memory than an equivalent full precision model when it comes to training. I'd also imagine it requires a multiple of epochs to get one epoch equivalent because the forward pass will need several goes to get the right choice between three states, rather than just moving a little bit in the right direction.
kristjansson 3y ago
One can forgive the lack of quality results for the 70B model, but apparently they trained 7B and 13B versions of their model, and don't report those either.