4 ms·

The logbook is awesome: https://github.com/facebookresearch/metaseq/blob/main/projects/OPT/chronicles/OPT175B_Logbook.pdf?fbclid=IwAR1SYEN38IY9FvdnN2UQgCOLk0Urp

The logbook is awesome: https://github.com/facebookresearch/metaseq/blob/main/projects/OPT/chronicles/OPT175B_Logbook.pdf?fbclid=IwAR1SYEN38IY9FvdnN2UQgCOLk0UrpQRj8eR5F6gsesSErC6rcQnCTDMz8GI https://github.com/facebookresearch/metaseq/blob/main/projec... This is the true secret sauce -- all the tricks on how to get these things to train properly that aren't really published.

sp527 4y ago
This really does read like 'DevOps: Nightmare Edition' > CSP fat fingered and deleted our entire cluster when trying to replenish our buffer nodes Yikes
zubspace 4y ago
I have no knowledge of such things, but it seems they run Cuda jobs on about 150 nodes? But why do they have so many problems to keep this cluster stable? Network failures? Bad GPU's? Bad drivers? Bad software? Running fixmycloud and going after all those cryptic errors every day seems like a nightmare to me...
- semi-extrinsic 4y ago
  Seems kind of par for the course for an HPC cluster, no? It makes sense to think of these things like Formula 1 cars, they are trying to eke out the absolute maximum performance, and reliability suffers because of that. "Ordinary" cloud is more like a Toyota where you optimize for fuel economy and low maintenance.
amelius 4y ago
This all makes me wonder: how reproducible is the final output model?
- dekhn 4y ago
  the details of that specific model they ended up with? Irreproducible, unless the system was carefully designed and every detail required to do a fully reproducible computation was recorded and replayed. But they could easily produce a bunch of models that all sort of end up in roughly the same place and perform the same, ideally reducing the number of things they needed to change ad-hoc during the training.
- screye 4y ago
  Not this one, but Google's PaLM (which is 4x OPT3) trains semi-deterministically. These kinds of large transformers can be relatively reproduceable in results and benchmarks. However, making them converge to the exact same parameter set might not be a reasonable expectation.
- ur-whale 4y ago
  > how reproducible is the final output model? Not much, but it also depends on what you mean by "reproducible". Do you expect a similar internal representation (weights) or a similar behavior?
dang 4y ago
Related: 100 Pages of raw notes released with the language model OPT-175 - https://news.ycombinator.com/item?id=31260665 https://news.ycombinator.com/item?id=31260665 - May 2022 (26 comments)
gxqoz 4y ago
Any particular highlights from this?
- fny 4y ago
  Honestly, no.
axg11 4y ago
It really is great that Meta released the notes/logbook. Credit where credit is due. Very few other academic or industry labs release materials like this, especially when the reality is so messy. Some interesting takeaways: - Meta aren't using any software for scientific logbooks, just prepending a document - So many hardware/cluster issues. - Hot-swapping algorithms is common and likely underreported (in this case activation functions and optimization method) - A well resourced team didn't solve enough issues to fully utilize compute resources until >50% of the total time into the project
- domenicrosati 4y ago
  I wonder what software would be good for a logbook like this... I just use google docs for these kinds of things. Sure wandb and jupyter notebooks are good but they are not so good for notes and ideas and documentation
  - deleted 4y ago
    [deleted]
  - asadlionpk 4y ago
    LogSeq? I use it for this kinda work...
  - Gigachad 4y ago
    Sometimes the generic solution is just the best. No one requires special training on Google Docs and it just comes with handy features like version control and live updates.
  - semi-extrinsic 4y ago
    We tried jupyter notebooks for this kind of thing once, and after the equivalent of ~50 pages it becomes unusably slow in my experience.
  - EricLeer 4y ago
    Personally I use notion for this. Not dedicated software, but the extra options in formatting/structuring and linking can make a document like this a lot more readible.

sp527 4y ago

This really does read like 'DevOps: Nightmare Edition' > CSP fat fingered and deleted our entire cluster when trying to replenish our buffer nodes Yikes

zubspace 4y ago

I have no knowledge of such things, but it seems they run Cuda jobs on about 150 nodes? But why do they have so many problems to keep this cluster stable? Network failures? Bad GPU's? Bad drivers? Bad software? Running fixmycloud and going after all those cryptic errors every day seems like a nightmare to me...

semi-extrinsic 4y ago
Seems kind of par for the course for an HPC cluster, no? It makes sense to think of these things like Formula 1 cars, they are trying to eke out the absolute maximum performance, and reliability suffers because of that. "Ordinary" cloud is more like a Toyota where you optimize for fuel economy and low maintenance.

amelius 4y ago

This all makes me wonder: how reproducible is the final output model?

dekhn 4y ago
the details of that specific model they ended up with? Irreproducible, unless the system was carefully designed and every detail required to do a fully reproducible computation was recorded and replayed. But they could easily produce a bunch of models that all sort of end up in roughly the same place and perform the same, ideally reducing the number of things they needed to change ad-hoc during the training.
screye 4y ago
Not this one, but Google's PaLM (which is 4x OPT3) trains semi-deterministically. These kinds of large transformers can be relatively reproduceable in results and benchmarks. However, making them converge to the exact same parameter set might not be a reasonable expectation.
ur-whale 4y ago
> how reproducible is the final output model? Not much, but it also depends on what you mean by "reproducible". Do you expect a similar internal representation (weights) or a similar behavior?

dang 4y ago

Related: 100 Pages of raw notes released with the language model OPT-175 - https://news.ycombinator.com/item?id=31260665 https://news.ycombinator.com/item?id=31260665 - May 2022 (26 comments)

gxqoz 4y ago

Any particular highlights from this?

fny 4y ago
Honestly, no.

axg11 4y ago

It really is great that Meta released the notes/logbook. Credit where credit is due. Very few other academic or industry labs release materials like this, especially when the reality is so messy. Some interesting takeaways: - Meta aren't using any software for scientific logbooks, just prepending a document - So many hardware/cluster issues. - Hot-swapping algorithms is common and likely underreported (in this case activation functions and optimization method) - A well resourced team didn't solve enough issues to fully utilize compute resources until >50% of the total time into the project

domenicrosati 4y ago
I wonder what software would be good for a logbook like this... I just use google docs for these kinds of things. Sure wandb and jupyter notebooks are good but they are not so good for notes and ideas and documentation
- deleted 4y ago
  [deleted]
- asadlionpk 4y ago
  LogSeq? I use it for this kinda work...
- Gigachad 4y ago
  Sometimes the generic solution is just the best. No one requires special training on Google Docs and it just comes with handy features like version control and live updates.
- semi-extrinsic 4y ago
  We tried jupyter notebooks for this kind of thing once, and after the equivalent of ~50 pages it becomes unusably slow in my experience.
- EricLeer 4y ago
  Personally I use notion for this. Not dedicated software, but the extra options in formatting/structuring and linking can make a document like this a lot more readible.