3 ms·
It really is great that Meta released the notes/logbook. Credit where credit is due. Very few other academic or industry labs release materials like this, espec
by axg11 4y ago
It really is great that Meta released the notes/logbook. Credit where credit is due. Very few other academic or industry labs release materials like this, especially when the reality is so messy.
Some interesting takeaways:
- Meta aren't using any software for scientific logbooks, just prepending a document
- So many hardware/cluster issues.
- Hot-swapping algorithms is common and likely underreported (in this case activation functions and optimization method)
- A well resourced team didn't solve enough issues to fully utilize compute resources until >50% of the total time into the project
- domenicrosati 4y agoI wonder what software would be good for a logbook like this... I just use google docs for these kinds of things. Sure wandb and jupyter notebooks are good but they are not so good for notes and ideas and documentation
- deleted 4y ago[deleted]
- asadlionpk 4y agoLogSeq? I use it for this kinda work...
- Gigachad 4y agoSometimes the generic solution is just the best. No one requires special training on Google Docs and it just comes with handy features like version control and live updates.
- semi-extrinsic 4y agoWe tried jupyter notebooks for this kind of thing once, and after the equivalent of ~50 pages it becomes unusably slow in my experience.
- EricLeer 4y agoPersonally I use notion for this. Not dedicated software, but the extra options in formatting/structuring and linking can make a document like this a lot more readible.
- sanxiyn 4y agoIf you are interested in hot-swapping, I highly recommend OpenAI Five paper: https://arxiv.org/abs/1912.06680 https://arxiv.org/abs/1912.06680. It includes 10 months log of how they adapted the model to releases of Dota 2 version 7.19, 7.20, and 7.21. I agree hot-swapping is underreported, but there are some good existing reports. In fact, OpenAI Five paper today is probably more valuable for its details on hot-swapping than its details on main model which used LSTM and not transformer.
- hoseja 4y agoWeird they felt the need to update the version since they used a very limited subset of Dota anyway.