4 ms·
Don’t sleep on NVIDIA and Nemotron. It’s not completely open source, but they actually release their pretraining and post-training datasets with some redaction
by dannyw 2mo ago
Don’t sleep on NVIDIA and Nemotron.
It’s not completely open source, but they actually release their pretraining and post-training datasets with some redactions for (cough) pirated content.
They also have very good code and playbooks for actually doing a fine-tune, CPT, etc.
Even if you’re not tuning a Nemotron model, its mixes are very excellent for your replay data slice; or general experiments. Way better curation and quality than Dolma, etc; or other large huggingface data mixes I tested.
- nickludlam 2mo agoYes, I second Nemotron. I'm using Ultra remotely and Super locally, and I find them very useful for RAG-like problems. I wouldn't really use them for coding.