4 ms·

Code challenges like this are interesting - thank you for putting this together! I do have a question about the set-up, if that's alright. Netflix and others h

by s_Hogg 7y ago

Code challenges like this are interesting - thank you for putting this together! I do have a question about the set-up, if that's alright. Netflix and others have found that shared tasks can lead to great models, but not necessarily ones that are suited for use in a production environment. Have you put much thought into how best to set up a challenge such at this to make the obvious "ensemble everything" solution be less worthwhile? Similarly, have you put much thought into how to encourage the sharing of information between participants? Thanks again.

slewis 7y ago
There's been a lot of thought at Weights & Biases about the tradeoff between competition and collaboration. Competition certainly fosters activity, but it encourages behaviors like ensembling everything to eek out a few more hundredths of a percent. This benchmark isn't incentivized in any way other than "let's drive the field forward", so we may see less of that behavior. We've considered benchmarks that proceed in phases: a closed competitive phase for 3 months, then award a prize to the top result, and another prize for best user writeup. Follow that by a collaborative phase where it's more about sharing, teamwork etc. Rinse and repeat. The question of attribution is really interesting. Who made the largest contribution to the development of a model? It could have actually happened in a hallway conversation, or something equally as untrackable. We'd love to hear other peoples' thoughts on this. Stacey on our team has put a lot of thought into these topics and may have more to say here!
mloncode 7y ago
What is exciting about the tools we provided in this competition -- especially the weights and biases leaderboard is the level of transparency you get that you don't always see in a Kaggle competition (unless it is shared voluntarily). You can see: - All the system logging information including CPU/GPU utilization, with the runtime and type of GPU card used - Extensive logging of model training progression - All of the model artifacts and metadata - A link to the code on GitHub with the code that ran that data. - Anything emitted to stdout (for logging) - etc. This allows for extreme reproducibility and insight that is very helpful. With these tools, the community can see if an "ensemble everything: method is used and how long the model takes to train and what resources are consumed, etc. Good question!
cavisne 7y ago
The Netflix challenge was quite a while ago now. Since then Kaggle has added things like Kaggle kernels, where the models are trained on data they haven’t seen before (not just evaluated). Resources and training time are also kept even between submissions.
metaphdor 7y ago
Excellent questions, thank you! 1. We could log additional information about the model, such as inference time, number of parameters, memory usage, etc. and have the primary metric be overall efficiency (best NDCG with fewest parameters/fastest runtime/etc). 2. We're experimenting with different kinds of benchmarks, and I am most excited about explicitly collaborative ones. In these there is no contest/prize (hence no incentive to cheat/withhold information); only the shared goal of improving the model and our collective understanding of the problem. I hope we can incentivize information sharing by tracking and acknowledging individual contributions to the eventual best model in the benchmark. We could approximate individual contribution by seeing which scripts, code segments, workflows, architectural changes, writeups, or discussion comments other participants rate as the most helpful or choose to include in their experiments most often as the benchmark evolves. Of course this could only be an estimate--as Shawn says above, any idea could have "actually happened in a hallway conversation". Still, this is much easier to achieve in a logging/visualization platform like W&B than in the current paradigm of "read research papers, clone relevant repos, spend weeks trying to synthesize/reproduce their results, run your own experiments, write them up in a research paper, hope it gets accepted to a conference before other people publish the same idea, try to integrate your changes/publish your own repo, repeat"--and for hundreds of practitioners, ranging from brand new students to PhDs, working on related problems. This cycle is especially challenging for folks who are new to, working outside of, or trying to collaborate across the relatively few established/well-funded academic/industrial teams. Collaborative benchmarks can be especially impactful for social good projects, where the primary incentive is to figure out and broadly implement the best solution ASAP (e.g. climate change!), not to make money or argue over the credit attribution. So, my long-term goal is for as much sharing of information and collaboration from as many folks as possible--the more inclusive and transparent the field of deep learning research becomes, the safer and better its outcomes. Very open to ideas on how to help make this happen. ~Stacey, deep learning engineer at W&B

by s_Hogg 7y ago

slewis 7y ago

There's been a lot of thought at Weights & Biases about the tradeoff between competition and collaboration. Competition certainly fosters activity, but it encourages behaviors like ensembling everything to eek out a few more hundredths of a percent. This benchmark isn't incentivized in any way other than "let's drive the field forward", so we may see less of that behavior. We've considered benchmarks that proceed in phases: a closed competitive phase for 3 months, then award a prize to the top result, and another prize for best user writeup. Follow that by a collaborative phase where it's more about sharing, teamwork etc. Rinse and repeat. The question of attribution is really interesting. Who made the largest contribution to the development of a model? It could have actually happened in a hallway conversation, or something equally as untrackable. We'd love to hear other peoples' thoughts on this. Stacey on our team has put a lot of thought into these topics and may have more to say here!

mloncode 7y ago

What is exciting about the tools we provided in this competition -- especially the weights and biases leaderboard is the level of transparency you get that you don't always see in a Kaggle competition (unless it is shared voluntarily). You can see: - All the system logging information including CPU/GPU utilization, with the runtime and type of GPU card used - Extensive logging of model training progression - All of the model artifacts and metadata - A link to the code on GitHub with the code that ran that data. - Anything emitted to stdout (for logging) - etc. This allows for extreme reproducibility and insight that is very helpful. With these tools, the community can see if an "ensemble everything: method is used and how long the model takes to train and what resources are consumed, etc. Good question!

cavisne 7y ago

The Netflix challenge was quite a while ago now. Since then Kaggle has added things like Kaggle kernels, where the models are trained on data they haven’t seen before (not just evaluated). Resources and training time are also kept even between submissions.

metaphdor 7y ago

Excellent questions, thank you! 1. We could log additional information about the model, such as inference time, number of parameters, memory usage, etc. and have the primary metric be overall efficiency (best NDCG with fewest parameters/fastest runtime/etc). 2. We're experimenting with different kinds of benchmarks, and I am most excited about explicitly collaborative ones. In these there is no contest/prize (hence no incentive to cheat/withhold information); only the shared goal of improving the model and our collective understanding of the problem. I hope we can incentivize information sharing by tracking and acknowledging individual contributions to the eventual best model in the benchmark. We could approximate individual contribution by seeing which scripts, code segments, workflows, architectural changes, writeups, or discussion comments other participants rate as the most helpful or choose to include in their experiments most often as the benchmark evolves. Of course this could only be an estimate--as Shawn says above, any idea could have "actually happened in a hallway conversation". Still, this is much easier to achieve in a logging/visualization platform like W&B than in the current paradigm of "read research papers, clone relevant repos, spend weeks trying to synthesize/reproduce their results, run your own experiments, write them up in a research paper, hope it gets accepted to a conference before other people publish the same idea, try to integrate your changes/publish your own repo, repeat"--and for hundreds of practitioners, ranging from brand new students to PhDs, working on related problems. This cycle is especially challenging for folks who are new to, working outside of, or trying to collaborate across the relatively few established/well-funded academic/industrial teams. Collaborative benchmarks can be especially impactful for social good projects, where the primary incentive is to figure out and broadly implement the best solution ASAP (e.g. climate change!), not to make money or argue over the credit attribution. So, my long-term goal is for as much sharing of information and collaboration from as many folks as possible--the more inclusive and transparent the field of deep learning research becomes, the safer and better its outcomes. Very open to ideas on how to help make this happen. ~Stacey, deep learning engineer at W&B