5 ms·

Why aren't these data sets editable instead of static? Treat them like a collaborative wiki or something (OpenStreetMap being the closest fit) and allow everyon

Why aren't these data sets editable instead of static? Treat them like a collaborative wiki or something (OpenStreetMap being the closest fit) and allow everyone to submit improvements so that all may benefit. I hope the people in this article had a way to contribute back their improvements, and did so.

seveibar 7y ago
I'm working on this[1], my theory is the lack of a good IDE (rather than simple crowdsourcing interface) is the reason why it hasn't been done. Imagine if github had an integrated ide for editing large datasets. Also see dolt which is doing good work here. [1] https://github.com/UniversalDataTool/universal-data-tool https://github.com/UniversalDataTool/universal-data-tool
lmkg 7y ago
One major use of the public datasets in the academic community is to serve as a common reference when comparing new techniques against the existing standard. A static baseline is desirable for this task. You could maybe split the difference by having an "original" or "reference" version, and a separate moving target that incorporates crowdsourced improvements.
- CydeWeys 7y ago
  This sounds like a revisioning system would help a lot. Have a quarterly or annual release cycle or something, so that when you want to compare performance across techniques, you just train both of them to the same target (and ideally all the papers coming out at roughly the same time would already be using the same revision anyway). You'd always work with a versioned release when training models, and you'd only typically work with HEAD when you were specifically looking to correct flaws in the data (as the authors in the linked article are).
6gvONxR4sf7o 7y ago
The datasets serve as benchmarks. You get an idea for a new model that solves a problem current models have. These ideas don't pan out, so you need empirical evidence that it works. To show that your model does better than previous models, you need some task that your model and previous models can share for both training and evaluation. It's more complicated than that, but that's the gist. It would be so wasteful to have to retrain a dozen models that require a month of GPU time each on to serve as baselines for your new model...
- barkingcat 7y ago
  That's not wasteful. That's correction. Is it wasteful to throw away a batch of food when 20% of it has been studied to contain the wrong substance, which ends up causing disease? Isn't it even more wasteful to continue using unedited and unverified data sets just because all the previous models were trained on it, and thus we can no longer advance the state of the research? It's a case of garbage in garbage out.
  - 6gvONxR4sf7o 7y ago
    >By one estimate, the training time for AlphaGo cost $35 million [0] How about XLNet which cost something like $30k-60k to train [1]? GPT-2 may have been around the same [2] is estimated around the same, while thankfully BERT only costs about $7k[3], unless of course you're going to do any new hyperparameter tuning on their models which you of course will do on your own model. Who cares about apples-to-apples comparisons? We're not talking about spending an extra couple hours and a little money on updated replication. We're talking about an immediate overhead of tens to hundreds of thousands of dollars per new paper. Tasks are updated over time already to take issues into account, but not continuously as far as I know. [0] https://www.wired.com/story/deepminds-losses-future-artificial-intelligence/ https://www.wired.com/story/deepminds-losses-future-artifici... [1] https://twitter.com/jekbradbury/status/1143397614093651969 https://twitter.com/jekbradbury/status/1143397614093651969 [2] https://news.ycombinator.com/item?id=19402666 https://news.ycombinator.com/item?id=19402666 [3] https://syncedreview.com/2019/06/27/the-staggering-cost-of-training-sota-ai-models/ https://syncedreview.com/2019/06/27/the-staggering-cost-of-t...
    - deleted 7y ago
      [deleted]
    - barkingcat 7y ago
      Yah it is by no means wasteful for AlphaGo to throw away all their training data and then re-train itself! That kind of ruthless experimentation is how AlphaGo was able to exceed even itself. The willingness to say - all these human games we've fed the computer? All these terabytes of data? It's all meaningless! We're going to throw it all away! We will have AlphaGo determine what is good by playing games against itself! And I bet you that for the next iteration of AlphaGo, the creators of this system will again, delete their own data and retrain when they have a better approach. If you don't "waste" your existing datasets (once you reallze the flaws in your data sets), you are being held back by the sunk cost principle. You only have yourself to blame when someone does train for the exact same purposes, but with cleaner data. The person who has the cleanest source of training data will win in deep learning. You're sabotaging yourself in my opinion. 30k is nothing when you're just sabotaging the training with faulty data.
polm23 7y ago
Multiple reasons, but to name a few: - Don't want to deal with vandalism - Hosting static data is dramatically easier than making a public editing interface - You want reference versions of the dataset for papers to refer to so that results are comparable. Sometimes this is used as a justification for not fixing completely broken data, like with Fasttext. https://github.com/facebookresearch/fastText/issues/710 https://github.com/facebookresearch/fastText/issues/710 - Building on the previous point, large datasets like this don't play nice with Git. There are lots of "git for data" things but none of them are very mature, and most people don't spend time trying to figure something out.
deleted 7y ago
[deleted]
xiphias2 7y ago
One problem with correcting the benchmark datasets is that it's important for the algorithms to be robust to labelling errors as well. But having multiple versions sounds important anyways.
andreyk 7y ago
Some datasets are indeed like this, see eg Common Voice - https://voice.mozilla.org/en https://voice.mozilla.org/en In general these things are open source, so you can always contribute an improved version of the dataset. But as another commenter said having relatively static ones is also important for benchmarking purposes.

seveibar 7y ago

I'm working on this[1], my theory is the lack of a good IDE (rather than simple crowdsourcing interface) is the reason why it hasn't been done. Imagine if github had an integrated ide for editing large datasets. Also see dolt which is doing good work here. [1] https://github.com/UniversalDataTool/universal-data-tool https://github.com/UniversalDataTool/universal-data-tool

lmkg 7y ago

One major use of the public datasets in the academic community is to serve as a common reference when comparing new techniques against the existing standard. A static baseline is desirable for this task. You could maybe split the difference by having an "original" or "reference" version, and a separate moving target that incorporates crowdsourced improvements.

CydeWeys 7y ago
This sounds like a revisioning system would help a lot. Have a quarterly or annual release cycle or something, so that when you want to compare performance across techniques, you just train both of them to the same target (and ideally all the papers coming out at roughly the same time would already be using the same revision anyway). You'd always work with a versioned release when training models, and you'd only typically work with HEAD when you were specifically looking to correct flaws in the data (as the authors in the linked article are).

6gvONxR4sf7o 7y ago

The datasets serve as benchmarks. You get an idea for a new model that solves a problem current models have. These ideas don't pan out, so you need empirical evidence that it works. To show that your model does better than previous models, you need some task that your model and previous models can share for both training and evaluation. It's more complicated than that, but that's the gist. It would be so wasteful to have to retrain a dozen models that require a month of GPU time each on to serve as baselines for your new model...

barkingcat 7y ago
That's not wasteful. That's correction. Is it wasteful to throw away a batch of food when 20% of it has been studied to contain the wrong substance, which ends up causing disease? Isn't it even more wasteful to continue using unedited and unverified data sets just because all the previous models were trained on it, and thus we can no longer advance the state of the research? It's a case of garbage in garbage out.
- 6gvONxR4sf7o 7y ago
  >By one estimate, the training time for AlphaGo cost $35 million [0] How about XLNet which cost something like $30k-60k to train [1]? GPT-2 may have been around the same [2] is estimated around the same, while thankfully BERT only costs about $7k[3], unless of course you're going to do any new hyperparameter tuning on their models which you of course will do on your own model. Who cares about apples-to-apples comparisons? We're not talking about spending an extra couple hours and a little money on updated replication. We're talking about an immediate overhead of tens to hundreds of thousands of dollars per new paper. Tasks are updated over time already to take issues into account, but not continuously as far as I know. [0] https://www.wired.com/story/deepminds-losses-future-artificial-intelligence/ https://www.wired.com/story/deepminds-losses-future-artifici... [1] https://twitter.com/jekbradbury/status/1143397614093651969 https://twitter.com/jekbradbury/status/1143397614093651969 [2] https://news.ycombinator.com/item?id=19402666 https://news.ycombinator.com/item?id=19402666 [3] https://syncedreview.com/2019/06/27/the-staggering-cost-of-training-sota-ai-models/ https://syncedreview.com/2019/06/27/the-staggering-cost-of-t...
  - deleted 7y ago
    [deleted]
  - barkingcat 7y ago
    Yah it is by no means wasteful for AlphaGo to throw away all their training data and then re-train itself! That kind of ruthless experimentation is how AlphaGo was able to exceed even itself. The willingness to say - all these human games we've fed the computer? All these terabytes of data? It's all meaningless! We're going to throw it all away! We will have AlphaGo determine what is good by playing games against itself! And I bet you that for the next iteration of AlphaGo, the creators of this system will again, delete their own data and retrain when they have a better approach. If you don't "waste" your existing datasets (once you reallze the flaws in your data sets), you are being held back by the sunk cost principle. You only have yourself to blame when someone does train for the exact same purposes, but with cleaner data. The person who has the cleanest source of training data will win in deep learning. You're sabotaging yourself in my opinion. 30k is nothing when you're just sabotaging the training with faulty data.

polm23 7y ago

Multiple reasons, but to name a few: - Don't want to deal with vandalism - Hosting static data is dramatically easier than making a public editing interface - You want reference versions of the dataset for papers to refer to so that results are comparable. Sometimes this is used as a justification for not fixing completely broken data, like with Fasttext. https://github.com/facebookresearch/fastText/issues/710 https://github.com/facebookresearch/fastText/issues/710 - Building on the previous point, large datasets like this don't play nice with Git. There are lots of "git for data" things but none of them are very mature, and most people don't spend time trying to figure something out.

deleted 7y ago

[deleted]

xiphias2 7y ago

One problem with correcting the benchmark datasets is that it's important for the algorithms to be robust to labelling errors as well. But having multiple versions sounds important anyways.

andreyk 7y ago

Some datasets are indeed like this, see eg Common Voice - https://voice.mozilla.org/en https://voice.mozilla.org/en In general these things are open source, so you can always contribute an improved version of the dataset. But as another commenter said having relatively static ones is also important for benchmarking purposes.