3 ms·

The TIMIT (speech) benchmark they use is not freely available - you have to pay for it. It is a great shame that papers use this benchmark - how do I reproduc

The TIMIT (speech) benchmark they use is not freely available - you have to pay for it. It is a great shame that papers use this benchmark - how do I reproduce their results? How do I compare my methods to theirs? Meanwhile the image benchmarks MNIST, NORB, CIFAR-10, CIFAR-100 are freely available. Kudos to the people who made them.

espadrine 12y ago
Could we crowdsource a speech benchmark? We start with one individual with a trust level of 1 (a probability of correct work in the bayesian sense). All other contributors start with a trust level of 0. Anyone with a trust level above MIN_TRUST (say, 0.6), called a trustee, can validate others' work. This status is dynamic, such that a trustee can stop being one, invalidating all its verifications. Valid work is work that has a score above MIN_TRUST. Such work is included in the benchmark (with a possible added check, such as a lower bound for the number of votes received). The score of a work is the lower bound of the Wilson score confidence interval for a Bernoulli parameter with a confidence level of 95%. Given `total` the number of votes from trustees and `valid` the number of votes that claimed this was a valid work: z = 1.96 z2 = z * z positive = valid / total score(work) = positive + z2 / (2*total) - z * sqrt((positive*(1-positive) + z2/(4*total)) / total)) / (1 + z2/n) The trust level of each contributor is computed as the proportion of validated work (by a trustee) amongst their work, minus the proportion of invalid work. In math: trust(contributor) = max(0, (valid_work - invalid_work) / total_work) Each trustee may verify as many pieces of work as they have produced themselves. They receive work randomly amongst work that is not yet valid. A piece of work can also be discarded if it reached a certain number of votes and has a certain low score. In this case, each piece of work would be some text read out loud by the contributor.
- bainsfather 12y ago
  There is this: http://www.voxforge.org/home http://www.voxforge.org/home "VoxForge was set up to collect transcribed speech for use with Free and Open Source Speech Recognition Engines (on Linux, Windows and Mac)." I also wondered about using librivox audiobooks+text for training, maybe.
jwatte 12y ago
Research costs money. You need machines, a place to work, employees need to eat and pay the rent (even if you're the only employee.) Cost of database is just another cost of research. Most scientific fields have significantly higher costs than computer science. Just be glad you're not in high energy physics...
- bainsfather 12y ago
  I'm not asking for access to their hardware (computers or particle accelerators). I am asking for access to their data - something that costs them nothing more whether I have it or not. Science and the free sharing of information go together.

espadrine 12y ago

Could we crowdsource a speech benchmark? We start with one individual with a trust level of 1 (a probability of correct work in the bayesian sense). All other contributors start with a trust level of 0. Anyone with a trust level above MIN_TRUST (say, 0.6), called a trustee, can validate others' work. This status is dynamic, such that a trustee can stop being one, invalidating all its verifications. Valid work is work that has a score above MIN_TRUST. Such work is included in the benchmark (with a possible added check, such as a lower bound for the number of votes received). The score of a work is the lower bound of the Wilson score confidence interval for a Bernoulli parameter with a confidence level of 95%. Given `total` the number of votes from trustees and `valid` the number of votes that claimed this was a valid work: z = 1.96 z2 = z * z positive = valid / total score(work) = positive + z2 / (2*total) - z * sqrt((positive*(1-positive) + z2/(4*total)) / total)) / (1 + z2/n) The trust level of each contributor is computed as the proportion of validated work (by a trustee) amongst their work, minus the proportion of invalid work. In math: trust(contributor) = max(0, (valid_work - invalid_work) / total_work) Each trustee may verify as many pieces of work as they have produced themselves. They receive work randomly amongst work that is not yet valid. A piece of work can also be discarded if it reached a certain number of votes and has a certain low score. In this case, each piece of work would be some text read out loud by the contributor.

bainsfather 12y ago
There is this: http://www.voxforge.org/home http://www.voxforge.org/home "VoxForge was set up to collect transcribed speech for use with Free and Open Source Speech Recognition Engines (on Linux, Windows and Mac)." I also wondered about using librivox audiobooks+text for training, maybe.

jwatte 12y ago

Research costs money. You need machines, a place to work, employees need to eat and pay the rent (even if you're the only employee.) Cost of database is just another cost of research. Most scientific fields have significantly higher costs than computer science. Just be glad you're not in high energy physics...

bainsfather 12y ago
I'm not asking for access to their hardware (computers or particle accelerators). I am asking for access to their data - something that costs them nothing more whether I have it or not. Science and the free sharing of information go together.