4 ms·

Hi! We are comparing what you get by using an alternative service like api.ai (which you need to start by feeding manually) with what you can do with our solut

by oulipo 9y ago

Hi! We are comparing what you get by using an alternative service like api.ai (which you need to start by feeding manually) with what you can do with our solution. Manually feeding 70 queries per intent already takes a good deal of energy and imagination, so we thought it was representative of a good effort from a user. This is why Snips proposes to its users to automatically to use generative methods to create high-quality datasets (over 2000 queries per intent). All hypotheses are clarified in our blog post on performances which I encourage you to read! https://medium.com/@alicecoucke/benchmarking-natural-language-understanding-systems-google-facebook-microsoft-and-snips-2b8ddcf9fb19 https://medium.com/@alicecoucke/benchmarking-natural-languag... We even post all the underlying data and results on GitHub so people will be able to verify our claims: https://github.com/snipsco/nlu-benchmark/tree/master/2017-06-custom-intent-engines https://github.com/snipsco/nlu-benchmark/tree/master/2017-06...

Eridrus 9y ago
> Manually feeding 70 queries per intent already takes a good deal of energy and imagination Only if you have to manually make them up. If you have access to the query stream (which I imagine api.ai etc give you since they're online platforms), it's quite quick and easy. I can definitely appreciate integrating things together to make things easy, but this is a pretty biased comparison IMO. It's great that you've made the data available, but the headline metric that most people will see seems misleading.
- oulipo 9y ago
  API.ai and companies which keep user data on their servers are putting user privacy at a risk, and they don't really match the performance we can have when we do datascience carefully
vmarsy 9y ago
In the links you provide, you compare the F1 score for snips with other major competitors on 70 queries. Then you show the F1 score for snips on 2000 queries. Obviously you see improvements and that's good. What I don't understand is why don't you show the F1 score for the other competitors on 2000 queries? If you're able to generate 2000 input from the original 70, what prevents you from feeding those to the other NLP engines? Do they all get a 93% F1 score ?

by oulipo 9y ago

Eridrus 9y ago

> Manually feeding 70 queries per intent already takes a good deal of energy and imagination Only if you have to manually make them up. If you have access to the query stream (which I imagine api.ai etc give you since they're online platforms), it's quite quick and easy. I can definitely appreciate integrating things together to make things easy, but this is a pretty biased comparison IMO. It's great that you've made the data available, but the headline metric that most people will see seems misleading.

oulipo 9y ago
API.ai and companies which keep user data on their servers are putting user privacy at a risk, and they don't really match the performance we can have when we do datascience carefully

vmarsy 9y ago

In the links you provide, you compare the F1 score for snips with other major competitors on 70 queries. Then you show the F1 score for snips on 2000 queries. Obviously you see improvements and that's good. What I don't understand is why don't you show the F1 score for the other competitors on 2000 queries? If you're able to generate 2000 input from the original 70, what prevents you from feeding those to the other NLP engines? Do they all get a 93% F1 score ?