53 ms·

as a newcomer to machine learning.. how does one build a training set of "tens of millions of emails" ? This is one of the things I have struggled with... its p

as a newcomer to machine learning.. how does one build a training set of "tens of millions of emails" ? This is one of the things I have struggled with... its probably much harder to build a training set than the libraries and the algorithms themselves. are there companies that build training sets for you... or something else ?

golergka 10y ago
That's often a harder task than the actual machine learning. Also, the reason why established big companies have another upper hand in this space.
anonymousDan 10y ago
Isn't there a public Enron email dump?
yazr 10y ago
Boomerang is one of "bundle-ware" companies with some very dubious chrome extensions or whatever. This is how they how they collect plenty of data.
caseysoftware 10y ago
Yes, that is one of the biggest challenges for any machine learning system to be effective. And the more specific the data set, the harder it is to collect (or get access to) and probably the more valuable. Someone in this thread cites the Enron emails. Odds are if you're building system to analyze energy trading or (maybe) financial trading in general in American English, then that will work. The problem is that that's a small segment. The main way to collect this data is to effectively do a Trojan Horse by building something else in a complementary area that is useful that lets you start collecting the data. Once you have it, machine learning is useful. This is what Boomerang has done for email and what Google has done with Google Voice.
moah 10y ago
We have the Enron data set loaded into a VM ready to use if you use Python. It's on our blog somewhere. We also used the Jeb Bush data set and the Sony data set from wiki leaks. If you'd like, we can help you with setting them up. you can email me at moah@boomerangapp.com
- sandGorgon 10y ago
  @moah - thank you so much for this. This is really awesome. I will take you up on that offer.

golergka 10y ago

That's often a harder task than the actual machine learning. Also, the reason why established big companies have another upper hand in this space.

anonymousDan 10y ago

Isn't there a public Enron email dump?

yazr 10y ago

Boomerang is one of "bundle-ware" companies with some very dubious chrome extensions or whatever. This is how they how they collect plenty of data.

caseysoftware 10y ago

Yes, that is one of the biggest challenges for any machine learning system to be effective. And the more specific the data set, the harder it is to collect (or get access to) and probably the more valuable. Someone in this thread cites the Enron emails. Odds are if you're building system to analyze energy trading or (maybe) financial trading in general in American English, then that will work. The problem is that that's a small segment. The main way to collect this data is to effectively do a Trojan Horse by building something else in a complementary area that is useful that lets you start collecting the data. Once you have it, machine learning is useful. This is what Boomerang has done for email and what Google has done with Google Voice.

moah 10y ago

We have the Enron data set loaded into a VM ready to use if you use Python. It's on our blog somewhere. We also used the Jeb Bush data set and the Sony data set from wiki leaks. If you'd like, we can help you with setting them up. you can email me at moah@boomerangapp.com

sandGorgon 10y ago
@moah - thank you so much for this. This is really awesome. I will take you up on that offer.