3 ms·

Making preprocessors for creating bag or words vectors from free text, a special IP address feature processor, etc - these are all the ways we're starting to ma

Making preprocessors for creating bag or words vectors from free text, a special IP address feature processor, etc - these are all the ways we're starting to make the automation smarter and more of a value add to experienced folks like yourself. What value add would you want to see?

agibsonccc 12y ago
I usually roll my own. I've found that certain things, especially NLP pipelines, are usually very tricky to get right. Let me give you one example from my day to day: Sentence segmentation/tokenization ---> part of speech tagger ---> filter words by part of speech and then rank by tfidf scores for topic relevance scoring. Computer vision also has its own problems ---> binarize images for detection of certain kinds of features vs needing colors, so different kinds of image transformations for handling of different kinds of object recognition, or scene detection. I also do churn prediction, by the time you're done vectorizing users, why should I take time out of my day to then upload all of this to an external service when I could just run logistic regression, random forest or what have you locally? I typically calculate profit curves and the like as well. Re: your word vectors and deep learning. I do this in my distributed deep learning lib I hand wrote myself[1]. In my word2vec computations, I have found it takes a significant amount of data to tune right, I usually want control over this pipeline as well. Again, not your target audience ;). To give you some helpful feed back, don't try being everything to everybody, own a certain niche really well and run with it. If you'd like to take this discussion offline, I'd be more than happy to help, email is in my profile. Again, good luck with the service! [1] http://deeplearning4j.org/ http://deeplearning4j.org/

3 ms·

Making preprocessors for creating bag or words vectors from free text, a special IP address feature processor, etc - these are all the ways we're starting to ma

by muzakthings 12y ago

agibsonccc 12y ago
I usually roll my own. I've found that certain things, especially NLP pipelines, are usually very tricky to get right. Let me give you one example from my day to day: Sentence segmentation/tokenization ---> part of speech tagger ---> filter words by part of speech and then rank by tfidf scores for topic relevance scoring. Computer vision also has its own problems ---> binarize images for detection of certain kinds of features vs needing colors, so different kinds of image transformations for handling of different kinds of object recognition, or scene detection. I also do churn prediction, by the time you're done vectorizing users, why should I take time out of my day to then upload all of this to an external service when I could just run logistic regression, random forest or what have you locally? I typically calculate profit curves and the like as well. Re: your word vectors and deep learning. I do this in my distributed deep learning lib I hand wrote myself[1]. In my word2vec computations, I have found it takes a significant amount of data to tune right, I usually want control over this pipeline as well. Again, not your target audience ;). To give you some helpful feed back, don't try being everything to everybody, own a certain niche really well and run with it. If you'd like to take this discussion offline, I'd be more than happy to help, email is in my profile. Again, good luck with the service! [1] http://deeplearning4j.org/ http://deeplearning4j.org/