4 ms·

Why not use a well tested, fast, off the shelf summary api? https://www.mashape.com/stremor/ https://www.mashape.com/stremor/ We do all the hard work. Even do

by drakaal 14y ago

Why not use a well tested, fast, off the shelf summary api? https://www.mashape.com/stremor/ https://www.mashape.com/stremor/ We do all the hard work. Even do the Content extraction from the page. And yes I have ties to this product, I'm the CTO of Stremor. No. that doesn't bias me against you building your own. I just know that we have 7.5 Million words in our code and I need more to get it perfect. Most people don't have time to assign traits to that many words. (Wordnet has about 350k) Edit: Also you have to deal with sentence disambiguation. That is hard. Which is why NLTK and Core NLP suck at it. Polyword Nouns and pronouns mean that you can't just use tokens, because "the President", "President Obama" and "He" are all the same "token". you have to rectify that when summarizing.

emidln 14y ago
As with most of the API services for text analytics, sending out confidential data to third parties is simply not acceptable. I run into this almost daily where I work. Contractually, we are neither allowed to send our clients' data to a fly-by-night startup nor a an established 800 pound gorilla. The other part is that often in text analytics, the training corpus matters significantly (also this seems less so with summaries at first glance).
- drakaal 14y ago
  Then you license it for onsite. The training corpus doesn't matter one bit. The type of document does. (Report, op-ed, News, thesis, Fiction, etc) We even do summary based on query. Summarize a 1000 page report only looking at things related to surgeries, or IT. We have hundreds of traits you can mix and match. Want to read the Snows of Kilimanjaro with only the parts that are directly about winter, or foreshadow winter themes? We can do that. Lord of the Rings only when encountering evil. We can do that.
  - emidln 14y ago
    Licensing it runs into another hurdle. How much value does random feature give you? Is the feature valuable enough to justify custom licensing costs? Would 100 lines of Python/NLTK give you enough value for your target market? I'm researching this right now for theme clustering. There are some products available for purchase/license that provide reasonable results. The question is if the difference between these reasonable results and 100 lines of Python/Clojure sitting ontop of Storm is enough to justify the cost of the license. That question relates to how much value our 100 line knockoff get our clients vs the extras that a full-feature solution would bring. In the few times I've done the comparison before, it has not been worth buying. Edit: I want to make it clear I'm not saying build it yourself 100% or that building it yourself will give you a reasonable facsimile in all (maybe any instances) with NLTK/OpenNLP/etc. If summarizing is a core feature of your product, by all means, license away. What I'm specifically referencing are situations where maybe I'd like to add a feature, but it's highly unlikely that the single feature along is going to drive a sale. In these circumstances, build it yourself with NLTK or the like is extremely attractive.
    - drakaal 14y ago
      The better question is if your clients care so much about privacy, is the risk that you shorten 3 sentences in such a way that "My client, Mr. Smith would only kill in self-defense. On the other hand Mr. Jones is a cold blooded killer. He has killed dozens of people" Becomes "My client, Mr. Jones is a cold blooded Killer" Because of sentence parsing. Or "Smith would only kill in self-defense. He has killed dozens of people." Because you used keyword density.
adventured 14y ago
Is there someone at Stremor I should specifically talk to? When I looked at your site recently, referred by a Forbes article, I got the impression your technology for summary had become a closed solution (available just to your products). Or can I access it just through mashape perhaps?
- akavlie 14y ago
  Yes, our summarization technology is now available as an API at the Mashape link in drakaal's post. If you have further questions, feel free to email us: support at stremor dot com.

by drakaal 14y ago

emidln 14y ago

As with most of the API services for text analytics, sending out confidential data to third parties is simply not acceptable. I run into this almost daily where I work. Contractually, we are neither allowed to send our clients' data to a fly-by-night startup nor a an established 800 pound gorilla. The other part is that often in text analytics, the training corpus matters significantly (also this seems less so with summaries at first glance).

drakaal 14y ago
Then you license it for onsite. The training corpus doesn't matter one bit. The type of document does. (Report, op-ed, News, thesis, Fiction, etc) We even do summary based on query. Summarize a 1000 page report only looking at things related to surgeries, or IT. We have hundreds of traits you can mix and match. Want to read the Snows of Kilimanjaro with only the parts that are directly about winter, or foreshadow winter themes? We can do that. Lord of the Rings only when encountering evil. We can do that.
- emidln 14y ago
  Licensing it runs into another hurdle. How much value does random feature give you? Is the feature valuable enough to justify custom licensing costs? Would 100 lines of Python/NLTK give you enough value for your target market? I'm researching this right now for theme clustering. There are some products available for purchase/license that provide reasonable results. The question is if the difference between these reasonable results and 100 lines of Python/Clojure sitting ontop of Storm is enough to justify the cost of the license. That question relates to how much value our 100 line knockoff get our clients vs the extras that a full-feature solution would bring. In the few times I've done the comparison before, it has not been worth buying. Edit: I want to make it clear I'm not saying build it yourself 100% or that building it yourself will give you a reasonable facsimile in all (maybe any instances) with NLTK/OpenNLP/etc. If summarizing is a core feature of your product, by all means, license away. What I'm specifically referencing are situations where maybe I'd like to add a feature, but it's highly unlikely that the single feature along is going to drive a sale. In these circumstances, build it yourself with NLTK or the like is extremely attractive.
  - drakaal 14y ago
    The better question is if your clients care so much about privacy, is the risk that you shorten 3 sentences in such a way that "My client, Mr. Smith would only kill in self-defense. On the other hand Mr. Jones is a cold blooded killer. He has killed dozens of people" Becomes "My client, Mr. Jones is a cold blooded Killer" Because of sentence parsing. Or "Smith would only kill in self-defense. He has killed dozens of people." Because you used keyword density.

adventured 14y ago

Is there someone at Stremor I should specifically talk to? When I looked at your site recently, referred by a Forbes article, I got the impression your technology for summary had become a closed solution (available just to your products). Or can I access it just through mashape perhaps?

akavlie 14y ago
Yes, our summarization technology is now available as an API at the Mashape link in drakaal's post. If you have further questions, feel free to email us: support at stremor dot com.