3 ms·

...surely in a domain-specific corpus such as HN comments you have to take domain, if not context, into account? That's a great insight and I believe you are c

by tansey 15y ago

...surely in a domain-specific corpus such as HN comments you have to take domain, if not context, into account? That's a great insight and I believe you are correct. In general, I think the domain does have an impact and I am working on some automatic demographic profiling technology. However, for a first-look, I think our general word dictionary covers an abundant set of terms that do not vary much between domains. For instance, your comment has the following anxiety-evoking words: not, important, but, example, positive, used, hacker, negative, long, bag, but, away Of all those words, only "hacker" would likely need to be adjusted for the HN domain. For a first look, we'll chalk that up to some minor added noise in the data. EffectCheck does include the ability to automatically profile the document type (e.g., "Press Release" or "Motivational Speech") based on a training corpus. For example, most press releases have a high degree of confidence-- based on a corpus of press releases, we normalize the levels so that "Very High" on confidence for a press release requires much higher confidence elicited per-word than it would in a complaint letter.

jlees 15y ago
Is there really much of interest in press releases, or were they just a handy corpus you had around? I started my PhD off looking at affect in scientific papers, and changed tack because there basically is none. Anyway, I guess I'm not clearly understanding your definition of "anxiety" because few of those words evoke anxiety as I would define it -- not? bag? away? I'll try to dig into whatever you've published to deeper understand your system, but I'm not convinced by the standalone word approach yet, though I see what you're trying to do (I think). I'd really want to check out the bigrams/trigrams, even if you don't do full parsing or any intermediate sophistication. Obviously there's an efficiency aspect, so I guess if your results are good enough already and you have customers, no need to make sacrifices. On a tangent, how does your system interpret common HN words like "healthcare", "Obama" and "education", I wonder? Could you have an "intellectual" or "hacker-ish" dimension based on manual categorisation of words into "this is stuff HN should be talking about" vs "this is stuff that belongs on reddit"?
- tansey 15y ago
  >Is there really much of interest in press releases, or were they just a handy corpus you had around? Actually, both. Corporate marketing departments spend a lot of effort crafting the proper press releases. Entire agencies exist to help companies with press releases. I think we make a great companion tool for professionals in that area. We have an example of how EffectCheck can help with press releases on our blog [1]. >On a tangent, how does your system interpret common HN words like "healthcare", "Obama" and "education", I wonder? Could you have an "intellectual" or "hacker-ish" dimension based on manual categorisation of words into "this is stuff HN should be talking about" vs "this is stuff that belongs on reddit"? We certainly can do that, and we offer it as a service to clients that want hand-crafted, demographic-specific dictionaries (like political campaigns). [1] http://blog.effectcheck.com/2011/04/14/delivering-bad-news-a-twitter-case-study/ http://blog.effectcheck.com/2011/04/14/delivering-bad-news-a...

by tansey 15y ago

jlees 15y ago

Is there really much of interest in press releases, or were they just a handy corpus you had around? I started my PhD off looking at affect in scientific papers, and changed tack because there basically is none. Anyway, I guess I'm not clearly understanding your definition of "anxiety" because few of those words evoke anxiety as I would define it -- not? bag? away? I'll try to dig into whatever you've published to deeper understand your system, but I'm not convinced by the standalone word approach yet, though I see what you're trying to do (I think). I'd really want to check out the bigrams/trigrams, even if you don't do full parsing or any intermediate sophistication. Obviously there's an efficiency aspect, so I guess if your results are good enough already and you have customers, no need to make sacrifices. On a tangent, how does your system interpret common HN words like "healthcare", "Obama" and "education", I wonder? Could you have an "intellectual" or "hacker-ish" dimension based on manual categorisation of words into "this is stuff HN should be talking about" vs "this is stuff that belongs on reddit"?

tansey 15y ago
>Is there really much of interest in press releases, or were they just a handy corpus you had around? Actually, both. Corporate marketing departments spend a lot of effort crafting the proper press releases. Entire agencies exist to help companies with press releases. I think we make a great companion tool for professionals in that area. We have an example of how EffectCheck can help with press releases on our blog [1]. >On a tangent, how does your system interpret common HN words like "healthcare", "Obama" and "education", I wonder? Could you have an "intellectual" or "hacker-ish" dimension based on manual categorisation of words into "this is stuff HN should be talking about" vs "this is stuff that belongs on reddit"? We certainly can do that, and we offer it as a service to clients that want hand-crafted, demographic-specific dictionaries (like political campaigns). [1] http://blog.effectcheck.com/2011/04/14/delivering-bad-news-a-twitter-case-study/ http://blog.effectcheck.com/2011/04/14/delivering-bad-news-a...