5 ms·

Knwl.js – A JavaScript NLP

88 points by tilt 11y ago

eterm 11y ago
This is going to look like "oh typical HN shooting things down", but: Parses 13/4/2012 as: year: 2012 month: 13 day: 4
- lubomir 11y ago
  Also it does not consider 12:33 to be a time.
- tonyblundell 11y ago
  Yeah seems obvious, but I don't think it really makes sense to guess that it's a UK format date based on whether one of the numbers is bigger than 12. 4/5/2012 could be 4th May or 5th April and it would be impossible to tell programmatically. People really need to start using yyyy/mm/dd :-)
  - gtf21 11y ago
    dd/mm/yyyy and yyyy/mm/dd are as valid as each other, since it proceeds in either increasing or decreasing unit size. Having dates as mm/dd/yyyy makes about as much logical sense as writing: 10492 as 49210.
    - tonyblundell 11y ago
      It doesn't make sense mathematically because it isn't a mathematical notation, it's linguistic in nature. In Europe, '4th May' is shortened to 4/5. In the US, 'May 4th' is shortened to 5/4. Neither is right, wrong or illogical. I'm from the UK, but have spent a lot of time working with US based clients/colleagues. They only way I've found to avoid confusion and ambiguity is to use YYYY/MM/DD.
      - gtf21 11y ago
        It still makes no sense to have MM/DD/YYYY.
  - david-given 11y ago
    I live in Switzerland; I speak English (and a very slowly increasing amount of German). Switzerland uses DD.MM.YYYY almost exclusively (yes, with dots). I was filling in a web form using Google Translate. It took me several phone calls and about two weeks to sort out the problems caused when Translate back-converted some (but not all) of the dates that I'd entered into the form from DD.MM.YYYY to MM.DD.YYYY before submitting. /self rage
- richmarr 11y ago
  Not at all, it's a valid bug that's indicitaive of a deeper problem, presenting uncertain and locale-specific conclusions as being certain. Even if you do language and locale-sniffing using n-grams you still have a certainty problem. In the UK we traditionally use DD/MM/YY format, although since "9/11" the papers have started slipping into MM/DD format, and younger people have followed suit somewhat. In the UK it's therefore completely crazy to rely on NLP to interpret "6/7/12" which might mean 7th June or 6th July. If you're going to capture it you need to ask the user, so you need both.
  - wodenokoto 11y ago
    With that being said, it still needs to do sanity check. Is it a valid date? if not, will it be if we change the format? Nobody really knows what you mean when you write a data as xx/xx/xx. It has to be assumed from context. I've even have border control ask me about the expiration date of my passport since they couldn't figure out if it was day-month or month-day. I couldn't tell them either, because it wasn't written explicitly. Later I realised I could deduce it from my birthdate.
    - richmarr 11y ago
      Not to be overly inflamattory, but whoever designed that passport (an inherently international object) without taking into account international date format deserves to be fired. Then re-hired. Then fired again.
dlsym 11y ago
This could be a little more robust... Miss a dot and you get weird places like "if". Relative times don't get extracted. Something like: "Hi Frank, I've looked up the OIDs in the Cisco manual. Let's meet for lunch at burger king tomorrow at noon. S" yields only "Cisco" - as a place. Well. I know parsing natural language is hard. So keep up the good work and keep on tweaking :-)
talleyrand 11y ago
Can someone explain to me how that demo matches "Chicago"? From the source provided, I don't see how that is possible.
- yAnonymous 11y ago
  https://github.com/loadfive/Knwl.js/blob/master/experimental_plugins/english.js https://github.com/loadfive/Knwl.js/blob/master/experimental... knwl.english.prepositionalPhrases = [ ["about"], ["below"], ["in", "spite", "of"], ["regarding"], ["above"], ["beneath"], ["instead", "of"], ["since"], ["according", "to"], ["beside"], ["into"], ["through"], ["across"], ["between"], ["like"], ["throughout"], ["after"], ["beyond"], ["near"], ["to"], ["against"], ["but"], ["of"], ["toward"], ["along"], ["by"], ["off"], ["under"], ["amid"], ["concerning"], ["on"], ["underneath"], ["among"], ["down"], ["on", "account", "of"], ["until"], ["around"], ["during"], ["onto"], ["up"], ["at"], ["except"], ["out"], ["upon"], ["atop"], ["for"], ["out", "of"], ["with"], ["because", "of"], ["from"], ["outside"], ["within"], ["before"], ["in"], ["over"], ["without"], ["behind"], ["inside"], ["past"], ["away", "from"] ];
  - deleted 11y ago
    [deleted]
- lelf 11y ago
  "in" + word-starting-with-capital-letter.
  - talleyrand 11y ago
    Ah, brilliant! Thanks.
    - crossroads091 11y ago
      This fails if "in Chicago and London"
lelf 11y ago
sed -e "s,NLP,some hardcoded parsers for English-only dates/emails/links/…,"
jlink 11y ago
Do not find "London" when in the demo text I added this: "Regards from London,"
n8m 11y ago
I like it. I wonder if it would work with mixed in Markdown annotation.
esamek 11y ago
These two sentences were not parseable: "What if we head to Rockville and buy Daniel a beer or 2. May you fetch me a bottle of water?"
sagivo 11y ago
This is not NLP. The code just finds words in list of pre-defined words. It ignores the context of the sentence or perform any "real" NLP analyzing.
- jmsduran 11y ago
  You are correct, it is not NLP. But I wouldn't downplay the potential of this javascript library. I have hundreds of digital documents that need to be sorted and archived. I am able to extract a document's text via OCR, but now have to find a way to sort/file them based off certain keywords present within the document (date, time, place, etc). I think Knwl.js will be a perfect fit for this. So far I have only spent a couple of minutes looking through the demo and GitHub repository, but it looks like with this library I will be able to write an extension that can help detect restaurants and stores I frequently visit (which will further help me sort all those pesky paper receipts). I will definitely start playing around with Knwl.js once I get home. Good job!
- skillachie 11y ago
  It is NLP just not the latest "real" as you would call it. Its just rule based. http://www.quora.com/Why-are-rule-based-methods-becoming-unpopular-in-NLP http://www.quora.com/Why-are-rule-based-methods-becoming-unp... And http://en.wikipedia.org/wiki/Rule-based_system http://en.wikipedia.org/wiki/Rule-based_system
arihant 11y ago
It is faster than some of the others I've tried. Good thing is, since it is built to be basic and domain specific, the core should be kept as simple as it is, and allow extensions to be built on top. There are mess ups around sanity checks, but looking at their source it's actually trivial to fix. Forking it, I hope they accept community pull requests.
alistproducer2 11y ago
This NLP is extremely naive. It's just a hard-coded dictionary lookup with rudimentary English language syntax and structure recognition.
dboshardy 11y ago
knwl.english.adverbs = [ 'red', 'blue', 'green', 'orange', 'black', 'white', 'yellow', 'purple', 'really', 'sweet', 'sour', 'bitter', 'evil', ]; NLP. You keep using that word. I do not think it means what you think it means.
- ohitsdom 11y ago
  NLP or adverbs? Both seemed to be applied incorrectly...
  - Mahn 11y ago
    He means to say that Natural Language Processing isn't just scanning a text for patterns, like this library does, hence it's not a NLP library in the strict meaning of the term. It's an interesting project but it's really a parser, not a NLP.
    - ohitsdom 11y ago
      Yeah, I got that part. I was pointing out that this array of "adverbs" really contains adjectives.
- Gladdyu 11y ago
  Even though it might not match some definition of NLP, it does process a natural language. It might not include AI of some sorts but it does its job. It's (probably) fast and extracting basic information from written text is quite useful. Nice library!
- gflamino 11y ago
  Inconceivable!
T3RMINATED 11y ago
demo dont even work - i dont trust these guys... i looked at their code and its written by amateurs.
spodkowinski 11y ago
Nice parser collection. But there a more powerful NLP libraries out there for JS: https://github.com/spencermountain/nlp_compromise https://github.com/spencermountain/nlp_compromise
- thomasfoster96 11y ago
  That module looks pretty useful. I've been resorting to node-natural [0] and then using a quick part of speech tagger so far, but this looks a lot better and doesn't have a heap of baggage. [0] https://github.com/NaturalNode/natural https://github.com/NaturalNode/natural
  - nyxtom 11y ago
    A while back when I used to work a lot with this stuff I wrote a library to do a lot of sentiment, tokenization, and part of speech tagging based on several corpus I came across when I realized natural didn't have what I needed at the time. https://github.com/nyxtom/salient https://github.com/nyxtom/salient
elcct 11y ago
At thirst I was scared to click, because I though it is going to program me neurolinguistically to use Knwl.js whatever that is.

88 points by tilt 11y ago

eterm 11y ago

This is going to look like "oh typical HN shooting things down", but: Parses 13/4/2012 as: year: 2012 month: 13 day: 4

lubomir 11y ago
Also it does not consider 12:33 to be a time.
tonyblundell 11y ago
Yeah seems obvious, but I don't think it really makes sense to guess that it's a UK format date based on whether one of the numbers is bigger than 12. 4/5/2012 could be 4th May or 5th April and it would be impossible to tell programmatically. People really need to start using yyyy/mm/dd :-)
- gtf21 11y ago
  dd/mm/yyyy and yyyy/mm/dd are as valid as each other, since it proceeds in either increasing or decreasing unit size. Having dates as mm/dd/yyyy makes about as much logical sense as writing: 10492 as 49210.
  - tonyblundell 11y ago
    It doesn't make sense mathematically because it isn't a mathematical notation, it's linguistic in nature. In Europe, '4th May' is shortened to 4/5. In the US, 'May 4th' is shortened to 5/4. Neither is right, wrong or illogical. I'm from the UK, but have spent a lot of time working with US based clients/colleagues. They only way I've found to avoid confusion and ambiguity is to use YYYY/MM/DD.
    - gtf21 11y ago
      It still makes no sense to have MM/DD/YYYY.
- david-given 11y ago
  I live in Switzerland; I speak English (and a very slowly increasing amount of German). Switzerland uses DD.MM.YYYY almost exclusively (yes, with dots). I was filling in a web form using Google Translate. It took me several phone calls and about two weeks to sort out the problems caused when Translate back-converted some (but not all) of the dates that I'd entered into the form from DD.MM.YYYY to MM.DD.YYYY before submitting. /self rage
richmarr 11y ago
Not at all, it's a valid bug that's indicitaive of a deeper problem, presenting uncertain and locale-specific conclusions as being certain. Even if you do language and locale-sniffing using n-grams you still have a certainty problem. In the UK we traditionally use DD/MM/YY format, although since "9/11" the papers have started slipping into MM/DD format, and younger people have followed suit somewhat. In the UK it's therefore completely crazy to rely on NLP to interpret "6/7/12" which might mean 7th June or 6th July. If you're going to capture it you need to ask the user, so you need both.
- wodenokoto 11y ago
  With that being said, it still needs to do sanity check. Is it a valid date? if not, will it be if we change the format? Nobody really knows what you mean when you write a data as xx/xx/xx. It has to be assumed from context. I've even have border control ask me about the expiration date of my passport since they couldn't figure out if it was day-month or month-day. I couldn't tell them either, because it wasn't written explicitly. Later I realised I could deduce it from my birthdate.
  - richmarr 11y ago
    Not to be overly inflamattory, but whoever designed that passport (an inherently international object) without taking into account international date format deserves to be fired. Then re-hired. Then fired again.

dlsym 11y ago

This could be a little more robust... Miss a dot and you get weird places like "if". Relative times don't get extracted. Something like: "Hi Frank, I've looked up the OIDs in the Cisco manual. Let's meet for lunch at burger king tomorrow at noon. S" yields only "Cisco" - as a place. Well. I know parsing natural language is hard. So keep up the good work and keep on tweaking :-)

talleyrand 11y ago

Can someone explain to me how that demo matches "Chicago"? From the source provided, I don't see how that is possible.

yAnonymous 11y ago
https://github.com/loadfive/Knwl.js/blob/master/experimental_plugins/english.js https://github.com/loadfive/Knwl.js/blob/master/experimental... knwl.english.prepositionalPhrases = [ ["about"], ["below"], ["in", "spite", "of"], ["regarding"], ["above"], ["beneath"], ["instead", "of"], ["since"], ["according", "to"], ["beside"], ["into"], ["through"], ["across"], ["between"], ["like"], ["throughout"], ["after"], ["beyond"], ["near"], ["to"], ["against"], ["but"], ["of"], ["toward"], ["along"], ["by"], ["off"], ["under"], ["amid"], ["concerning"], ["on"], ["underneath"], ["among"], ["down"], ["on", "account", "of"], ["until"], ["around"], ["during"], ["onto"], ["up"], ["at"], ["except"], ["out"], ["upon"], ["atop"], ["for"], ["out", "of"], ["with"], ["because", "of"], ["from"], ["outside"], ["within"], ["before"], ["in"], ["over"], ["without"], ["behind"], ["inside"], ["past"], ["away", "from"] ];
- deleted 11y ago
  [deleted]
lelf 11y ago
"in" + word-starting-with-capital-letter.
- talleyrand 11y ago
  Ah, brilliant! Thanks.
  - crossroads091 11y ago
    This fails if "in Chicago and London"

lelf 11y ago

sed -e "s,NLP,some hardcoded parsers for English-only dates/emails/links/…,"

jlink 11y ago

Do not find "London" when in the demo text I added this: "Regards from London,"

n8m 11y ago

I like it. I wonder if it would work with mixed in Markdown annotation.

esamek 11y ago

These two sentences were not parseable: "What if we head to Rockville and buy Daniel a beer or 2. May you fetch me a bottle of water?"

sagivo 11y ago

This is not NLP. The code just finds words in list of pre-defined words. It ignores the context of the sentence or perform any "real" NLP analyzing.

jmsduran 11y ago
You are correct, it is not NLP. But I wouldn't downplay the potential of this javascript library. I have hundreds of digital documents that need to be sorted and archived. I am able to extract a document's text via OCR, but now have to find a way to sort/file them based off certain keywords present within the document (date, time, place, etc). I think Knwl.js will be a perfect fit for this. So far I have only spent a couple of minutes looking through the demo and GitHub repository, but it looks like with this library I will be able to write an extension that can help detect restaurants and stores I frequently visit (which will further help me sort all those pesky paper receipts). I will definitely start playing around with Knwl.js once I get home. Good job!
skillachie 11y ago
It is NLP just not the latest "real" as you would call it. Its just rule based. http://www.quora.com/Why-are-rule-based-methods-becoming-unpopular-in-NLP http://www.quora.com/Why-are-rule-based-methods-becoming-unp... And http://en.wikipedia.org/wiki/Rule-based_system http://en.wikipedia.org/wiki/Rule-based_system

arihant 11y ago

It is faster than some of the others I've tried. Good thing is, since it is built to be basic and domain specific, the core should be kept as simple as it is, and allow extensions to be built on top. There are mess ups around sanity checks, but looking at their source it's actually trivial to fix. Forking it, I hope they accept community pull requests.

alistproducer2 11y ago

This NLP is extremely naive. It's just a hard-coded dictionary lookup with rudimentary English language syntax and structure recognition.

dboshardy 11y ago

knwl.english.adverbs = [ 'red', 'blue', 'green', 'orange', 'black', 'white', 'yellow', 'purple', 'really', 'sweet', 'sour', 'bitter', 'evil', ]; NLP. You keep using that word. I do not think it means what you think it means.

ohitsdom 11y ago
NLP or adverbs? Both seemed to be applied incorrectly...
- Mahn 11y ago
  He means to say that Natural Language Processing isn't just scanning a text for patterns, like this library does, hence it's not a NLP library in the strict meaning of the term. It's an interesting project but it's really a parser, not a NLP.
  - ohitsdom 11y ago
    Yeah, I got that part. I was pointing out that this array of "adverbs" really contains adjectives.
Gladdyu 11y ago
Even though it might not match some definition of NLP, it does process a natural language. It might not include AI of some sorts but it does its job. It's (probably) fast and extracting basic information from written text is quite useful. Nice library!
gflamino 11y ago
Inconceivable!

T3RMINATED 11y ago

demo dont even work - i dont trust these guys... i looked at their code and its written by amateurs.

spodkowinski 11y ago

Nice parser collection. But there a more powerful NLP libraries out there for JS: https://github.com/spencermountain/nlp_compromise https://github.com/spencermountain/nlp_compromise

thomasfoster96 11y ago
That module looks pretty useful. I've been resorting to node-natural [0] and then using a quick part of speech tagger so far, but this looks a lot better and doesn't have a heap of baggage. [0] https://github.com/NaturalNode/natural https://github.com/NaturalNode/natural
- nyxtom 11y ago
  A while back when I used to work a lot with this stuff I wrote a library to do a lot of sentiment, tokenization, and part of speech tagging based on several corpus I came across when I realized natural didn't have what I needed at the time. https://github.com/nyxtom/salient https://github.com/nyxtom/salient

elcct 11y ago

At thirst I was scared to click, because I though it is going to program me neurolinguistically to use Knwl.js whatever that is.