5 ms·

Microsoft.Recognizers.Text: numbers, units, and date/time in multiple languages

111 points by nailer 4y ago

gardenfelder 4y ago
It's way bigger that that: covers not just JS but Java, c# and more, and many languages https://github.com/microsoft/Recognizers-Text https://github.com/microsoft/Recognizers-Text
- johns 4y ago
  The link should be updated to this.
- dang 4y ago
  Ok, we've changed the URL to that from https://github.com/microsoft/Recognizers-Text/tree/master/JavaScript/packages/recognizers-text-suite https://github.com/microsoft/Recognizers-Text/tree/master/Ja.... Thanks!
UglyToad 4y ago
Used this for a quick proof of concept I was putting together where I needed to parse the output of an OCRed date and number (price) string (with known country). Seems to work very nicely on the sample size of 3 I tried with - so I can't speak to how well it works on a more diverse sample set - but it's great to have something like this available for free. In my second job we maintained our own, also Regex based, logic for this and it was a nightmare so having a library to do it is quite the timesaver!
tellarin 4y ago
I'm one of the maintainers of this project. Comments, suggestions, or contributions are very welcome! Specially regarding expansion to support new languages/cultures. We're looking at multiple ways to improve the project.
- nailer 4y ago
  Hey thanks! I'm the submitter and am using the the project in a real app - I appreciate the comprehensiveness of the project! One piece of feedback is: the documentation https://github.com/Microsoft/Recognizers-Text/tree/master/JavaScript/packages/recognizers-text-suite https://github.com/Microsoft/Recognizers-Text/tree/master/Ja... doesn't seem to cover the the various terms used in the results. Eg from my code: const recognizedDateTimes = recognizeDateTime(filterValue, "English"); // TODO: clarify what is distinct between each 'recognizedDateTime' // Seems like these are the 'recognisable items' that were found but I can't find docs. const firstDateFound = recognizedDateTimes[0]; // TODO: clarify what is a 'resolution' // Seems like this is each possible value for each recognisable item but I can't find docs. // eg if the user enters 'october' // values[0] is in the past // values[1] is in the future // we always want the past, hence picking 0. const firstResolution = firstDateFound?.resolution?.values[0] || null; A simple explainer in the README would be great.
  - tellarin 4y ago
    Thanks for the feedback. Could you please open an issue on GitHub? Thanks.
    - nailer 4y ago
      Sure and done. https://github.com/microsoft/Recognizers-Text/issues/3064 https://github.com/microsoft/Recognizers-Text/issues/3064
deleted 4y ago
[deleted]
KRAKRISMOTT 4y ago
Are these done via NER or old school heuristics?
- Closi 4y ago
  It's all Regex. > The Patterns folder contains all the regular expressions that fulfill the recognizers logic. It's divided by supported language.
  - sergiotapia 4y ago
    jesus! https://github.com/microsoft/Recognizers-Text/blob/master/JavaScript/packages/recognizers-number/src/resources/englishNumeric.ts https://github.com/microsoft/Recognizers-Text/blob/master/Ja...
    - shadowgovt 4y ago
      Yep. Regular expressions are great, but they rapidly fall over into write-only code for nontrivial applications.
    - Closi 4y ago
      Worth noting that these are machine-generated from a more verbose and clear codebase.
      - mhio 4y ago
        It looks like most PR's are for RE definitions in yaml files. Is there something that generates the yaml?
- meindnoch 4y ago
  https://github.com/microsoft/Recognizers-Text/blob/master/Patterns/English/English-DateTime.yaml https://github.com/microsoft/Recognizers-Text/blob/master/Pa...
binarymax 4y ago
This is awesome! A nice replacement for the Duckling service if you need fast regex based NER in your code.
- sc4les 4y ago
  Finally! Maintaining Duckling as a native Python library isn’t trivial unfortunately
revskill 4y ago
There's no Ruby yet.
stuaxo 4y ago
It could do with a link early on to some docs about what a recogniser is.
xnx 4y ago
Now if only Microsoft would put some of their recognizer smarts into Excel so that ZIP codes like "02201" don't get converted into "2,201".
- midasuni 4y ago
  That would break the business logic of millions of small businesses globally who rely on write-only excel spreadsheets to encode their business logic.
- deleted 4y ago
  [deleted]
- hnlmorg 4y ago
  You already can do that. Albeit it doesn’t detect it as a zip code specifically, but you can format the cells so that values that look like numbers aren’t automatically converted into numbers. Excel does have its quirks (CSV handling is one of my pet peeves) but it does also have a surprising amount of sophisticated adjustable logic that is often hidden in plain sight.
  - xnx 4y ago
    Definitely. I wish there was a global/permanent setting to not strip leading zeros. I imagine the design decision might have been made decades ago for Lotus 1-2-3 compatibility or something, but it's almost certainly cost much more time and mistakes than it has saved.
    - trekkie1024 4y ago
      This might be the setting you're looking for? https://www.howtogeek.com/816620/microsoft-is-finally-fixing-excels-data-conversion-problem/ https://www.howtogeek.com/816620/microsoft-is-finally-fixing...
s4i 4y ago
Can someone be awesome and ELI5 this? Not sure if I'm dumb or the people behind this have a very focused target audience in mind.
- marcosdumay 4y ago
  From what I understood, is a library that you can sends strings with "twenty thousand ninety eight" on it, and it tells you there is the number 20098 there.
  - nailer 4y ago
    More practically: 'last October' gives you a date range.
- perryizgr8 4y ago
  https://github.com/microsoft/Recognizers-Text/tree/master/.NET https://github.com/microsoft/Recognizers-Text/tree/master/.N... I think these examples explain the functionality very well.
- trilobyte 4y ago
  +1 for this. I have some sense reading the page but would love an ELI5.
  - kapp_in_life 4y ago
    It seems like its meant for standardizing/parsing freeform input, which I guess could be useful for things like chatbots? For example you send the user localized text, "Did you receive your package?" then parse the yes/no and continue your decision tree with something like https://www.nuget.org/packages/Microsoft.Recognizers.Text.Choice https://www.nuget.org/packages/Microsoft.Recognizers.Text.Ch...
    - WorldMaker 4y ago
      It's a part of the underpinnings of LUIS, Microsoft's Azure service for language understanding, and indeed part of building things like chatbots. An interesting comparison is that Microsoft.Recognizers.Text is a cross-platform cousin to JS library compromise/one: https://compromise.cool/ https://compromise.cool/

111 points by nailer 4y ago

It's way bigger that that: covers not just JS but Java, c# and more, and many languages https://github.com/microsoft/Recognizers-Text https://github.com/microsoft/Recognizers-Text

johns 4y ago
The link should be updated to this.
dang 4y ago
Ok, we've changed the URL to that from https://github.com/microsoft/Recognizers-Text/tree/master/JavaScript/packages/recognizers-text-suite https://github.com/microsoft/Recognizers-Text/tree/master/Ja.... Thanks!

UglyToad 4y ago

Used this for a quick proof of concept I was putting together where I needed to parse the output of an OCRed date and number (price) string (with known country). Seems to work very nicely on the sample size of 3 I tried with - so I can't speak to how well it works on a more diverse sample set - but it's great to have something like this available for free. In my second job we maintained our own, also Regex based, logic for this and it was a nightmare so having a library to do it is quite the timesaver!

tellarin 4y ago

I'm one of the maintainers of this project. Comments, suggestions, or contributions are very welcome! Specially regarding expansion to support new languages/cultures. We're looking at multiple ways to improve the project.

nailer 4y ago
Hey thanks! I'm the submitter and am using the the project in a real app - I appreciate the comprehensiveness of the project! One piece of feedback is: the documentation https://github.com/Microsoft/Recognizers-Text/tree/master/JavaScript/packages/recognizers-text-suite https://github.com/Microsoft/Recognizers-Text/tree/master/Ja... doesn't seem to cover the the various terms used in the results. Eg from my code: const recognizedDateTimes = recognizeDateTime(filterValue, "English"); // TODO: clarify what is distinct between each 'recognizedDateTime' // Seems like these are the 'recognisable items' that were found but I can't find docs. const firstDateFound = recognizedDateTimes[0]; // TODO: clarify what is a 'resolution' // Seems like this is each possible value for each recognisable item but I can't find docs. // eg if the user enters 'october' // values[0] is in the past // values[1] is in the future // we always want the past, hence picking 0. const firstResolution = firstDateFound?.resolution?.values[0] || null; A simple explainer in the README would be great.
- tellarin 4y ago
  Thanks for the feedback. Could you please open an issue on GitHub? Thanks.
  - nailer 4y ago
    Sure and done. https://github.com/microsoft/Recognizers-Text/issues/3064 https://github.com/microsoft/Recognizers-Text/issues/3064

deleted 4y ago

[deleted]

KRAKRISMOTT 4y ago

Are these done via NER or old school heuristics?

Closi 4y ago
It's all Regex. > The Patterns folder contains all the regular expressions that fulfill the recognizers logic. It's divided by supported language.
- sergiotapia 4y ago
  jesus! https://github.com/microsoft/Recognizers-Text/blob/master/JavaScript/packages/recognizers-number/src/resources/englishNumeric.ts https://github.com/microsoft/Recognizers-Text/blob/master/Ja...
  - shadowgovt 4y ago
    Yep. Regular expressions are great, but they rapidly fall over into write-only code for nontrivial applications.
  - Closi 4y ago
    Worth noting that these are machine-generated from a more verbose and clear codebase.
    - mhio 4y ago
      It looks like most PR's are for RE definitions in yaml files. Is there something that generates the yaml?
meindnoch 4y ago
https://github.com/microsoft/Recognizers-Text/blob/master/Patterns/English/English-DateTime.yaml https://github.com/microsoft/Recognizers-Text/blob/master/Pa...

binarymax 4y ago

This is awesome! A nice replacement for the Duckling service if you need fast regex based NER in your code.

sc4les 4y ago
Finally! Maintaining Duckling as a native Python library isn’t trivial unfortunately

revskill 4y ago

There's no Ruby yet.

stuaxo 4y ago

It could do with a link early on to some docs about what a recogniser is.

xnx 4y ago

Now if only Microsoft would put some of their recognizer smarts into Excel so that ZIP codes like "02201" don't get converted into "2,201".

midasuni 4y ago
That would break the business logic of millions of small businesses globally who rely on write-only excel spreadsheets to encode their business logic.
deleted 4y ago
[deleted]
hnlmorg 4y ago
You already can do that. Albeit it doesn’t detect it as a zip code specifically, but you can format the cells so that values that look like numbers aren’t automatically converted into numbers. Excel does have its quirks (CSV handling is one of my pet peeves) but it does also have a surprising amount of sophisticated adjustable logic that is often hidden in plain sight.
- xnx 4y ago
  Definitely. I wish there was a global/permanent setting to not strip leading zeros. I imagine the design decision might have been made decades ago for Lotus 1-2-3 compatibility or something, but it's almost certainly cost much more time and mistakes than it has saved.
  - trekkie1024 4y ago
    This might be the setting you're looking for? https://www.howtogeek.com/816620/microsoft-is-finally-fixing-excels-data-conversion-problem/ https://www.howtogeek.com/816620/microsoft-is-finally-fixing...

s4i 4y ago

Can someone be awesome and ELI5 this? Not sure if I'm dumb or the people behind this have a very focused target audience in mind.

marcosdumay 4y ago
From what I understood, is a library that you can sends strings with "twenty thousand ninety eight" on it, and it tells you there is the number 20098 there.
- nailer 4y ago
  More practically: 'last October' gives you a date range.
perryizgr8 4y ago
https://github.com/microsoft/Recognizers-Text/tree/master/.NET https://github.com/microsoft/Recognizers-Text/tree/master/.N... I think these examples explain the functionality very well.
trilobyte 4y ago
+1 for this. I have some sense reading the page but would love an ELI5.
- kapp_in_life 4y ago
  It seems like its meant for standardizing/parsing freeform input, which I guess could be useful for things like chatbots? For example you send the user localized text, "Did you receive your package?" then parse the yes/no and continue your decision tree with something like https://www.nuget.org/packages/Microsoft.Recognizers.Text.Choice https://www.nuget.org/packages/Microsoft.Recognizers.Text.Ch...
  - WorldMaker 4y ago
    It's a part of the underpinnings of LUIS, Microsoft's Azure service for language understanding, and indeed part of building things like chatbots. An interesting comparison is that Microsoft.Recognizers.Text is a cross-platform cousin to JS library compromise/one: https://compromise.cool/ https://compromise.cool/