3 ms·

Bistring – Bidirectionally Transformed Strings

72 points by varunagrawal 7y ago

deleted 7y ago
[deleted]
blt 7y ago
This is interesting, but the readme doesn't say much about use cases. What is a big application that could benefit from this?
- microcolonel 7y ago
  Normalizing strings for comparison, but storing the original source version of the substring that matched or was extracted from the normalized version, useful for search boxes and such. Not sure what their specific use case is, but it's something I've implemented before, so I know it has use cases.
zawerf 7y ago
I was confused about the intended use case but there's more information in the docs folder: https://github.com/microsoft/bistring/blob/master/docs/Introduction.rst https://github.com/microsoft/bistring/blob/master/docs/Intro... Apparently it's for machine learning where you want to pick out a span/substring in the original text but your model can only accept normalized text (I am guessing for stuff like transforming out-of-vocabulary words into UNK/unknown tokens). This solves that problem by keeping track of the index mapping between the original text and transformed text. (picking out spans is very common task in NLP, for example see the SQuAD dataset: https://rajpurkar.github.io/SQuAD-explorer/explore/v2.0/dev/Normans.html https://rajpurkar.github.io/SQuAD-explorer/explore/v2.0/dev/...)
- bpicolo 7y ago
  Took me a minute to comprehend (that the substring bit is the whole point was not obvious, because all string operations in Python are non-destructive regardless) but this is exactly what I imagined given that note. Glad that thought lined up with reality
- tavianator 7y ago
  Yeah that's the idea. I'm not sure exactly how to communicate the purpose of the library in a few words. Possibly some animation that shows selecting spans of text and highlighting the corresponding one in the original text. The formatted documentation can be viewed here btw: https://bistring.readthedocs.io/en/latest/Introduction.html https://bistring.readthedocs.io/en/latest/Introduction.html
andrewflnr 7y ago
Somewhat related: Boomerang https://www.seas.upenn.edu/~harmony/ https://www.seas.upenn.edu/~harmony/ Discussed here at least once: https://news.ycombinator.com/item?id=565874 https://news.ycombinator.com/item?id=565874 The title made me think of Boomerang, this looks like it has rather different use cases in mind.

72 points by varunagrawal 7y ago

deleted 7y ago

[deleted]

blt 7y ago

This is interesting, but the readme doesn't say much about use cases. What is a big application that could benefit from this?

microcolonel 7y ago
Normalizing strings for comparison, but storing the original source version of the substring that matched or was extracted from the normalized version, useful for search boxes and such. Not sure what their specific use case is, but it's something I've implemented before, so I know it has use cases.

zawerf 7y ago

I was confused about the intended use case but there's more information in the docs folder: https://github.com/microsoft/bistring/blob/master/docs/Introduction.rst https://github.com/microsoft/bistring/blob/master/docs/Intro... Apparently it's for machine learning where you want to pick out a span/substring in the original text but your model can only accept normalized text (I am guessing for stuff like transforming out-of-vocabulary words into UNK/unknown tokens). This solves that problem by keeping track of the index mapping between the original text and transformed text. (picking out spans is very common task in NLP, for example see the SQuAD dataset: https://rajpurkar.github.io/SQuAD-explorer/explore/v2.0/dev/Normans.html https://rajpurkar.github.io/SQuAD-explorer/explore/v2.0/dev/...)

bpicolo 7y ago
Took me a minute to comprehend (that the substring bit is the whole point was not obvious, because all string operations in Python are non-destructive regardless) but this is exactly what I imagined given that note. Glad that thought lined up with reality
tavianator 7y ago
Yeah that's the idea. I'm not sure exactly how to communicate the purpose of the library in a few words. Possibly some animation that shows selecting spans of text and highlighting the corresponding one in the original text. The formatted documentation can be viewed here btw: https://bistring.readthedocs.io/en/latest/Introduction.html https://bistring.readthedocs.io/en/latest/Introduction.html

andrewflnr 7y ago

Somewhat related: Boomerang https://www.seas.upenn.edu/~harmony/ https://www.seas.upenn.edu/~harmony/ Discussed here at least once: https://news.ycombinator.com/item?id=565874 https://news.ycombinator.com/item?id=565874 The title made me think of Boomerang, this looks like it has rather different use cases in mind.