5 ms·

It would be interesting to see the major parts of speech (nouns, verbs, adjectives) colored. Instead this is a coloring of fairly random words. A bunch of short

It would be interesting to see the major parts of speech (nouns, verbs, adjectives) colored. Instead this is a coloring of fairly random words. A bunch of short words are grey, but they don't belong to any particular part of speech. They include some articles, prepositions, conjunctions and a few verbs...

cyphar 11y ago
I think we'd need to solve computational linguistics for a completely accurate parser to be able to tag the words properly. Although, the current state of the science works for something like 80% of cases (which are the "easy" ones).
- personjerry 11y ago
  I think a naive approach would do quite well. We're not looking to comprehend the sentence entirely, rather a fairly good accuracy, so that the reader only needs, usually, focus on the highlighted words. For example, serif'd, bolded verbs and noun would retain most of the information while filler words like "the" and "and" might not be necessary. The consequent highlighted parts reminds me a lot of Chinese, where essentially every word is "important" and there are few filler words, and hence there is often a lot of contextual information.
- foobarbecue 11y ago
  Looks like http://parts-of-speech.info/ http://parts-of-speech.info/ does a pretty good job at detection; too bad the styling is so garish.
- verroq 11y ago
  Sorry but POS tagging is pretty much solved already. It is already in the 97+% [1]. Current papers are now mostly improving it by less than one percent. 1. http://nlp.stanford.edu/pubs/CICLing2011-manning-tagging.pdf http://nlp.stanford.edu/pubs/CICLing2011-manning-tagging.pdf
  - glup 11y ago
    For the record, 97% on canonical test datasets with little recent progress doesn't mean that it's a solved problem. Admittedly, part of the problem is that elementary school POS categories aren't a great model of natural language. More generally, for true syntax highlighting I think you do need the parse tree, and parsers (as opposed to taggers) definitely aren't at 97%.
    - Al-Khwarizmi 11y ago
      They definitely aren't at 97% but they aren't that bad either. For English they are at around 92% (see for example http://arxiv.org/pdf/1603.04351.pdf http://arxiv.org/pdf/1603.04351.pdf) in labelled attachment score (right tag+right label). If you are interested only in the syntactic tag, not in the structure of the tree, the number is somewhat higher.
  - cyphar 11y ago
    To be fair, I'm not a computational linguist (some of my friends did their PhDs in the field though). From what I remember, one of the most glaring issues in the field is that the most used corpus is a bunch of issues of the Wall Street Journal (which is a very specific data set). The 80% figure was quoted from some talk I heard a few years ago, so I concede it's almost certainly improved since then.
  - Someone 11y ago
    On the other hand, 97% sounds impressive, but also would, on average, be slightly less than one error in your post (and more than one in this one) Graded as a school exercise, I think 97% wouldn't be that good.
  - Gorgor 11y ago
    It's true that POS tagging works fairly well. But consider that a sentence involves more than one word. Even at 97 % accuracy for one word, the probability of correctly tagging every word in a short sentence of only ten words, is still as low as 0.97^10 = 0.74. And sentences are generally longer than ten words. And as POS tagging is usually only done as preprocessing for some other task like syntactically parsing a text (which itself is usually preprocessing for yet another task), 97 % accuracy per word is not as good as it sounds. Parsers need to work with wrong data for every second or third sentence.
SixSigma 11y ago
One site I use to improve my prose is http://hemingwayapp.com/ http://hemingwayapp.com/ not quite what you're proposing but useful nonetheless
wodenokoto 11y ago
Those are stop words. You usually remove them from a text before analysing word frequencies because they are so common to all English text that they don't tell you anything specific about the current text. I don't know what the rational is for greying them out is, but that's the category.
- elros 11y ago
  Sure, but "stop words" are a computing concept as opposed to "parts of speech" which is a linguistic one, which I think is related to his point. https://en.wikipedia.org/wiki/Stop_words https://en.wikipedia.org/wiki/Stop_words https://en.wikipedia.org/wiki/Part_of_speech https://en.wikipedia.org/wiki/Part_of_speech
yegortimoshenko 11y ago
It is not really interesting, see: http://www.linusakesson.net/programming/syntaxhighlighting/index.php http://www.linusakesson.net/programming/syntaxhighlighting/i...
- userbinator 11y ago
  I thought of that article too, and while this highlighter isn't quite as colourful as that example, it still felt more distracting to read than monochrome text.
- yAnonymous 11y ago
  You can stop spamming that blog now. The author has a wrong understanding of what syntax highlighting tries to achieve and because of that he comes to questionable conclusions. It's really just a way to get people to visit his blog by making a bold statement.
  - koehr 11y ago
    Why is that spamming? Only because you don't share the writers opinion doesn't meant it doesn't fit here, right?
cced 11y ago
There's a cool app on ios which does this, iA Writer. I would love to know how they go about doing this.

cyphar 11y ago

I think we'd need to solve computational linguistics for a completely accurate parser to be able to tag the words properly. Although, the current state of the science works for something like 80% of cases (which are the "easy" ones).

personjerry 11y ago
I think a naive approach would do quite well. We're not looking to comprehend the sentence entirely, rather a fairly good accuracy, so that the reader only needs, usually, focus on the highlighted words. For example, serif'd, bolded verbs and noun would retain most of the information while filler words like "the" and "and" might not be necessary. The consequent highlighted parts reminds me a lot of Chinese, where essentially every word is "important" and there are few filler words, and hence there is often a lot of contextual information.
foobarbecue 11y ago
Looks like http://parts-of-speech.info/ http://parts-of-speech.info/ does a pretty good job at detection; too bad the styling is so garish.
verroq 11y ago
Sorry but POS tagging is pretty much solved already. It is already in the 97+% [1]. Current papers are now mostly improving it by less than one percent. 1. http://nlp.stanford.edu/pubs/CICLing2011-manning-tagging.pdf http://nlp.stanford.edu/pubs/CICLing2011-manning-tagging.pdf
- glup 11y ago
  For the record, 97% on canonical test datasets with little recent progress doesn't mean that it's a solved problem. Admittedly, part of the problem is that elementary school POS categories aren't a great model of natural language. More generally, for true syntax highlighting I think you do need the parse tree, and parsers (as opposed to taggers) definitely aren't at 97%.
  - Al-Khwarizmi 11y ago
    They definitely aren't at 97% but they aren't that bad either. For English they are at around 92% (see for example http://arxiv.org/pdf/1603.04351.pdf http://arxiv.org/pdf/1603.04351.pdf) in labelled attachment score (right tag+right label). If you are interested only in the syntactic tag, not in the structure of the tree, the number is somewhat higher.
- cyphar 11y ago
  To be fair, I'm not a computational linguist (some of my friends did their PhDs in the field though). From what I remember, one of the most glaring issues in the field is that the most used corpus is a bunch of issues of the Wall Street Journal (which is a very specific data set). The 80% figure was quoted from some talk I heard a few years ago, so I concede it's almost certainly improved since then.
- Someone 11y ago
  On the other hand, 97% sounds impressive, but also would, on average, be slightly less than one error in your post (and more than one in this one) Graded as a school exercise, I think 97% wouldn't be that good.
- Gorgor 11y ago
  It's true that POS tagging works fairly well. But consider that a sentence involves more than one word. Even at 97 % accuracy for one word, the probability of correctly tagging every word in a short sentence of only ten words, is still as low as 0.97^10 = 0.74. And sentences are generally longer than ten words. And as POS tagging is usually only done as preprocessing for some other task like syntactically parsing a text (which itself is usually preprocessing for yet another task), 97 % accuracy per word is not as good as it sounds. Parsers need to work with wrong data for every second or third sentence.

SixSigma 11y ago

One site I use to improve my prose is http://hemingwayapp.com/ http://hemingwayapp.com/ not quite what you're proposing but useful nonetheless

wodenokoto 11y ago

Those are stop words. You usually remove them from a text before analysing word frequencies because they are so common to all English text that they don't tell you anything specific about the current text. I don't know what the rational is for greying them out is, but that's the category.

elros 11y ago
Sure, but "stop words" are a computing concept as opposed to "parts of speech" which is a linguistic one, which I think is related to his point. https://en.wikipedia.org/wiki/Stop_words https://en.wikipedia.org/wiki/Stop_words https://en.wikipedia.org/wiki/Part_of_speech https://en.wikipedia.org/wiki/Part_of_speech

yegortimoshenko 11y ago

It is not really interesting, see: http://www.linusakesson.net/programming/syntaxhighlighting/index.php http://www.linusakesson.net/programming/syntaxhighlighting/i...

userbinator 11y ago
I thought of that article too, and while this highlighter isn't quite as colourful as that example, it still felt more distracting to read than monochrome text.
yAnonymous 11y ago
You can stop spamming that blog now. The author has a wrong understanding of what syntax highlighting tries to achieve and because of that he comes to questionable conclusions. It's really just a way to get people to visit his blog by making a bold statement.
- koehr 11y ago
  Why is that spamming? Only because you don't share the writers opinion doesn't meant it doesn't fit here, right?

cced 11y ago

There's a cool app on ios which does this, iA Writer. I would love to know how they go about doing this.