5 ms·

"How do you differentiate between dr. mr., 2004. jun. , and valid sentence-enders?" Fun fact: there exists a convention stipulating a double space after a peri

by praptak 16y ago

"How do you differentiate between dr. mr., 2004. jun. , and valid sentence-enders?" Fun fact: there exists a convention stipulating a double space after a period that ends a sentence. Not that I'm advocating relying on this for any serious purposes

RickHull 16y ago
There is also a convention of capitalizing the first letter of a sentence. The example text demonstrates that corpus quality (forgive the fuzzy terminology) trumps clean, simple rules.
- billybob 16y ago
  Yep. First letter of a sentence, or title, or any other proper noun. So... how can your program tell which it is? The correct algorithm approaches AI in complexity.
  - sdrinf 16y ago
    In data/text mining disciplines, correctness is fuzzy, rather than being boolean. For most industry-wide applications, having a solution that covers 99.9% of the cases (1 mis-classification in 1000 sample) is well below acceptable bounds. So, one particular solution with these performance characteristics is building a decision tree using a bunch of training data, and eg. a maximum entropy classifier. Add some sample data from any openly available corpus's (or fire up mturk, and create your own), and you're pretty much done with it. Of course, sentence-tokenization is only the tip of the iceberg :)
geekdesigngirl 16y ago
I grew up with that drilled into my head in typing class but, come to find out, now it is not followed in many cases. Most type faces should only have 1 space after the period. http://www.wsu.edu/~brians/errors/spaces.html http://www.wsu.edu/~brians/errors/spaces.html It took me forever to stop hitting the space bar twice after ending a sentence.
- timcederman 16y ago
  For my thesis, the easiest thing was to do a find/replace on ". " with ". " when I was done.
  - presidentender 16y ago
    And because of the tendency for a browser to strip whitespace from HTML to render pages readably, those two things look exactly the same. If you want consecutive spaces in a bare HTML page, you have to use " " which doesn't work here because pg isn't a moron.
    - spicyj 16y ago
      You can use the appropriate special character instead of the entity. Like this.
      - jrockway 16y ago
        Because you aren't a moron.
        presidentender 16y ago
        I am though :(
- jedbrown 16y ago
  TeX
billybob 16y ago
This is far from universal. I know that when I studied journalism, I had to unlearn that habit - I think the official style guide for the industry mandates one space.
- joshuacc 16y ago
  Indeed, it seems to have gone out of fashion at the same time as the mechanical typewriter. http://en.wikipedia.org/wiki/Double_spacing_at_the_end_of_sentences http://en.wikipedia.org/wiki/Double_spacing_at_the_end_of_se...
jerf 16y ago
HTML ended that. I have followed all periods with two spaces in this post. How can you tell on the screen? In fact I just checked, and HN is honoring the two spaces, in that they are output to the actual HTML sent to your browser. And of course, yes, there were other trends that would have ended this anyhow, "two spaces" in meaningless in a non-monospaced font and ever more stuff is going proportional as the computing power necessary to do that continues its steady march from "prohibitive" to "trivial", but the WWW certainly beat the corpse to death again.
- caf 16y ago
  It is not meaningless in a non-monospaced font; two spaces should still be wider than one space. I use a double space after a sentence in all of my technical writing, and I'd be extremely disappointed if my wordprocessing package did not honour that.
  - eru 16y ago
    LaTeX would not honour it. But you can ask it to treat the end of sentences different than regular spaces.
- nitrogen 16y ago
  The monospaced or proportional fonts aren't the issue; it's that HTML treats all sequences of whitespace within text as a single space. It's quite annoying to those of us who like our double-spaced sentences. When a word processor generates HTML, every double-spaced sentence ends with this: [space]  Or even this: [space]<span class="something-about-space">[space]</span>
  - jerf 16y ago
    HTML and proportional fonts are two separate issues. HTML ignores them, which is one problem; two spaces in a proportional font being less immediately obvious than on a typewriter is another problem. You can come up with some other issues too if you think about it. It all adds up to a dead tradition.
boulderdash 16y ago
That was an old typewriter convention. I learned typing on a typewriter in highshool. Wordprocessors in the 80's eliminated that convention. They would automatically perform the proper spacing.

by praptak 16y ago

RickHull 16y ago

There is also a convention of capitalizing the first letter of a sentence. The example text demonstrates that corpus quality (forgive the fuzzy terminology) trumps clean, simple rules.

billybob 16y ago
Yep. First letter of a sentence, or title, or any other proper noun. So... how can your program tell which it is? The correct algorithm approaches AI in complexity.
- sdrinf 16y ago
  In data/text mining disciplines, correctness is fuzzy, rather than being boolean. For most industry-wide applications, having a solution that covers 99.9% of the cases (1 mis-classification in 1000 sample) is well below acceptable bounds. So, one particular solution with these performance characteristics is building a decision tree using a bunch of training data, and eg. a maximum entropy classifier. Add some sample data from any openly available corpus's (or fire up mturk, and create your own), and you're pretty much done with it. Of course, sentence-tokenization is only the tip of the iceberg :)

geekdesigngirl 16y ago

I grew up with that drilled into my head in typing class but, come to find out, now it is not followed in many cases. Most type faces should only have 1 space after the period. http://www.wsu.edu/~brians/errors/spaces.html http://www.wsu.edu/~brians/errors/spaces.html It took me forever to stop hitting the space bar twice after ending a sentence.

timcederman 16y ago
For my thesis, the easiest thing was to do a find/replace on ". " with ". " when I was done.
- presidentender 16y ago
  And because of the tendency for a browser to strip whitespace from HTML to render pages readably, those two things look exactly the same. If you want consecutive spaces in a bare HTML page, you have to use " " which doesn't work here because pg isn't a moron.
  - spicyj 16y ago
    You can use the appropriate special character instead of the entity. Like this.
    - jrockway 16y ago
      Because you aren't a moron.
      - presidentender 16y ago
        I am though :(
jedbrown 16y ago
TeX

billybob 16y ago

This is far from universal. I know that when I studied journalism, I had to unlearn that habit - I think the official style guide for the industry mandates one space.

joshuacc 16y ago
Indeed, it seems to have gone out of fashion at the same time as the mechanical typewriter. http://en.wikipedia.org/wiki/Double_spacing_at_the_end_of_sentences http://en.wikipedia.org/wiki/Double_spacing_at_the_end_of_se...

jerf 16y ago

HTML ended that. I have followed all periods with two spaces in this post. How can you tell on the screen? In fact I just checked, and HN is honoring the two spaces, in that they are output to the actual HTML sent to your browser. And of course, yes, there were other trends that would have ended this anyhow, "two spaces" in meaningless in a non-monospaced font and ever more stuff is going proportional as the computing power necessary to do that continues its steady march from "prohibitive" to "trivial", but the WWW certainly beat the corpse to death again.

caf 16y ago
It is not meaningless in a non-monospaced font; two spaces should still be wider than one space. I use a double space after a sentence in all of my technical writing, and I'd be extremely disappointed if my wordprocessing package did not honour that.
- eru 16y ago
  LaTeX would not honour it. But you can ask it to treat the end of sentences different than regular spaces.
nitrogen 16y ago
The monospaced or proportional fonts aren't the issue; it's that HTML treats all sequences of whitespace within text as a single space. It's quite annoying to those of us who like our double-spaced sentences. When a word processor generates HTML, every double-spaced sentence ends with this: [space]  Or even this: [space]<span class="something-about-space">[space]</span>
- jerf 16y ago
  HTML and proportional fonts are two separate issues. HTML ignores them, which is one problem; two spaces in a proportional font being less immediately obvious than on a typewriter is another problem. You can come up with some other issues too if you think about it. It all adds up to a dead tradition.

boulderdash 16y ago

That was an old typewriter convention. I learned typing on a typewriter in highshool. Wordprocessors in the 80's eliminated that convention. They would automatically perform the proper spacing.