Based on my experience writing this Ruby library (https://github.com/SlyShy/Tactful_Tokenizer/ https://github.com/SlyShy/Tactful_Tokenizer/) it's a somewhat dif

Based on my experience writing this Ruby library (https://github.com/SlyShy/Tactful_Tokenizer/ https://github.com/SlyShy/Tactful_Tokenizer/) it's a somewhat difficult challenge. Most programs don't bother doing it accurately, as a naive solution will get you 80% of the way there. Of course, if you actually care about sentences (like I do), that's unacceptable.