4 ms·

Sigh, thank you for ranting for me. I just want to add to what you said: > > There is not much choice how to split a string into words (split on 0x20? yes. sp

by PeCaN 11y ago

Sigh, thank you for ranting for me. I just want to add to what you said: > > There is not much choice how to split a string into words (split on 0x20? yes. split on 0x0a? mostly. split on 0x09? maybe. allow carriage return? maybe.). > > For Unicode, I don't even know where to look up how to do it. UAX #14 and #29 specify perfectly well how to do this[1][2]. There are even C libraries to do this for you[3][4]. In general, if you have a Unicode question, the standard clears things up reasonably well. It's somewhat easy reading for a standardization document. 1. http://www.unicode.org/reports/tr14/ http://www.unicode.org/reports/tr14/ 2. http://unicode.org/reports/tr29/ http://unicode.org/reports/tr29/ 3. http://site.icu-project.org/ http://site.icu-project.org/ 4. http://julialang.org/utf8proc/ http://julialang.org/utf8proc/ (Not Julia specific; it's used to implement Unicode support in Julia and is maintained as part of the Julia project. It's a very nice clean C library.)

jstimpfle 11y ago
And what can I do if I don't want to spend the next two months implementing this line breaking algorithm which probably no two implementations did in the same way? (exaggerating a little here). I just want to split my text into lines and fields, without second guessing if I did it right. And I want to be confident about interoperability. This means it has to be simple. If it's not simple then it's a sure thing that one of the communication partners screws up. I need a simple answer how I can compare two strings for certain properties, and not in a way which works in locale X but not in locale Y. I often don't want a thousand ways to express (almost) the same thing. This is for example quite a headache in practice when you search text. Worse if you search it in a pdf where you typically deal with ligatures and what not. I never said things are unspecified in Unicode. That's not my point. I think the things you and I want to do with text are just very very different.
- PeCaN 11y ago
  OK, wait, what? Of course two implementations will implement line breaking the same way, it's _specified_ how they should do it. That's the whole point of a spec. ICU and libutf8proc will both break lines for you. They exist specifically to provide implementations of the various Unicode algorithms. They'll also tell if you two character sequences can be considered identical. ICU even provides regexes where ft will match the ligature.
  - jstimpfle 11y ago
    There is no such one thing as "breaking lines" in Unicode. There are a thousand slightly different ways to do it, in every single locale. That is explicit in the introduction to the first beast you linked yourself... And assuming that a spec (especially a super-complex one, that moreover constantly evolves), is devoid of ambiguities, and makes every implementation behave in the same way, or even that every implementation implements the largest part of it, is being a tad idealistic. Sometimes it's all about common sense. For most tasks I will happily continue using strchr or my manual loop and be done with it, thanks.

by PeCaN 11y ago

jstimpfle 11y ago

And what can I do if I don't want to spend the next two months implementing this line breaking algorithm which probably no two implementations did in the same way? (exaggerating a little here). I just want to split my text into lines and fields, without second guessing if I did it right. And I want to be confident about interoperability. This means it has to be simple. If it's not simple then it's a sure thing that one of the communication partners screws up. I need a simple answer how I can compare two strings for certain properties, and not in a way which works in locale X but not in locale Y. I often don't want a thousand ways to express (almost) the same thing. This is for example quite a headache in practice when you search text. Worse if you search it in a pdf where you typically deal with ligatures and what not. I never said things are unspecified in Unicode. That's not my point. I think the things you and I want to do with text are just very very different.

PeCaN 11y ago
OK, wait, what? Of course two implementations will implement line breaking the same way, it's _specified_ how they should do it. That's the whole point of a spec. ICU and libutf8proc will both break lines for you. They exist specifically to provide implementations of the various Unicode algorithms. They'll also tell if you two character sequences can be considered identical. ICU even provides regexes where ft will match the ligature.
- jstimpfle 11y ago
  There is no such one thing as "breaking lines" in Unicode. There are a thousand slightly different ways to do it, in every single locale. That is explicit in the introduction to the first beast you linked yourself... And assuming that a spec (especially a super-complex one, that moreover constantly evolves), is devoid of ambiguities, and makes every implementation behave in the same way, or even that every implementation implements the largest part of it, is being a tad idealistic. Sometimes it's all about common sense. For most tasks I will happily continue using strchr or my manual loop and be done with it, thanks.