5 ms·

Misses the point, I think? There are plenty of times you must alter human text. For example, a word processor with a find/replace function must be able to repla

by RyanZAG 13y ago

Misses the point, I think? There are plenty of times you must alter human text. For example, a word processor with a find/replace function must be able to replace the correct text and not move extra symbols around while doing so. There are thousands of similar examples.

rbehrends 13y ago
Yes, but functionality that is about dealing with text for human consumption is properly the function of a library, not the core language, or even a single type in the standard library. Human language and culture is too complex, fuzzy, and variable to hardwire its rules into a programming language specification. Character boundaries and character transformations are really only the beginning of it. Consider finding the end of an English word. Is an apostrophe at the end an actual apostrophe denoting the possessive case (of a plural word) or meant as a closing single quote (even where there are different symbols, they are often mixed up when doing data entry)? How do you pluralize words? You have to consider exceptions to the usual rules (e.g., a dictionary), words that only exist in the plural, languages that don't have the concept of plural words, etc.
- informatimago 13y ago
  Nonetheless, that's what Wolfram is trying to do with his "programming" language. At least it's at the same level. Provide human level rules and knowledge embodied in a computer programming language.
nabla9 13y ago
The author argues that when string processing becomes complicated, it does not matter and incorrect results are usually OK. Everybody knows about Bobby Tables? How about Mr. เจ้าพระยาบดินทรเดชา Unicode? What if somebody told you that if certain exotic names are entered into the system by certain input method, government officials can't find those people in their searches, including full text search to a police report?
- bcoates 13y ago
  SQL injection is caused by exactly the confusion between human- and machine- readable text that the author is warning against.
skrebbel 13y ago
Actually, one of the points that I tried to make is that the amount of situations where you have to do that is remarkably small. You don't code a word processor every day. And, if you do need to do that, and you need to support more than a single language, you're talking about a rather complex, probably slightly fuzzy, major feature. I bet the MS Word team that made internationalized search/replace spent a lot of time getting it even somewhat right.
lelf 13y ago
> There are thousands of similar examples Not actually
- RyanZAG 13y ago
  Text search, text replacement, ellipsizing text at the end or middle (as done commonly on iOS), fuzzy search, password word limits, password length limits, matching passwords (try using bcrypt on international languages without first normalizing), detecting and filtering rude words in public chat, ... Basically anywhere that text needs to be checked or changed. Just off the top of my head. There are far more. You can't even have a native language user friendly login system without normalizing text.
  - lelf 13y ago
    Text search, search, that counts text replacement, why? ellipsizing text at the end or middle (as done commonly on iOS), you need to know width for that, and you don't, bad idea fuzzy search, search password word limits, password length limits, matching passwords (try using bcrypt on international languages without first normalising), i18n password? very bad idea detecting and filtering rude words in public chat search&replace, anyway you won't be able to, because f͌̊ͦuͥcͥ͑ͩ͛k͂͌̿ͣ̾
    - deathanatos 13y ago
      > you need to know width for that, and you don't, bad idea You also need to not chop off accents. "café" cannot become "cafe…" > i18n password? very bad idea Because string types are so broken.
maxk42 13y ago
Entirely misses the point. The original document was discussing the problem of text normalization, which still remains. If I have a database of employee bios and I write an app so HR can search that database, it's no problem for them to type in "management" and get a list of all employees with the word "management" in their bio. But when a division does the same thing in a non-English-speaking country where there's an é in the word, then that word can be composed of either two characters or one and searching one way will tend to miss any results which were composed of text the other way. The solution here is to normalize the data before it is stored in the DB and the search term before it is searched for. It's a simple concept. Not all programmers are aware of it, however, and this guy totally missed the point.

by RyanZAG 13y ago

rbehrends 13y ago

Yes, but functionality that is about dealing with text for human consumption is properly the function of a library, not the core language, or even a single type in the standard library. Human language and culture is too complex, fuzzy, and variable to hardwire its rules into a programming language specification. Character boundaries and character transformations are really only the beginning of it. Consider finding the end of an English word. Is an apostrophe at the end an actual apostrophe denoting the possessive case (of a plural word) or meant as a closing single quote (even where there are different symbols, they are often mixed up when doing data entry)? How do you pluralize words? You have to consider exceptions to the usual rules (e.g., a dictionary), words that only exist in the plural, languages that don't have the concept of plural words, etc.

informatimago 13y ago
Nonetheless, that's what Wolfram is trying to do with his "programming" language. At least it's at the same level. Provide human level rules and knowledge embodied in a computer programming language.

nabla9 13y ago

The author argues that when string processing becomes complicated, it does not matter and incorrect results are usually OK. Everybody knows about Bobby Tables? How about Mr. เจ้าพระยาบดินทรเดชา Unicode? What if somebody told you that if certain exotic names are entered into the system by certain input method, government officials can't find those people in their searches, including full text search to a police report?

bcoates 13y ago
SQL injection is caused by exactly the confusion between human- and machine- readable text that the author is warning against.

skrebbel 13y ago

Actually, one of the points that I tried to make is that the amount of situations where you have to do that is remarkably small. You don't code a word processor every day. And, if you do need to do that, and you need to support more than a single language, you're talking about a rather complex, probably slightly fuzzy, major feature. I bet the MS Word team that made internationalized search/replace spent a lot of time getting it even somewhat right.

lelf 13y ago

> There are thousands of similar examples Not actually

RyanZAG 13y ago
Text search, text replacement, ellipsizing text at the end or middle (as done commonly on iOS), fuzzy search, password word limits, password length limits, matching passwords (try using bcrypt on international languages without first normalizing), detecting and filtering rude words in public chat, ... Basically anywhere that text needs to be checked or changed. Just off the top of my head. There are far more. You can't even have a native language user friendly login system without normalizing text.
- lelf 13y ago
  Text search, search, that counts text replacement, why? ellipsizing text at the end or middle (as done commonly on iOS), you need to know width for that, and you don't, bad idea fuzzy search, search password word limits, password length limits, matching passwords (try using bcrypt on international languages without first normalising), i18n password? very bad idea detecting and filtering rude words in public chat search&replace, anyway you won't be able to, because f͌̊ͦuͥcͥ͑ͩ͛k͂͌̿ͣ̾
  - deathanatos 13y ago
    > you need to know width for that, and you don't, bad idea You also need to not chop off accents. "café" cannot become "cafe…" > i18n password? very bad idea Because string types are so broken.

maxk42 13y ago

Entirely misses the point. The original document was discussing the problem of text normalization, which still remains. If I have a database of employee bios and I write an app so HR can search that database, it's no problem for them to type in "management" and get a list of all employees with the word "management" in their bio. But when a division does the same thing in a non-English-speaking country where there's an é in the word, then that word can be composed of either two characters or one and searching one way will tend to miss any results which were composed of text the other way. The solution here is to normalize the data before it is stored in the DB and the search term before it is searched for. It's a simple concept. Not all programmers are aware of it, however, and this guy totally missed the point.