3 ms·

> we won’t even claim to tell you what the definition of “word” is! What's your best attempt? :-)

by throwup238 2y ago

flir 2y ago
\s\w+\s ;)
- eesmith 2y ago
  "Ain't ain't a word 'cause it don't match the regex." BTW, should be "\b\w+\b". The \b is a zero-width match for the start or end of a word. Your pattern requires a space before and after: >>> import re >>> re.compile(r"\b\w+\b").findall("What's the problem?") ['What', 's', 'the', 'problem'] >>> re.compile(r"\s\w+\s").findall("What's the problem?") [' the ']
  - flir 2y ago
    That makes sense. I was trying to translate "A word is a group of letters with whitespace on either side" into regex. I was being facetious, obviously, but you've already identified a serious problem with that definition (is "what's" two words or one?)
- koito17 2y ago
  cljs.user> (re-seq #"\s\w+\s" "やり直して") nil Joking aside, is it common to use regular expressions? Seems like the method only works for languages with spaces. I think a more sophisticated lexer may be necessary, but are there are non-regex, "fast approximations" that work across most languages? This is a problem that I have not tried solving before.
  - nine_k 2y ago
    すみません! It's because you have no space around. A more correct regexp would be \b\w+\b, with zero-width "word boundary" psttetbs instead of spaces.
    - Liquid_Fire 2y ago
      That's just passing the problem onto how you define \b. Since Japanese uses no spaces, it would match entire phrases or sentences as "words", treating only punctuation as word boundaries.
Xophmeister 2y ago
It's a bit like the old saying, "All models are wrong, some models are useful." The concept of a "word" is useful in everyday language, particularly in English -- it's host language, unsurprisingly enough -- and (probably) other Indo-European languages. However, in a more precise context, it breaks down because of edge cases. For example, "words" in agglutinative languages[1] (e.g., Turkish) act very differently from "words" in English. It's hard (impossible?) to capture all that variety in a pithy way. "A string of morphemes" might work, but that's hardly a satisfactory definition! Maybe a good analogy for the HN crowd would be like asking, "How many characters are in a string?" [1]: https://en.wikipedia.org/wiki/Agglutinative_language https://en.wikipedia.org/wiki/Agglutinative_language
- DiogenesKynikos 2y ago
  In Chinese, it's very difficult to define word boundaries. Each character is a syllable, with a particular pronunciation and a constellation of meanings, usually closely related to one another. There are very common combinations of characters that appear together, which one could define as words. However, often, you could just as easily view the individual characters as words, or the combination as a word. In some cases, the combination of two characters means something totally different from what the characters alone would mean (e.g., 东西, where the characters literally mean "East-West," but the combination means "thing"), so the combination is clearly a word. But sometimes, the meaning of the combination is basically a combination of the words' meanings (e.g., 吃饭, where the characters literally mean "eat-food," and the combination means "to eat, have a meal"). Because written Chinese doesn't use spaces, I guess it doesn't really matter what one defines as a word. The issue just doesn't come up, practically speaking.
  - tsimionescu 2y ago
    I should note that typically discussions of language are more fruitful around spoken (or signed) languages rather than writing. Writing is an artificial formal system, and as such sometimes has aspects which are much more socilogiclaly determined than spoken language, which tends to evolve more freely. Also, the problems you raise here are mostly just as applicable to English, though perhaps for somewhat fewer words. Is a "walkie-talkie" one word, or two? How about "unmarried"? The un- prefix has a distinct meaning on its own, even if it never appears alone, after all. Or how about "can't"? Technically it's a contraction of "can not", and those words do sometimes appear separately as well, even in this same meaning.
throw__away7391 2y ago
It depends on the architecture of your CPU.
nivertech 2y ago
There is no generally accepted definition in linguistics, but AI researchers have come to the consensus that a word is one or more LLM tokens ;) I’m not a linguist, but I would define a word as a part of a sentence composed out of one or more syllables, with word boundaries either implicitly or explicitly specified by different methods in different languages, e.g. by using pauses, longer or shorter phonemes, by using accents, rhythm, or intonation, or simply by remembering words as part of learning a lexical vocabulary. A word is something that can be categorized as to which part of speech it belongs (noun, verb, adjective, adverb, etc.) Depending on the languages it’s not always clear whether prefixes and/or suffixes are part of the word a separate words. Similarly with compound words - do they count as a single or multiple words? A short sentence in one language may enter another language as a single opaque word.

> we won’t even claim to tell you what the definition of “word” is! What's your best attempt? :-)

flir 2y ago

\s\w+\s ;)

eesmith 2y ago
"Ain't ain't a word 'cause it don't match the regex." BTW, should be "\b\w+\b". The \b is a zero-width match for the start or end of a word. Your pattern requires a space before and after: >>> import re >>> re.compile(r"\b\w+\b").findall("What's the problem?") ['What', 's', 'the', 'problem'] >>> re.compile(r"\s\w+\s").findall("What's the problem?") [' the ']
- flir 2y ago
  That makes sense. I was trying to translate "A word is a group of letters with whitespace on either side" into regex. I was being facetious, obviously, but you've already identified a serious problem with that definition (is "what's" two words or one?)
koito17 2y ago
cljs.user> (re-seq #"\s\w+\s" "やり直して") nil Joking aside, is it common to use regular expressions? Seems like the method only works for languages with spaces. I think a more sophisticated lexer may be necessary, but are there are non-regex, "fast approximations" that work across most languages? This is a problem that I have not tried solving before.
- nine_k 2y ago
  すみません! It's because you have no space around. A more correct regexp would be \b\w+\b, with zero-width "word boundary" psttetbs instead of spaces.
  - Liquid_Fire 2y ago
    That's just passing the problem onto how you define \b. Since Japanese uses no spaces, it would match entire phrases or sentences as "words", treating only punctuation as word boundaries.

Xophmeister 2y ago

It's a bit like the old saying, "All models are wrong, some models are useful." The concept of a "word" is useful in everyday language, particularly in English -- it's host language, unsurprisingly enough -- and (probably) other Indo-European languages. However, in a more precise context, it breaks down because of edge cases. For example, "words" in agglutinative languages[1] (e.g., Turkish) act very differently from "words" in English. It's hard (impossible?) to capture all that variety in a pithy way. "A string of morphemes" might work, but that's hardly a satisfactory definition! Maybe a good analogy for the HN crowd would be like asking, "How many characters are in a string?" [1]: https://en.wikipedia.org/wiki/Agglutinative_language https://en.wikipedia.org/wiki/Agglutinative_language

DiogenesKynikos 2y ago
In Chinese, it's very difficult to define word boundaries. Each character is a syllable, with a particular pronunciation and a constellation of meanings, usually closely related to one another. There are very common combinations of characters that appear together, which one could define as words. However, often, you could just as easily view the individual characters as words, or the combination as a word. In some cases, the combination of two characters means something totally different from what the characters alone would mean (e.g., 东西, where the characters literally mean "East-West," but the combination means "thing"), so the combination is clearly a word. But sometimes, the meaning of the combination is basically a combination of the words' meanings (e.g., 吃饭, where the characters literally mean "eat-food," and the combination means "to eat, have a meal"). Because written Chinese doesn't use spaces, I guess it doesn't really matter what one defines as a word. The issue just doesn't come up, practically speaking.
- tsimionescu 2y ago
  I should note that typically discussions of language are more fruitful around spoken (or signed) languages rather than writing. Writing is an artificial formal system, and as such sometimes has aspects which are much more socilogiclaly determined than spoken language, which tends to evolve more freely. Also, the problems you raise here are mostly just as applicable to English, though perhaps for somewhat fewer words. Is a "walkie-talkie" one word, or two? How about "unmarried"? The un- prefix has a distinct meaning on its own, even if it never appears alone, after all. Or how about "can't"? Technically it's a contraction of "can not", and those words do sometimes appear separately as well, even in this same meaning.

throw__away7391 2y ago

It depends on the architecture of your CPU.

nivertech 2y ago

There is no generally accepted definition in linguistics, but AI researchers have come to the consensus that a word is one or more LLM tokens ;) I’m not a linguist, but I would define a word as a part of a sentence composed out of one or more syllables, with word boundaries either implicitly or explicitly specified by different methods in different languages, e.g. by using pauses, longer or shorter phonemes, by using accents, rhythm, or intonation, or simply by remembering words as part of learning a lexical vocabulary. A word is something that can be categorized as to which part of speech it belongs (noun, verb, adjective, adverb, etc.) Depending on the languages it’s not always clear whether prefixes and/or suffixes are part of the word a separate words. Similarly with compound words - do they count as a single or multiple words? A short sentence in one language may enter another language as a single opaque word.