3 ms·
> It's a toy problem with no real consequences. It's an extremely simplified but recognizable version of a real problem: natural language input tokenization, w
by tetromino_ 4y ago
> It's a toy problem with no real consequences.
It's an extremely simplified but recognizable version of a real problem: natural language input tokenization, which of course has real consequences.
- rimliu 4y agoThen this problem is solved by a gazillion of libraries already, no?
- reedf1 4y agoAh yes, I whack out four or five natural language input tokenizers before lunch most days.
- bluepizza 4y agoThis problem was solved by medieval monks by introducing white spaces.
- tetromino_ 4y agoMany languages still don't put whitespace between some or all words (Chinese, Japanese). Many languages tend to use long compound words which are better represented as sequences of tokens rather than one token (German, Russian). Many inputs are dirty (coming from OCR, for example) so whitespace can be unreliable. And even in clean English text, inflected words may be better tokenized as sequences of tokens rather than a single token.
- glitchc 4y agoThe problem there is that natural language processing is a highly specialized field, not something one can expect a generic SRE to know unless it's explicitly mentioned in the posting. I'm willing to bet the posting said nothing about NLP. I could ask a detailed question on cryptography, robotics or reverse-engineering, and it would be just as (ir)relevant to the SRE. Computer science is a much broader field than your niche area. Arbitrary goal-posts are arbitrary.