3 ms·
Bit of an annoying title, assumes tokenisation only means one thing. The toy compiler course I did explained that tokenisation was breaking a string into parts
by bennysomething 5y ago
Bit of an annoying title, assumes tokenisation only means one thing. The toy compiler course I did explained that tokenisation was breaking a string into parts "tokens". Probably other meanings too I'm guessing
- nodja 5y agoYup, thought it was about text tokenization, i.e. for nlp/machine learning purposes.
- gillesjacobs 5y agoYeah, synonyms exist. If you get annoyed by technical terms having different meanings in different fields, you are going to have a frustrating ride through life.
- benob 5y agoFor some time I though it was all about the security implications of strtok().
- ledgerdev 5y agoYep, overloaded term. Anyone got a better idea for what to call it in the context of sensitive data within applications?
- kafkaesque 5y agoWe call it PII (Personal Identifiable Information) at my workplace
- rmbyrro 5y agoHashing, perhaps. It's probably what they're doing in background.