4 ms·
Can you ELICSUndergraduate. Tokenizing is normally for if you're writing a compiler of DSL right?
by domnomnom 6y ago
Can you ELICSUndergraduate. Tokenizing is normally for if you're writing a compiler of DSL right?
- TeMPOraL 6y agoYes. But any data format you read, particularly any plaintext data format you read, is essentially interpreting or compiling a DSL. On a typical job, people are writing compilers much more often than they think!
- orangeshark 6y agoIt is a general term for the process of breaking a string into "tokens" which have a sort of meaning. Definitely a common task in compilers, but not limited to it.