4 ms·
Having been there, there exists an intermediate step of tokenized code. So you store ascii strings as .. ascii but as you enter source code a "then" as in if/t
by VLM 13y ago
Having been there, there exists an intermediate step of tokenized code. So you store ascii strings as .. ascii but as you enter source code a "then" as in if/then gets tokenized into hex 0xD6 or something. So the poor CPU doesn't have to run a full lexer at runtime to see if the "t" belongs to "to" or "then" it just matches hex 0xd6 which is much faster. This works real well if you have 128 (or so) or less tokens in your language. This can also save a huge amount of memory, depending on your coding style I suppose.
Tokenization also allows some syntax error detection to occur as you type code in, which was interesting. I don't remember enough about this. Obviously some mistakes won't tokenize at all or will tokenize into gibberish.
So tiny basic in memory stored plain old ascii and saved plain old ascii to cassette tape. lvl2 msbasic stored tokens in memory although it could optionally save pure ascii to cassette tape. This had some interesting software distribution issues and compatibility issues as it was sorta kinda half way possible to save something on lvl1 and load it into lvl2 if you were careful and vice versa.