3 ms·

> TI-BASIC programs are stored as tokens, not text: every command, function, and variable is a token of 1 or 2 bytes. The OS detokenizes (token→display string)

> TI-BASIC programs are stored as tokens, not text: every command, function, and variable is a token of 1 or 2 bytes. The OS detokenizes (token→display string) to show a program and tokenizes (keypress/text→token) on entry; the parser walks tokens to execute. From my memory of using a TI-83 in the late 90s, I would not be surprised if the keypad UI injects tokens directly based on your keypress, rather than "tokenizing the text". I seem to recall, for example, you could not position the cursor in the middle of a BASIC token, and if you managed to type out the tokens it would not work; you needed to find the right menu item to inject the correct token.

duskwuff 5mo ago
I can confirm that. On the TI-83, many of the TI-BASIC tokens contained lowercase characters which couldn't be typed at all - you could only type uppercase letters on the keyboard. (There were a few lowercase letters available as tokens for special purposes, but it wasn't a full set.) Interestingly, you could print tokens in strings - e.g. you could Disp "Disp ".
- suburban_strike 5mo ago
  The 83+ let you type the full set of lowerchase chars as well, but they used 2x as many bytes per character for storage.
- 7jjjjjjj 5mo ago
  There's actually a hidden lowercase feature, you can use an assembly program to enable it.
siraben 5mo ago
Yes, to type a TI-BASIC program you have to go through the calculator menus which directly insert the tokenized input into the buffer. The weird thing about TI-BASIC is how seemingly innocent changes in the input can cause huge performance regressions e.g. https://siraben.github.io/ti84p-re/sub-tibasic-for-paren.html https://siraben.github.io/ti84p-re/sub-tibasic-for-paren.htm... For(I,1,N If 0 1 End is much slower than For(I,1,N) If 0 1 End
- asveikau 5mo ago
  The open paren being part of the tokens was always weird. I could imagine that doing strange things for the parser; when it sees a close paren it needs to know that several of the preceding tokens may have an open paren even without having a '(' token.
jamesfinlayson 5mo ago
Ah makes sense. I remember a younger me trying to open .8xp files back in the day and seeing gibberish, and eventually finding the TI IDE which... felt like it had been written a long time ago (the file select dialog capped the display of file names at 8.3 I think and used ~1 and ~2 etc as "the rest of the file name").

duskwuff 5mo ago

I can confirm that. On the TI-83, many of the TI-BASIC tokens contained lowercase characters which couldn't be typed at all - you could only type uppercase letters on the keyboard. (There were a few lowercase letters available as tokens for special purposes, but it wasn't a full set.) Interestingly, you could print tokens in strings - e.g. you could Disp "Disp ".

suburban_strike 5mo ago
The 83+ let you type the full set of lowerchase chars as well, but they used 2x as many bytes per character for storage.
7jjjjjjj 5mo ago
There's actually a hidden lowercase feature, you can use an assembly program to enable it.

siraben 5mo ago

Yes, to type a TI-BASIC program you have to go through the calculator menus which directly insert the tokenized input into the buffer. The weird thing about TI-BASIC is how seemingly innocent changes in the input can cause huge performance regressions e.g. https://siraben.github.io/ti84p-re/sub-tibasic-for-paren.html https://siraben.github.io/ti84p-re/sub-tibasic-for-paren.htm... For(I,1,N If 0 1 End is much slower than For(I,1,N) If 0 1 End

asveikau 5mo ago
The open paren being part of the tokens was always weird. I could imagine that doing strange things for the parser; when it sees a close paren it needs to know that several of the preceding tokens may have an open paren even without having a '(' token.

jamesfinlayson 5mo ago

Ah makes sense. I remember a younger me trying to open .8xp files back in the day and seeing gibberish, and eventually finding the TI IDE which... felt like it had been written a long time ago (the file select dialog capped the display of file names at 8.3 I think and used ~1 and ~2 etc as "the rest of the file name").