3 ms·
Interesting. I had been thinking recently about grep-friendly structured text file formats given the constraints of regex. But I hadn't considered that you coul
by btrettel 9mo ago
Interesting. I had been thinking recently about grep-friendly structured text file formats given the constraints of regex. But I hadn't considered that you could design a structured text file format to be LLM-friendly given token constraints.
- tsazan 9mo agoYou're right.If a format is easy to grep, it is almost always cheap to tokenize. We treat token density as a primary design constraint.