A lot of systems based on gpt-like LLM's are currently being limited by the stripping of certain characters during tokenization... if the linked source used a s

A lot of systems based on gpt-like LLM's are currently being limited by the stripping of certain characters during tokenization... if the linked source used a superscript exponent, that superscript probably got lost and interpreted as just a concatenated number. This is one of the large problems I see right now hindering how these models are trained- they crawl text with numerical figures and graphs, charts, etc on webpages that cannot be interpreted as text, so the model learns something, but misses context or, in this bad case, loses a very key element of the mathematical notation. It's a hard problem, because it isn't like every webpage with facts and figures is written in perfectly formed LaTeX that could be interpreted. 1.67x10^34, 1.67*10^34, 1.67E34, 1.67*10**34 plus any number of mathjax, LaTeX, non-ASCII charsets, etc are all ways that would need to be interpreted correctly by the preprocessor for training, the input tokenization, and represented in the charset of the output of the model for it to be more effective.