It's been possible to skip tokenization for a long time, my team and I did it here - https://github.com/capitalone/DataProfiler https://github.com/capitalone/Da

It's been possible to skip tokenization for a long time, my team and I did it here - https://github.com/capitalone/DataProfiler https://github.com/capitalone/DataProfiler For what it's worth, we actually were working with LSTMs with nearly a billion params back in 2016-2017 area. Transformers made it far more effective to train and execute, but ultimately LSTMs are able to achieve similar results, though slow & require more training data.