I don't see the advantage neither. Since tokenization is usually sequential, it only seems to add overhead. If you can identify chunks in your input at a much l

I don't see the advantage neither. Since tokenization is usually sequential, it only seems to add overhead. If you can identify chunks in your input at a much lower cost than actually parsing that chunk, it might be of use.