3 ms·
FWIW, a tokenizer is not really necessary. It (arguably) makes things cleaner / easier, but you can also just write a parser that operates directly on a byte st
by dilap 4y ago
FWIW, a tokenizer is not really necessary. It (arguably) makes things cleaner / easier, but you can also just write a parser that operates directly on a byte stream, if you feel so inclined, and it turns out fine.
- kccqzy 4y agoYes I agree. The article actually says the purpose is > To separate the logic that handles a language's microsyntax from the logic that handles regular syntax. But everything is just syntax. This distinction between microsyntax and regular syntax is really just for convenience; and sometimes it's not that much convenience.
- Waterluvian 4y agoAnd isn’t this developer convenience at the cost of performance in a piece of software where performance is very important? Today as my “end of the week fun experiment” I replaced Babel with esbuild on a years old 50k sloc production web application. It took 30 mins and builds now take 500ms instead of 63,000. Not quite apples to apples yet. But it’s such a compelling change that I can’t ignore it. I’m excited to see how Babel steps it up in the years ahead.
- dilap 4y agoShouldn't be any noticable perf impact of splitting out tokenization as a separate stage, I think.