3 ms·
I love this type of optimisation story. Interesting to note that his "in theory" best performance still calls to lxml. A true in theory best performance would
by kristianp 2y ago
I love this type of optimisation story.
Interesting to note that his "in theory" best performance still calls to lxml. A true in theory best performance would surely be in the multiple Gigabyte per second range (if simdjson's performance is taken as a goal), and not 0.56 seconds to process 230 MiB which is only about half a GBps. Half a gigabyte per second is probably more than sufficient for most uses, so I'm splitting hairs somewhat.
Here is the issue for slow sax parsing in encoding/xml. No work on it since around 2020:
https://github.com/golang/go/issues/21823 https://github.com/golang/go/issues/21823
- masklinn 2y ago> A true in theory best performance would surely be in the multiple Gigabyte per second range (if simdjson's performance is taken as a goal) I don't think it makes any sense for the parser for a completely different and much simpler format to be "taken as a goal".
- kristianp 2y agoPerhaps "goal" is the wrong word, I just mean the "in theory best performance". It doesn't matter that json is a different format, they both have the same worst case complexity for the particular example problem.
- mananaysiempre 2y agoFull XML is hard if you’re going to go nuts with entity declarations and such (I’ve heard SVG output from Adobe tools is fond of those for some reason?), but the part that people commonly use is not really that much more complicated than JSON; perhaps even simpler in some places, given the absence of backslash escapes (that the simdjson paper spends quite a lot of time on). So for the fast path I think it’d be reasonable to expect JSON-like performance.