2 ms·
Funny this is being posted as I just started to use specter for extracting entities from XML documents we receive. I never used it before but now I see the valu
by fmjrey 8y ago
Funny this is being posted as I just started to use specter for extracting entities from XML documents we receive. I never used it before but now I see the value of:
- paths as data
- navigators as transducers
First I started using plain clojure code, then played with using xml-in [1], then created my own transducers and channels to build a dataflow-like pipeline manually, and finally started using specter because I wanted paths to be a data structure I can use to build the pipeline programmatically instead of manually. For that I needed a way for paths to become tranducers (which specter traverse-all does [2]), and I also needed paths to be a sequence of literals or "interned" elements [3] such as keywords or symbols, but not functions created on the fly. That way I can create a tree from a sequence of paths, using something similar to this SO answer [4], and build the pipeline.
I'm almost there, I should end up in a situation where I can specify a list of paths to extract from an XML document, and the dataflow pipeline needed will be built automatically, with an input channel for the parsed XML, and output channels for each given path. If you use clojure.data.xml instead of clojure.xml, my hope is to have a one pass XML parsing of larger-than-RAM XML docs.
[1] https://github.com/tolitius/xml-in https://github.com/tolitius/xml-in
[2] https://github.com/nathanmarz/specter/wiki/List-of-Macros#traverse-all https://github.com/nathanmarz/specter/wiki/List-of-Macros#tr...
[3] http://www.yourdictionary.com/interning http://www.yourdictionary.com/interning
[4] https://stackoverflow.com/questions/49515858/clojure-convert-file-path-to-tree https://stackoverflow.com/questions/49515858/clojure-convert...