4 ms·

Would you have to do something similar to aggregate all my steps for the year (or miles running) for example?

minimaxir 9y ago
For my quick example, you'd have to change the key in the regex appropriately, and there may be other various metadata involved that has to be handled (i.e. whether the steps are from running or from walking). Again, there are 3rd party tools, but I do not want to use them. The actual data filtering/aggregation/visualization is a different story, and beyond the scope of getting the data. (here's why I was getting the heart rate data in the first place: https://twitter.com/minimaxir/status/949306256874913792 https://twitter.com/minimaxir/status/949306256874913792)
- killerdhmo 9y ago
  Thanks for your response! Even just an aggregation would be useful (outside of 3P apps). Congrats on your new gig! Looks a little more exciting than your previous one ;)
- mercutio2 9y ago
  Why are you using regular expressions to parse XML? Python has excellent XML libraries.
  - minimaxir 9y ago
    Because reading line-by-line is better than loading-and-parsing a 1 GB (in my case) XML file.
    - icebraining 9y ago
      Iterative parsing is your friend: http://boscoh.com/programming/reading-xml-serially.html http://boscoh.com/programming/reading-xml-serially.html There used to be a big argument between DOM and SAX parsers, but iterative/streaming parsers nowadays offer a pretty good compromise.
      - mercutio2 9y ago
        That link has some bizarre misconceptions about namespaces (just define the namespace URLs you’re interested in as constants, don’t depend on the particular short name/URL map you expect, as that guy does). It also looks like it predates cElementTree, which has an identical API but is vastly faster. But anyway, yes, iterparse will be much more reliable/flexible than a regular expression search of non-canonical XML data, and will likely be faster, too.
    - deleted 9y ago
      [deleted]
patmurraydev 9y ago
I've always had success with this web based [1] converter, it does all the conversion locally, but can take a while on a few years worth of data. [1]: http://ericwolter.com/projects/health-export.html http://ericwolter.com/projects/health-export.html

For my quick example, you'd have to change the key in the regex appropriately, and there may be other various metadata involved that has to be handled (i.e. whether the steps are from running or from walking). Again, there are 3rd party tools, but I do not want to use them. The actual data filtering/aggregation/visualization is a different story, and beyond the scope of getting the data. (here's why I was getting the heart rate data in the first place: https://twitter.com/minimaxir/status/949306256874913792 https://twitter.com/minimaxir/status/949306256874913792)

killerdhmo 9y ago
Thanks for your response! Even just an aggregation would be useful (outside of 3P apps). Congrats on your new gig! Looks a little more exciting than your previous one ;)
mercutio2 9y ago
Why are you using regular expressions to parse XML? Python has excellent XML libraries.
- minimaxir 9y ago
  Because reading line-by-line is better than loading-and-parsing a 1 GB (in my case) XML file.
  - icebraining 9y ago
    Iterative parsing is your friend: http://boscoh.com/programming/reading-xml-serially.html http://boscoh.com/programming/reading-xml-serially.html There used to be a big argument between DOM and SAX parsers, but iterative/streaming parsers nowadays offer a pretty good compromise.
    - mercutio2 9y ago
      That link has some bizarre misconceptions about namespaces (just define the namespace URLs you’re interested in as constants, don’t depend on the particular short name/URL map you expect, as that guy does). It also looks like it predates cElementTree, which has an identical API but is vastly faster. But anyway, yes, iterparse will be much more reliable/flexible than a regular expression search of non-canonical XML data, and will likely be faster, too.
  - deleted 9y ago
    [deleted]

patmurraydev 9y ago

I've always had success with this web based [1] converter, it does all the conversion locally, but can take a while on a few years worth of data. [1]: http://ericwolter.com/projects/health-export.html http://ericwolter.com/projects/health-export.html