4 ms·

How will you graph it all or draw correlations ?

karlicoss 6y ago
For graphing: different data requires different representations, so I'm figuring out how to do it in my dashboard project: https://github.com/karlicoss/dashboard https://github.com/karlicoss/dashboard For dashboard, I started with health related data, because feel like other stuff (like 'histogram of my tweets' would be amusing, but not super useful immediately), and don't really expect any interesting quantified self insights from it. I'm gradually figuring out 'generic' ways to compute, plot and examine correlations, e.g. here https://github.com/karlicoss/dashboard/blob/708eb183130de31e344fa0f38fa8ae74d1953e31/src/dashboard/combined.py#L84-L101 https://github.com/karlicoss/dashboard/blob/708eb183130de31e... In general I'd like to have some automatic system which can consume all of my data and suggest interesting correlations in almost unsupervised way. (got more notes/possible implementations on it here https://beepb00p.xyz/exobrain/projects/dashboard.html#mtvtn https://beepb00p.xyz/exobrain/projects/dashboard.html#mtvtn )
- deleted 6y ago
  [deleted]
- johnwyles 6y ago
  I feel like the QS space needs a suite not unlike Home Assistant where you can plugin multiple integrations to these various services and begin collecting data relatively simply. Have you found anything in your travels that is mature enough to extend or spend time enhancing?
  - karlicoss 6y ago
    I still need to incorporate Home Assistant into my infrastructure, so not sure if I understand the analogy exactly. But basically I think the answer is 'no' -- there are so many different 'APIs' (quoted because often there aren't proper apis, you have to scrape, export manually etc.), that very few people attempted to unify it. I have some thoughts on how such system could work like, described here https://beepb00p.xyz/sad-infra.html#data_mirror https://beepb00p.xyz/sad-infra.html#data_mirror and here: https://beepb00p.xyz/exports.html https://beepb00p.xyz/exports.html . The export tools I wrote are the first order approximation of this vision -- seems robust enough for me so far, so now I'm working on connecting it to other tools and looking to cooperate with other people.
    - johnwyles 6y ago
      Home Assistant takes many disparate brands, services, devices, APIs and architectures for the world of IoT devices and yet they are able to somewhat successfully pull them together. I see no reason QS is any different and couldn't do the same even if it is scraping or exporting at intervals where an API does not exist. The analogy is different devices with different things they do, measure, record, and interact coming under a single umbrella of a QS service in a time series database (not unlike and maybe even exactly as is done with: Home Assistant + InfluxDB + Grafana).
      - karlicoss 6y ago
        Ah, yes, putting it into a single database is something I have in mind! Basically so far I was solving the 'data import/scraping' part, but now finally starting to connect it elsewhere now that it's somewhat unified and normalized.
- neolog 6y ago
  I think Altair would make these plots easier. https://altair-viz.github.io/ https://altair-viz.github.io/

For graphing: different data requires different representations, so I'm figuring out how to do it in my dashboard project: https://github.com/karlicoss/dashboard https://github.com/karlicoss/dashboard For dashboard, I started with health related data, because feel like other stuff (like 'histogram of my tweets' would be amusing, but not super useful immediately), and don't really expect any interesting quantified self insights from it. I'm gradually figuring out 'generic' ways to compute, plot and examine correlations, e.g. here https://github.com/karlicoss/dashboard/blob/708eb183130de31e344fa0f38fa8ae74d1953e31/src/dashboard/combined.py#L84-L101 https://github.com/karlicoss/dashboard/blob/708eb183130de31e... In general I'd like to have some automatic system which can consume all of my data and suggest interesting correlations in almost unsupervised way. (got more notes/possible implementations on it here https://beepb00p.xyz/exobrain/projects/dashboard.html#mtvtn https://beepb00p.xyz/exobrain/projects/dashboard.html#mtvtn )

deleted 6y ago
[deleted]
johnwyles 6y ago
I feel like the QS space needs a suite not unlike Home Assistant where you can plugin multiple integrations to these various services and begin collecting data relatively simply. Have you found anything in your travels that is mature enough to extend or spend time enhancing?
- karlicoss 6y ago
  I still need to incorporate Home Assistant into my infrastructure, so not sure if I understand the analogy exactly. But basically I think the answer is 'no' -- there are so many different 'APIs' (quoted because often there aren't proper apis, you have to scrape, export manually etc.), that very few people attempted to unify it. I have some thoughts on how such system could work like, described here https://beepb00p.xyz/sad-infra.html#data_mirror https://beepb00p.xyz/sad-infra.html#data_mirror and here: https://beepb00p.xyz/exports.html https://beepb00p.xyz/exports.html . The export tools I wrote are the first order approximation of this vision -- seems robust enough for me so far, so now I'm working on connecting it to other tools and looking to cooperate with other people.
  - johnwyles 6y ago
    Home Assistant takes many disparate brands, services, devices, APIs and architectures for the world of IoT devices and yet they are able to somewhat successfully pull them together. I see no reason QS is any different and couldn't do the same even if it is scraping or exporting at intervals where an API does not exist. The analogy is different devices with different things they do, measure, record, and interact coming under a single umbrella of a QS service in a time series database (not unlike and maybe even exactly as is done with: Home Assistant + InfluxDB + Grafana).
    - karlicoss 6y ago
      Ah, yes, putting it into a single database is something I have in mind! Basically so far I was solving the 'data import/scraping' part, but now finally starting to connect it elsewhere now that it's somewhat unified and normalized.
neolog 6y ago
I think Altair would make these plots easier. https://altair-viz.github.io/ https://altair-viz.github.io/