9 ms·

What data on myself I collect and why (2020)

187 points by karlicoss 6y ago

ramraj07 6y ago
I want to do something similar, and load it in a place like snowflake. I say "like" snowflake because its annoying that they have a $25 a month minimum, otherwise it's already nicely suited to be a personal data warehouse!
- benmanns 6y ago
  BigQuery charges per query and for storage which might work for you. Where did you see the $25 minimum for Snowflake?
  - harporoeder 6y ago
    BigQuery or Snowflake seem rather extreme for anything I might consider personal data. Even logging as many things as in the article a local sqlitedb and your preferred backup solution would get pretty far. It might be easier to setup nice dashboards with a cloud product though.
    - throwaway3699 6y ago
      I can see the appeal. Those services might be designed for petabytes of data analysis, but the flip side is that you have something ready to go with little maintenance.
    - ramraj07 6y ago
      I want the db to be in the cloud. I'm not on my laptop most of my time and I'm definitely not on the same laptop more than 20% of my time. Hosting any sqlite db or postgres db on the cloud means I pay for hot storage which is significantly expensive if I have 200gb of data say. (Which I do; I have a ton of data from my lab days that also I want queryable like this; I also intend to purchase some datasets that I think might be fun to keep at hand so)
  - ramraj07 6y ago
    Forgot about BigQuery, will have to give it a shot! The Snowflakes minimum Is something my colleague got from them when we set up an account for a startup idea.
- simonw 6y ago
  This is one of the reasons I've been building my version of this on top of SQLite: it's incredibly cheap. All you need is a writable disk somewhere. I started with a $5/month VPS.
  - ramraj07 6y ago
    How much data are you storing? How much is 200gb of storage? At least on lightsail it wasn't cheap! Also that's not even backed up!
    - simonw 6y ago
      Only about 20GB, so it's pretty inexpensive. I'm not actually bothering to run backups because theoretically ALL of the data there can be retrieved from other sources - pulled back out of the Twitter archive exports for example. But another similar project uses tarsnap for backups, which is pretty inexpensive.
- edmundsauto 6y ago
  You can pretty easily store most of these data in any database. Don't need something like Snowflake - postgres stock can handle about any analytics queries, especially if you're not writing too much.
  - ramraj07 6y ago
    So the issue is not paying for compute when you're not using it. Firstly I've tried running postgres on a low powered lightsail instance and while it runs, it's kinda insane that it takes a ton of time to load data and then to query it. Indexless scans of data in large tables (let's say a db with all my emails) can take a while! I've tried the same in snowflake, and I'm only using their smallest warehouse, and everything happens in seconds. Loading GBS of data is seconds, scanning GBs of data is seconds. Their smallest compute warehouse is still significantly beefier than the cpu you get with a puny lightsail with a shared vcpu and while it's disk io is shit (since it's reading from S3) the parallelism makes up for it. In the end I think it's the pricing model that makes the difference. With snowflake you seamlessly launch a very powerful machine to run your query and only get billed for a minute, and with running postgres you need to run your machine all the time. Also hot storage (as opposed to s3) is also non trivially expensive. Haven't seen any way to get 200gb of ssd storage without spending that 25 dollar minimum that snowflake costs anyway. This is data I intend to not even update or query more than a few times a month, so I really don't want to pay for compute running all the time in its name.
    - edmundsauto 6y ago
      That's a fair point - the 200GB disk costs a bit more if you need an SSD (although my experience shows an SSD is not super necessary unless you have write heavy workloads or otherwise unusual access patterns). If you drop the SSD requirement, a cheap dedicated server will blow Snowflake out of the water. As a bonus, you can run your own code (ETL, scrapers, dataviz). Kimsufi has 4 core 2TB (spinning) w/ 16GB RAM for $17/mo. Personally, that is the route I go for my personal warehouse, as I find I almost always want to put an API or django app in front (and other software like Celery, scrapers, etc.)
t0ughcritic 6y ago
How will you graph it all or draw correlations ?
- karlicoss 6y ago
  For graphing: different data requires different representations, so I'm figuring out how to do it in my dashboard project: https://github.com/karlicoss/dashboard https://github.com/karlicoss/dashboard For dashboard, I started with health related data, because feel like other stuff (like 'histogram of my tweets' would be amusing, but not super useful immediately), and don't really expect any interesting quantified self insights from it. I'm gradually figuring out 'generic' ways to compute, plot and examine correlations, e.g. here https://github.com/karlicoss/dashboard/blob/708eb183130de31e344fa0f38fa8ae74d1953e31/src/dashboard/combined.py#L84-L101 https://github.com/karlicoss/dashboard/blob/708eb183130de31e... In general I'd like to have some automatic system which can consume all of my data and suggest interesting correlations in almost unsupervised way. (got more notes/possible implementations on it here https://beepb00p.xyz/exobrain/projects/dashboard.html#mtvtn https://beepb00p.xyz/exobrain/projects/dashboard.html#mtvtn )
  - deleted 6y ago
    [deleted]
  - johnwyles 6y ago
    I feel like the QS space needs a suite not unlike Home Assistant where you can plugin multiple integrations to these various services and begin collecting data relatively simply. Have you found anything in your travels that is mature enough to extend or spend time enhancing?
    - karlicoss 6y ago
      I still need to incorporate Home Assistant into my infrastructure, so not sure if I understand the analogy exactly. But basically I think the answer is 'no' -- there are so many different 'APIs' (quoted because often there aren't proper apis, you have to scrape, export manually etc.), that very few people attempted to unify it. I have some thoughts on how such system could work like, described here https://beepb00p.xyz/sad-infra.html#data_mirror https://beepb00p.xyz/sad-infra.html#data_mirror and here: https://beepb00p.xyz/exports.html https://beepb00p.xyz/exports.html . The export tools I wrote are the first order approximation of this vision -- seems robust enough for me so far, so now I'm working on connecting it to other tools and looking to cooperate with other people.
1996 6y ago
> How: many shells support keeping timestamps along your commands in history. > E.g. "Remember all your bash history forever". It would be far more important to get $? to get the rate of error of the commands you type (ex: typo) and use that as an indicator of poor mental acuity (ex: bad sleep, stress, ...)
- karlicoss 6y ago
  Interesting idea! I suspect I'd need to use shell far more often than I do currently to get anything conclusive though. But I guess it's possible to infer error rate even from past data by looking at closely spaced commands wish short Levenshtein distance.
  - 1996 6y ago
    It depends on your workflow. I'd say your approach is interesting, but it's mostly driven by availability of the data ("searching under a streetlight") except for your sleep monitor (however, being an under pillow solution, it's not essentially different from the accelerometer data from a smart watch) I would suggest you invert the logic: design objective (ex: mental acuity), conclude on what things you want to track (ex: time slept, cups of coffee per day, time you drink it, errors in shell) and get the data accordingly, spending as necessary (ex: smart coffee maker? or maybe a smart plug that can let you infer when the machine is being used from the wattage drawn) As for data tricks, L distance is a good one, yet not applicable for shell, as it's sensitive to the string length so you would need correction. Also, it's missing the essential metric: did the command work? Only the error return code will give you that. TLDR: think about what you want, but before that think about why you want it. Collecting useless metrics is another form of data hoarding.
    - karlicoss 6y ago
      Yeah, it's a good point, and I'm indeed a bit guilty of hoarding data I can't immediately process. However, the problem with the 'objective first' approach is that it's gonna require a lot of data to draw meaningful rigorous conclusions from small interventions, so I'm making sure to 'secure' the data first, and then gradually process it. But also, it's a challenge in itself -- I'm automating a lot, sharing my system and trying to interoperate with existing tools, in the hope that my work can be useful to other people and make quantified self easier for them.
uaas 6y ago
Have you maybe compared Emfit QS to a customer-grade tracker like Fitbit directly?
- karlicoss 6y ago
  Yeah, I also have a Garmin watch, so wanted to write a script to properly compare the data for a while. From the few spot checks (e.g. actually comparing 20 select days against my manual records) - heart rate and respiration rate match pretty closely (kind of expected I guess) - Emfit is noticeably more accurate at detecting asleep time/awake time. But even so, sometimes emfit doesn't detect that you've woken up and just lying in bed with the phone. - 'sleep phases' at a glance don't match well. Definitely need a comparison script to do any stronger claims. In addition, I'm still not sure if sleep phases actually mean anything at all (for my body, anyway), need to do more analysis about it. - Emfit detects HRV, which doesn't seem to correlate with any of my subjective feelings. However, it does seem to change after exercise, for example.
hluska 6y ago
I got heavy into tracking personal data after spending a week in a cardiac unit in July 2017. My own tracking efforts have lead me to the absolute best physical shape I have ever been in. My weight went from 340 pounds (hence the cardiac unit) to 160. I have gotten into ultramarathons and lately, triathlons. Strength wise, I managed to drop 180 pounds while increasing my strength. It is amazing what the mind can do if you prime it with the right kinds of metrics.
- elorant 6y ago
  Care to describe what devices/programs you're using?
  - hluska 6y ago
    Sure, I would love to but I'll warn you that this sounds a little fucked up (because it is). I was writing code and was two days from launching a startup when my heart went to hell. I called our local Healthline and was told to call 911 because "you are having a heart attack" but I'm a moron so decided to drive myself to the hospital. (I want to start with that story because I got myself into that mess by being a moron and figure that it's important you don't think this came from a place of intellect - it was 99% fear, trauma and trying to overcome the stupid defaults that got me into that mess in the first place.) The first day that I got out of the hospital, I had to move back in with my Mom. I decided to go out for a walk and was in such a bad place that two little old ladies (one of whom had a walker) actually smoked me on the path back to my Mom's area. I was remarkably close to killing myself after that experience and in all truth, if it wasn't for my daughter (she was 18 months old at the time), I would not be here to write this. My GP referred me to a psychiatrist. My psychiatrist is an amazing man and an incredible athlete. He suggested two tools that could help me with my cardio recovery but that have also been extraordinarily useful for my mental health. He recommended a Masimo fingertip oximeter and a training band by Polar. The Polar training band is/was amazing. My first Polar was an H7 with an XXL band. Now I use an H10 with a small band. I'm not sure that I would choose the Polar again if I was starting out now - at the time, my psychiatrist suggested it because I could find a band that would fit in between my massive gut and the band of fat around my chest that used to lead coworkers to tease me about my bra size (I was assigned male at birth and continue to identify as male so that was not complimentary. It was funny at the time but quite sad to think about now.) Program wise is where this story starts to get a little bit fucked up. Remember the first paragraph where I told you that I was writing code when my heart went to hell? Well, that really fucked me up and I quite literally could not write code or even sit in front of a computer. Computers have been my biggest love and practically my sole hobby since I was a 7 year old with a bad stutter. Suddenly, that was replaced with the most unbelievable fear whenever I would open up a text editor or even sit down at a keyboard. Consequently, at first I started using mobile apps and ran everything through my iPhone. But as I started to recover and as my psych meds, therapy and other psychiatric help started to kick in a little more, my brain started to come back. I have a marketing degree and really like data so as my brain started to come back, I started noticing correlations between cardiac metrics and purchasing activity. That was strange. In a sense, my capitalism came back and I saw an opportunity. But, at the same time, I was a patient and doing cardiac rehab/support with ten older adults from my Mom's community. This sounds weird to anyone with a healthy heart but I'm a firm believer that whoever coined the term 'a broken heart' had had a cardiac intervention. It feels exactly the same and I had this group of older adults who cared enough about me to check in on me, come by and make me go for walks or even just sit and watch baseball with me. Consequently, I was never able to turn that opportunity into a business, but it gave me an idea about some code to write. So, I started writing my own fitness tracker. At some point I might release it because it's been damned good for me. The first version is unbelievably sad and I have one hell of a lot of trouble reading that code over now. It was somewhere between a suicide note and a eulogy for my former self but that minimizes how brutally sad some of my variable and method names were. My favourite example is fatFuck.name = Greg. That project turned into rehab and through the years I have kept adding to it. At first, it was purely about rehabbing my brain so I could write code again and maybe graduate to a point where I could comfortably refer to myself as user.name and heck, maybe even remove the word 'dead' from all my method names. But as it has grown, it has turned into a really good tool for teaching myself how to think in terms of healthy defaults. Here is where stuff gets really fucked up sounding so if you've made it this far, good for you! I don't know if the app I wrote has actually helped me or if I hacked myself. I am deeply obsessive, to the point that when my psychiatrist did my OCD screening, I went home and rewrote my resume because the screening materials described my talents better than anything else I had ever written. When I build anything, I test it obsessively. In this case, to test it, I had to live it and 3.5 years later, I have a six pack, no love handles, two marathons down and an ultra in my imminent future. Tracking nutrition against performance metrics yielded the most powerful results of all. At first, I started by writing down what I ate and then tracking my macros. I learned that what I thought was 2000 calories was actually closer to 4000 calories. As I started paying more attention to macros like protein and carbohydrates, I found a great feedback loop where eating healthier foods resulted in improved athletic performance. As my athleticism increased, I was able to work harder and then started tracking my moods against that data. That created another feedback loop where the better I ate, the harder I could exercise and the better I would feel. At this point, 3.5 years later, I actually have healthy defaults. It sounds insane, but I genuinely have healthy defaults. The way I eat, the foods I eat and even the textures I enjoy have completely shifted around. The only downside at this point is that I'm rather skeptical of most of the wearable industry. I've seen way too many metrics that make my capitalism tingle in funny ways. I know that I could enhance tracking with other tools but at this point, I'm pretty obsessed with reading their privacy policies and speaking as a patient instead of an entrepreneur, I don't think the marketing industry needs (or deserves) this type of data. I feel like I've missed a lot. At some point, I'll put everything together into written form and do a better job of this reply. Sorry if this was disjointed!!
lifeisstillgood 6y ago
This is ... the future. I am going to start off with a few this week - well done and good luck !
WeepinStolzham 6y ago
Never go Full Stephen Wolfram.
WephenSolfram 6y ago
Never go full Stephen Wolfram.
simonw 6y ago
Have you considered loading this stuff into SQLite in addition to keeping the raw exports in files on disk? Being able to query your personal data with SQL can get really interesting. I've been using it for my own version of a personal data warehouse, described here: https://simonwillison.net/2020/Nov/14/personal-data-warehouses/ https://simonwillison.net/2020/Nov/14/personal-data-warehous...
- karlicoss 6y ago
  Yep! In fact I've tried interoperating with Datasette (e.g. shared here https://news.ycombinator.com/item?id=25090643 https://news.ycombinator.com/item?id=25090643 ) One secret sauce is using 'automatic' caching of data in sqlite -- this allows both for faster access and having an additional interface for the data as a collateral https://github.com/karlicoss/cachew#readme https://github.com/karlicoss/cachew#readme Still need to polish this a bit, but ultimately hoping to properly plug into Datasette, I'm impressed by its data exploration capabilities!
andrewnc 6y ago
I love the idea of personal data analysis. In fact, I recently wrote a book with that premise called "Everyday Data Science". I just wrote about it in another thread, but it seems relevant to the discussion here. I definitely agree that you can really take control of your life with a bit of data and some not so fancy analysis. In the book I shared the story of an old professor of mine that tracked a personal health marker from yearly screenings. The threshold for concern was "4 ng/ml". His numbers were always right around 1. However, there was a super noticeable trend up to 2.5 and 3. The doctors all waved it off because it was still below the magic "4" number. However, when he showed them the graph of historic data, they rushed him to a specialist and detected (and removed) cancer. I think data collection and personal analysis for day-to-day stuff is vital and fascinating. Thanks for sharing :)
- karlicoss 6y ago
  Thanks! Yep, one of the motivations for me is learning & establishing health baseline. For example, with blood tests it's unlikely that I'll be able to infer any interesting correlations (at the very least because you can't do them every day, with the exception of glucose/ketones perhaps). However I'll know what is 'normal' for my body and if something breaks, hopefully will have some clues.
friggeri 6y ago
Over the past few years I’ve thought about building something similar to Home Assistant, but for human centered data and APIs vs home centered. Essentially a shared bus on which different data sources could be plugged into a standardized format, on which people could build dashboards and automations. Never got around to doing so, because sidetracked by other projects, but if others are interested in teaming up I’d love to work on something like that (email in profile if anyone wants to start this)
tomaszs 6y ago
What we need is a law that obligates companies to enable scheduled auto export to a given URL https://github.com/tomaszs/RightToBeRemembered/blob/main/README.md https://github.com/tomaszs/RightToBeRemembered/blob/main/REA...
fjfaase 6y ago
In that past ten years, I have been recording my daily activities in Moleskine daily planners. I am using the small size daily planner with one page per day. I always carries it with me. The main motivation to start doing it, was to fight the feeling that time is going faster and faster. It takes me about a quarter of an hour per day. I do not record every things, just the things I think a worth recording. I do track all of my movements. I have thought about recording everything in a database, but never come to it. For some time, I have been tracking my travels with Google Earth. The last year, I have started doing this again for for walking and biking 'trips' I make. I do not do it for any purpose, but now that I have started this habit, I am afraid I will stick to it. In some occasions, it has been of some resolve some questions, but in other instances, my notes where not detailed enough. I do use my personal website to track certain events. I wonder whether it is worth collecting large amount of data and thinking that in the future it might be of some benefit. If it is worth investing some energy now to record it in an organized manner, it might be of some use in the future.
h0p3 6y ago
Just wanted to say, I'm a fan of your site (even directing my offspring to think about your work). `/salute`
luord 6y ago
> this is arguably the most important thing you should export considering how heavily everything relies on email I found this one interesting because I practice inbox zero so I delete 99.9% of the emails I get. Regardless, I do have some I keep just in case, so I should back those up.
simonebrunozzi 6y ago
Imagine if a service like this existed. Meaning: a service that would allow you to collect all this data, without any effort. It's doable. It could be an app on your phone. Properly done, it would be amazing.

187 points by karlicoss 6y ago

I want to do something similar, and load it in a place like snowflake. I say "like" snowflake because its annoying that they have a $25 a month minimum, otherwise it's already nicely suited to be a personal data warehouse!

benmanns 6y ago
BigQuery charges per query and for storage which might work for you. Where did you see the $25 minimum for Snowflake?
- harporoeder 6y ago
  BigQuery or Snowflake seem rather extreme for anything I might consider personal data. Even logging as many things as in the article a local sqlitedb and your preferred backup solution would get pretty far. It might be easier to setup nice dashboards with a cloud product though.
  - throwaway3699 6y ago
    I can see the appeal. Those services might be designed for petabytes of data analysis, but the flip side is that you have something ready to go with little maintenance.
  - ramraj07 6y ago
    I want the db to be in the cloud. I'm not on my laptop most of my time and I'm definitely not on the same laptop more than 20% of my time. Hosting any sqlite db or postgres db on the cloud means I pay for hot storage which is significantly expensive if I have 200gb of data say. (Which I do; I have a ton of data from my lab days that also I want queryable like this; I also intend to purchase some datasets that I think might be fun to keep at hand so)
- ramraj07 6y ago
  Forgot about BigQuery, will have to give it a shot! The Snowflakes minimum Is something my colleague got from them when we set up an account for a startup idea.
simonw 6y ago
This is one of the reasons I've been building my version of this on top of SQLite: it's incredibly cheap. All you need is a writable disk somewhere. I started with a $5/month VPS.
- ramraj07 6y ago
  How much data are you storing? How much is 200gb of storage? At least on lightsail it wasn't cheap! Also that's not even backed up!
  - simonw 6y ago
    Only about 20GB, so it's pretty inexpensive. I'm not actually bothering to run backups because theoretically ALL of the data there can be retrieved from other sources - pulled back out of the Twitter archive exports for example. But another similar project uses tarsnap for backups, which is pretty inexpensive.
edmundsauto 6y ago
You can pretty easily store most of these data in any database. Don't need something like Snowflake - postgres stock can handle about any analytics queries, especially if you're not writing too much.
- ramraj07 6y ago
  So the issue is not paying for compute when you're not using it. Firstly I've tried running postgres on a low powered lightsail instance and while it runs, it's kinda insane that it takes a ton of time to load data and then to query it. Indexless scans of data in large tables (let's say a db with all my emails) can take a while! I've tried the same in snowflake, and I'm only using their smallest warehouse, and everything happens in seconds. Loading GBS of data is seconds, scanning GBs of data is seconds. Their smallest compute warehouse is still significantly beefier than the cpu you get with a puny lightsail with a shared vcpu and while it's disk io is shit (since it's reading from S3) the parallelism makes up for it. In the end I think it's the pricing model that makes the difference. With snowflake you seamlessly launch a very powerful machine to run your query and only get billed for a minute, and with running postgres you need to run your machine all the time. Also hot storage (as opposed to s3) is also non trivially expensive. Haven't seen any way to get 200gb of ssd storage without spending that 25 dollar minimum that snowflake costs anyway. This is data I intend to not even update or query more than a few times a month, so I really don't want to pay for compute running all the time in its name.
  - edmundsauto 6y ago
    That's a fair point - the 200GB disk costs a bit more if you need an SSD (although my experience shows an SSD is not super necessary unless you have write heavy workloads or otherwise unusual access patterns). If you drop the SSD requirement, a cheap dedicated server will blow Snowflake out of the water. As a bonus, you can run your own code (ETL, scrapers, dataviz). Kimsufi has 4 core 2TB (spinning) w/ 16GB RAM for $17/mo. Personally, that is the route I go for my personal warehouse, as I find I almost always want to put an API or django app in front (and other software like Celery, scrapers, etc.)

t0ughcritic 6y ago

How will you graph it all or draw correlations ?

karlicoss 6y ago
For graphing: different data requires different representations, so I'm figuring out how to do it in my dashboard project: https://github.com/karlicoss/dashboard https://github.com/karlicoss/dashboard For dashboard, I started with health related data, because feel like other stuff (like 'histogram of my tweets' would be amusing, but not super useful immediately), and don't really expect any interesting quantified self insights from it. I'm gradually figuring out 'generic' ways to compute, plot and examine correlations, e.g. here https://github.com/karlicoss/dashboard/blob/708eb183130de31e344fa0f38fa8ae74d1953e31/src/dashboard/combined.py#L84-L101 https://github.com/karlicoss/dashboard/blob/708eb183130de31e... In general I'd like to have some automatic system which can consume all of my data and suggest interesting correlations in almost unsupervised way. (got more notes/possible implementations on it here https://beepb00p.xyz/exobrain/projects/dashboard.html#mtvtn https://beepb00p.xyz/exobrain/projects/dashboard.html#mtvtn )
- deleted 6y ago
  [deleted]
- johnwyles 6y ago
  I feel like the QS space needs a suite not unlike Home Assistant where you can plugin multiple integrations to these various services and begin collecting data relatively simply. Have you found anything in your travels that is mature enough to extend or spend time enhancing?
  - karlicoss 6y ago
    I still need to incorporate Home Assistant into my infrastructure, so not sure if I understand the analogy exactly. But basically I think the answer is 'no' -- there are so many different 'APIs' (quoted because often there aren't proper apis, you have to scrape, export manually etc.), that very few people attempted to unify it. I have some thoughts on how such system could work like, described here https://beepb00p.xyz/sad-infra.html#data_mirror https://beepb00p.xyz/sad-infra.html#data_mirror and here: https://beepb00p.xyz/exports.html https://beepb00p.xyz/exports.html . The export tools I wrote are the first order approximation of this vision -- seems robust enough for me so far, so now I'm working on connecting it to other tools and looking to cooperate with other people.

1996 6y ago

> How: many shells support keeping timestamps along your commands in history. > E.g. "Remember all your bash history forever". It would be far more important to get $? to get the rate of error of the commands you type (ex: typo) and use that as an indicator of poor mental acuity (ex: bad sleep, stress, ...)

karlicoss 6y ago
Interesting idea! I suspect I'd need to use shell far more often than I do currently to get anything conclusive though. But I guess it's possible to infer error rate even from past data by looking at closely spaced commands wish short Levenshtein distance.
- 1996 6y ago
  It depends on your workflow. I'd say your approach is interesting, but it's mostly driven by availability of the data ("searching under a streetlight") except for your sleep monitor (however, being an under pillow solution, it's not essentially different from the accelerometer data from a smart watch) I would suggest you invert the logic: design objective (ex: mental acuity), conclude on what things you want to track (ex: time slept, cups of coffee per day, time you drink it, errors in shell) and get the data accordingly, spending as necessary (ex: smart coffee maker? or maybe a smart plug that can let you infer when the machine is being used from the wattage drawn) As for data tricks, L distance is a good one, yet not applicable for shell, as it's sensitive to the string length so you would need correction. Also, it's missing the essential metric: did the command work? Only the error return code will give you that. TLDR: think about what you want, but before that think about why you want it. Collecting useless metrics is another form of data hoarding.
  - karlicoss 6y ago
    Yeah, it's a good point, and I'm indeed a bit guilty of hoarding data I can't immediately process. However, the problem with the 'objective first' approach is that it's gonna require a lot of data to draw meaningful rigorous conclusions from small interventions, so I'm making sure to 'secure' the data first, and then gradually process it. But also, it's a challenge in itself -- I'm automating a lot, sharing my system and trying to interoperate with existing tools, in the hope that my work can be useful to other people and make quantified self easier for them.

uaas 6y ago

Have you maybe compared Emfit QS to a customer-grade tracker like Fitbit directly?

karlicoss 6y ago
Yeah, I also have a Garmin watch, so wanted to write a script to properly compare the data for a while. From the few spot checks (e.g. actually comparing 20 select days against my manual records) - heart rate and respiration rate match pretty closely (kind of expected I guess) - Emfit is noticeably more accurate at detecting asleep time/awake time. But even so, sometimes emfit doesn't detect that you've woken up and just lying in bed with the phone. - 'sleep phases' at a glance don't match well. Definitely need a comparison script to do any stronger claims. In addition, I'm still not sure if sleep phases actually mean anything at all (for my body, anyway), need to do more analysis about it. - Emfit detects HRV, which doesn't seem to correlate with any of my subjective feelings. However, it does seem to change after exercise, for example.

hluska 6y ago

I got heavy into tracking personal data after spending a week in a cardiac unit in July 2017. My own tracking efforts have lead me to the absolute best physical shape I have ever been in. My weight went from 340 pounds (hence the cardiac unit) to 160. I have gotten into ultramarathons and lately, triathlons. Strength wise, I managed to drop 180 pounds while increasing my strength. It is amazing what the mind can do if you prime it with the right kinds of metrics.

elorant 6y ago
Care to describe what devices/programs you're using?
- hluska 6y ago
  Sure, I would love to but I'll warn you that this sounds a little fucked up (because it is). I was writing code and was two days from launching a startup when my heart went to hell. I called our local Healthline and was told to call 911 because "you are having a heart attack" but I'm a moron so decided to drive myself to the hospital. (I want to start with that story because I got myself into that mess by being a moron and figure that it's important you don't think this came from a place of intellect - it was 99% fear, trauma and trying to overcome the stupid defaults that got me into that mess in the first place.) The first day that I got out of the hospital, I had to move back in with my Mom. I decided to go out for a walk and was in such a bad place that two little old ladies (one of whom had a walker) actually smoked me on the path back to my Mom's area. I was remarkably close to killing myself after that experience and in all truth, if it wasn't for my daughter (she was 18 months old at the time), I would not be here to write this. My GP referred me to a psychiatrist. My psychiatrist is an amazing man and an incredible athlete. He suggested two tools that could help me with my cardio recovery but that have also been extraordinarily useful for my mental health. He recommended a Masimo fingertip oximeter and a training band by Polar. The Polar training band is/was amazing. My first Polar was an H7 with an XXL band. Now I use an H10 with a small band. I'm not sure that I would choose the Polar again if I was starting out now - at the time, my psychiatrist suggested it because I could find a band that would fit in between my massive gut and the band of fat around my chest that used to lead coworkers to tease me about my bra size (I was assigned male at birth and continue to identify as male so that was not complimentary. It was funny at the time but quite sad to think about now.) Program wise is where this story starts to get a little bit fucked up. Remember the first paragraph where I told you that I was writing code when my heart went to hell? Well, that really fucked me up and I quite literally could not write code or even sit in front of a computer. Computers have been my biggest love and practically my sole hobby since I was a 7 year old with a bad stutter. Suddenly, that was replaced with the most unbelievable fear whenever I would open up a text editor or even sit down at a keyboard. Consequently, at first I started using mobile apps and ran everything through my iPhone. But as I started to recover and as my psych meds, therapy and other psychiatric help started to kick in a little more, my brain started to come back. I have a marketing degree and really like data so as my brain started to come back, I started noticing correlations between cardiac metrics and purchasing activity. That was strange. In a sense, my capitalism came back and I saw an opportunity. But, at the same time, I was a patient and doing cardiac rehab/support with ten older adults from my Mom's community. This sounds weird to anyone with a healthy heart but I'm a firm believer that whoever coined the term 'a broken heart' had had a cardiac intervention. It feels exactly the same and I had this group of older adults who cared enough about me to check in on me, come by and make me go for walks or even just sit and watch baseball with me. Consequently, I was never able to turn that opportunity into a business, but it gave me an idea about some code to write. So, I started writing my own fitness tracker. At some point I might release it because it's been damned good for me. The first version is unbelievably sad and I have one hell of a lot of trouble reading that code over now. It was somewhere between a suicide note and a eulogy for my former self but that minimizes how brutally sad some of my variable and method names were. My favourite example is fatFuck.name = Greg. That project turned into rehab and through the years I have kept adding to it. At first, it was purely about rehabbing my brain so I could write code again and maybe graduate to a point where I could comfortably refer to myself as user.name and heck, maybe even remove the word 'dead' from all my method names. But as it has grown, it has turned into a really good tool for teaching myself how to think in terms of healthy defaults. Here is where stuff gets really fucked up sounding so if you've made it this far, good for you! I don't know if the app I wrote has actually helped me or if I hacked myself. I am deeply obsessive, to the point that when my psychiatrist did my OCD screening, I went home and rewrote my resume because the screening materials described my talents better than anything else I had ever written. When I build anything, I test it obsessively. In this case, to test it, I had to live it and 3.5 years later, I have a six pack, no love handles, two marathons down and an ultra in my imminent future. Tracking nutrition against performance metrics yielded the most powerful results of all. At first, I started by writing down what I ate and then tracking my macros. I learned that what I thought was 2000 calories was actually closer to 4000 calories. As I started paying more attention to macros like protein and carbohydrates, I found a great feedback loop where eating healthier foods resulted in improved athletic performance. As my athleticism increased, I was able to work harder and then started tracking my moods against that data. That created another feedback loop where the better I ate, the harder I could exercise and the better I would feel. At this point, 3.5 years later, I actually have healthy defaults. It sounds insane, but I genuinely have healthy defaults. The way I eat, the foods I eat and even the textures I enjoy have completely shifted around. The only downside at this point is that I'm rather skeptical of most of the wearable industry. I've seen way too many metrics that make my capitalism tingle in funny ways. I know that I could enhance tracking with other tools but at this point, I'm pretty obsessed with reading their privacy policies and speaking as a patient instead of an entrepreneur, I don't think the marketing industry needs (or deserves) this type of data. I feel like I've missed a lot. At some point, I'll put everything together into written form and do a better job of this reply. Sorry if this was disjointed!!

lifeisstillgood 6y ago

This is ... the future. I am going to start off with a few this week - well done and good luck !

WeepinStolzham 6y ago

Never go Full Stephen Wolfram.

WephenSolfram 6y ago

Never go full Stephen Wolfram.

simonw 6y ago

Have you considered loading this stuff into SQLite in addition to keeping the raw exports in files on disk? Being able to query your personal data with SQL can get really interesting. I've been using it for my own version of a personal data warehouse, described here: https://simonwillison.net/2020/Nov/14/personal-data-warehouses/ https://simonwillison.net/2020/Nov/14/personal-data-warehous...

karlicoss 6y ago
Yep! In fact I've tried interoperating with Datasette (e.g. shared here https://news.ycombinator.com/item?id=25090643 https://news.ycombinator.com/item?id=25090643 ) One secret sauce is using 'automatic' caching of data in sqlite -- this allows both for faster access and having an additional interface for the data as a collateral https://github.com/karlicoss/cachew#readme https://github.com/karlicoss/cachew#readme Still need to polish this a bit, but ultimately hoping to properly plug into Datasette, I'm impressed by its data exploration capabilities!

andrewnc 6y ago

I love the idea of personal data analysis. In fact, I recently wrote a book with that premise called "Everyday Data Science". I just wrote about it in another thread, but it seems relevant to the discussion here. I definitely agree that you can really take control of your life with a bit of data and some not so fancy analysis. In the book I shared the story of an old professor of mine that tracked a personal health marker from yearly screenings. The threshold for concern was "4 ng/ml". His numbers were always right around 1. However, there was a super noticeable trend up to 2.5 and 3. The doctors all waved it off because it was still below the magic "4" number. However, when he showed them the graph of historic data, they rushed him to a specialist and detected (and removed) cancer. I think data collection and personal analysis for day-to-day stuff is vital and fascinating. Thanks for sharing :)

karlicoss 6y ago
Thanks! Yep, one of the motivations for me is learning & establishing health baseline. For example, with blood tests it's unlikely that I'll be able to infer any interesting correlations (at the very least because you can't do them every day, with the exception of glucose/ketones perhaps). However I'll know what is 'normal' for my body and if something breaks, hopefully will have some clues.

friggeri 6y ago

Over the past few years I’ve thought about building something similar to Home Assistant, but for human centered data and APIs vs home centered. Essentially a shared bus on which different data sources could be plugged into a standardized format, on which people could build dashboards and automations. Never got around to doing so, because sidetracked by other projects, but if others are interested in teaming up I’d love to work on something like that (email in profile if anyone wants to start this)

tomaszs 6y ago

What we need is a law that obligates companies to enable scheduled auto export to a given URL https://github.com/tomaszs/RightToBeRemembered/blob/main/README.md https://github.com/tomaszs/RightToBeRemembered/blob/main/REA...

fjfaase 6y ago

In that past ten years, I have been recording my daily activities in Moleskine daily planners. I am using the small size daily planner with one page per day. I always carries it with me. The main motivation to start doing it, was to fight the feeling that time is going faster and faster. It takes me about a quarter of an hour per day. I do not record every things, just the things I think a worth recording. I do track all of my movements. I have thought about recording everything in a database, but never come to it. For some time, I have been tracking my travels with Google Earth. The last year, I have started doing this again for for walking and biking 'trips' I make. I do not do it for any purpose, but now that I have started this habit, I am afraid I will stick to it. In some occasions, it has been of some resolve some questions, but in other instances, my notes where not detailed enough. I do use my personal website to track certain events. I wonder whether it is worth collecting large amount of data and thinking that in the future it might be of some benefit. If it is worth investing some energy now to record it in an organized manner, it might be of some use in the future.

h0p3 6y ago

Just wanted to say, I'm a fan of your site (even directing my offspring to think about your work). `/salute`

luord 6y ago

> this is arguably the most important thing you should export considering how heavily everything relies on email I found this one interesting because I practice inbox zero so I delete 99.9% of the emails I get. Regardless, I do have some I keep just in case, so I should back those up.

simonebrunozzi 6y ago

Imagine if a service like this existed. Meaning: a service that would allow you to collect all this data, without any effort. It's doable. It could be an app on your phone. Properly done, it would be amazing.