5 ms·

Maintain access 20 year old fitness data? What for? I'm fairly far along on the data hoarding spectrum. This extends, for example, to over 200K photos from m

Maintain access 20 year old fitness data? What for? I'm fairly far along on the data hoarding spectrum. This extends, for example, to over 200K photos from my life, categorized and indexed. It's trivial to look at "what happened on this day 20 years ago" or at least on the nearest date to that day that had pictures taken. And here's the thing: As the memories fade, so does interest in the photos. 10 years ago you still remember clearly. 20 years ago is another era. And that's photos. Fitness data? Do I really want to see how I declined over the last 20 years? I see that already from the average speed display on my bike computer. Fitness data is the ultimate "looking forward" thing. But maybe that's my age; it may look different from a younger perspective. Agree with others that just use a platform that lets you export open-ish files. .gpx doesn't do heart rate, but .fit does, and gpsbabel can translate that to another format as needed.

gpetukhov 5y ago
Would love to know how you categorize and index 200k photos.
- MarkusWandel 5y ago
  Homemade system that is not in any way publishable. It's a folder hierarchy but also viewable as a chronological timeline. The main key is to organize them as you go, and not fall too far behind.
  - skytreader 5y ago
    > Homemade system that is not in any way publishable. I can still access all the photos I took since around 2008, when I first started to nurture an interest in photography as a hobby. I never delete b-rolls unless they're absolute crap (i.e., I triggered with my lens cap on, too dark with no flash---blurry, out-of-focus shots do not count as absolute crap). I also manage it with a homemade system! This system is closer to my darkest shame than proudest achievement. It kinda reminds me of Destin's pre-Linus-organized system[1] except way smaller and a few scripts more sophisticated (my system has redundancy, ha!). Despite that, I have maintained access despite moving across continents. In fact, last year, slightly spurred on by the pandemic situation, a bunch of friends asked me if I still have copies of photographs I took years prior. The oldest query was only for 4-5 years ago but I am proud to say that I was able to oblige despite my shameful system. Kids, don't do this in prod, but homemade systems FTW! :) [1] https://www.youtube.com/watch?v=lcWSrIiR1tY https://www.youtube.com/watch?v=lcWSrIiR1tY
matsemann 5y ago
Hmm. While I don't really need my heart rate or speed for every activity I've done the last years, some of it is actually interesting. Like looking at accumulated time or distance of various activities over the years. I also keep a list of all big events I've been to and my stats from those, which I find cool. But of course, that doesn't have to be exportable. Just a few days ago I could find out the answer to a Q about when in the year I normally do my first cross country ski session, by looking at the graphs per year. So on average I will be going skiing next weekend 8)
markdown 5y ago
OT but... > 200K photos from my life, categorized and indexed. What app do you use to manage your photo library?
- mceachen 5y ago
  (...he _should_ be using PhotoStructure, but I _may_ be biased...)
- jjav 5y ago
  > What app do you use to manage your photo library? It doesn't seem like a good idea to tie a photo library to any app. Applications come and go, the photo library should last a lifetime (or more if you pass it on). You don't ever want to get caught with your photo data in some proprietary structure when the app that created it is no longer available for whatever reason. My photo library certainly has outlived many applications I've used to process the photos. After processing I store them in just a directory hierarchy. Top level dirs for each year, event/topic subdirectories under each year. That's it.
  - markdown 5y ago
    That's fine for file storage, but doesn't provide a way to find all photos with dog in them, or photos from 2007 - 2012 with bird and dog, but not cat, in them. Most apps will let you store your photos how you want them to be stored on the filesystem.
hvgk 5y ago
I think this is a resolution argument. Data should be curated carefully and reduced in resolution as it gets older. If you’re never going to look at it, it’s probably not worth keeping. As per photos you don’t need 11 pictures of those muffins you baked in 2003. One will probably suffice, if you even care about them 18 years later that is. The same goes for health data. But overall trends over 20 years are really useful as you get older. Expected decline is inevitable but unexpected decline may be indicated earlier and result in some preventative action to improve your quality of life. There are a lot of long term metrics appearing in Apple health like walking stability on that front. On photos, I’m not sure what I’d do with 200,000 photos. I have 7,623 which span 4 generations and 125 years at the moment. I spend a lot of time curating these photos, adding metadata, editing and pruning garbage. They are all managed with Apple’s native Photos app which is good enough for the job (non destructive, sqlite DB underneath it, easy to back up)
- acadapter 5y ago
  >On photos, I’m not sure what I’d do with 200,000 photos. I have 7,623 which span 4 generations and 125 years at the moment. Some of them might be historically interesting enough to preserve on Wikimedia Commons, for example if they depict places that look very different today, or stuff that you don't see in everyday life anymore.
  - hvgk 5y ago
    That’s quite possible. I have around 150 historical photos i.e. over 100 years old which depict family life in Switzerland in the 1800s. I will look into that. Thanks for the suggestion.
- NoboruWataya 5y ago
  Out of interest, what is the argument for reducing resolution other than space concerns? Fitness data is not really space-intensive. I recently ran a marathon and the associated GPX file is less than a megabyte; the FIT file is about 180KB. I don't really think even 30 years of regular running data would come to much. I appreciate cycling data will take more space but not that much more. My point is, it seems like once you have put in place the habits and technological solutions necessary to store fitness data, it actually takes more effort to reduce the resolution of older data. I'd rather have data and not want it than want it and not have it, but then I am an almost pathological hoarder so maybe it's just me.
KineticLensman 5y ago
(since photo management seems to be of interest) I have approx. 220k photos, of which 210k are in camera raw, 10k are ‘processed’, which typically means exposure corrected, cropped, etc. They are stored in two parallel directories (raw and processed), each of which is structured into years which are further subdivided into ‘shoot’ folders – one for each distinct upload from the camera. I use Adobe Bridge to do the uploads and to change the raw files names to support my indexing approach. The year folder names are simple (e.g. ‘2021’) and the shoot folders have names like ‘2021-010 garden birds young’ (where 010 is the sequence # of the shoot in the year). The folders thus automatically list themselves in chronological order when sorted alphabetically. A processed image might have a name like ‘20210523_010_my-IRL-name-here_0019.psd’ (so that I can immediately find the raw or processed shoot folder given a file name). I insert my name so that if I a send an image to someone I am identified as a creator (image metadata contains additional contact / copyright info). I’ll often extend the automatically generated file name to include the photo subject as I process specific images. My indexing system relies on my memory and the fact that the relatively small number of processed pictures are indicative of the contents of the wider raw set – so I can browse the relatively small number of processed shots (using Bridge) if I am looking for something thematically. I usually manage to cull obvious fails from the raw set when I do an upload before doing any processing. Storage costs being what they are I have not yet had to further cull raw or processed images to make room on my hard-drives – currently 1.7Tb of images. I have left it too late to add meaningful tags to this collection, and because I am the only librarian, I don’t have a need to help other users find particular images.
- xattt 5y ago
  Any suggestions for managing an advanced photo library locally with a concurrent Google Photos library to share with family?
  - mceachen 5y ago
    I'd recommend using one of several mobile phone apps to sync your phone's camera/DCIM directory with a computer/NAS at home, so you have your originals. https://photostructure.com/faq/how-do-i-safely-store-files/#how-do-i-back-up-files-on-my-phone https://photostructure.com/faq/how-do-i-safely-store-files/#... I wouldn't recommend syncing from, say, your Google Photos library: even in "original quality," your files may be mucked with (it depends on your platform). Once you've got them local, there are several apps to share photos via local webserver. I wrote PhotoStructure to deduplicate and manage my 500k+ photos and videos from the past 20+ years.
Angostura 5y ago
> As the memories fade, so does interest in the photos. 10 years ago you still remember clearly. 20 years ago is another era. That's what makes them so interesting. The photos from 15 years ago with my parents still alive and my kids as toddlers are more interesting than last year's holiday.
- MarkusWandel 5y ago
  Fair enough. My older is just under 10 years old, so that kind of sets the "current era" event horizon. 20 years ago is relatives who aren't alive any more, parties full of people I can't even put a name to any more, trips of which the memories have faded, stuff like that.
xyzzy21 5y ago
For medical diagnosis when you get older, having that data will actually be useful in the next 20-40 years to diagnosing things that will happen to you then.
jjav 5y ago
> Maintain access 20 year old fitness data? What for? Why not? I have about 12 years worth because that's when I started tracking, no reason to ever delete it. Saving the 0.001% (actual value, I just checked) on disk space isn't a motivation. It's fun to review old data. Occasionally useful, even. Mostly for fun. > And here's the thing: As the memories fade, so does interest in the photos. 10 years ago you still remember clearly. 20 years ago is another era. Wait another 30 years and those now-50 year old photos will be treasure. I don't look much at my photos from 20 years ago, that's pretty recent so I remember it well. But photos from 50 years ago are quite the treasure. Even older ones (like when grandparents were toddlers) are invaluable. Wish there were tons more of those. But yes that's photos. Fitness data is certainly less interesting, but still fun and costs approximately zero space to keep.

gpetukhov 5y ago

Would love to know how you categorize and index 200k photos.

MarkusWandel 5y ago
Homemade system that is not in any way publishable. It's a folder hierarchy but also viewable as a chronological timeline. The main key is to organize them as you go, and not fall too far behind.
- skytreader 5y ago
  > Homemade system that is not in any way publishable. I can still access all the photos I took since around 2008, when I first started to nurture an interest in photography as a hobby. I never delete b-rolls unless they're absolute crap (i.e., I triggered with my lens cap on, too dark with no flash---blurry, out-of-focus shots do not count as absolute crap). I also manage it with a homemade system! This system is closer to my darkest shame than proudest achievement. It kinda reminds me of Destin's pre-Linus-organized system[1] except way smaller and a few scripts more sophisticated (my system has redundancy, ha!). Despite that, I have maintained access despite moving across continents. In fact, last year, slightly spurred on by the pandemic situation, a bunch of friends asked me if I still have copies of photographs I took years prior. The oldest query was only for 4-5 years ago but I am proud to say that I was able to oblige despite my shameful system. Kids, don't do this in prod, but homemade systems FTW! :) [1] https://www.youtube.com/watch?v=lcWSrIiR1tY https://www.youtube.com/watch?v=lcWSrIiR1tY

matsemann 5y ago

Hmm. While I don't really need my heart rate or speed for every activity I've done the last years, some of it is actually interesting. Like looking at accumulated time or distance of various activities over the years. I also keep a list of all big events I've been to and my stats from those, which I find cool. But of course, that doesn't have to be exportable. Just a few days ago I could find out the answer to a Q about when in the year I normally do my first cross country ski session, by looking at the graphs per year. So on average I will be going skiing next weekend 8)

markdown 5y ago

OT but... > 200K photos from my life, categorized and indexed. What app do you use to manage your photo library?

mceachen 5y ago
(...he _should_ be using PhotoStructure, but I _may_ be biased...)
jjav 5y ago
> What app do you use to manage your photo library? It doesn't seem like a good idea to tie a photo library to any app. Applications come and go, the photo library should last a lifetime (or more if you pass it on). You don't ever want to get caught with your photo data in some proprietary structure when the app that created it is no longer available for whatever reason. My photo library certainly has outlived many applications I've used to process the photos. After processing I store them in just a directory hierarchy. Top level dirs for each year, event/topic subdirectories under each year. That's it.
- markdown 5y ago
  That's fine for file storage, but doesn't provide a way to find all photos with dog in them, or photos from 2007 - 2012 with bird and dog, but not cat, in them. Most apps will let you store your photos how you want them to be stored on the filesystem.

hvgk 5y ago

I think this is a resolution argument. Data should be curated carefully and reduced in resolution as it gets older. If you’re never going to look at it, it’s probably not worth keeping. As per photos you don’t need 11 pictures of those muffins you baked in 2003. One will probably suffice, if you even care about them 18 years later that is. The same goes for health data. But overall trends over 20 years are really useful as you get older. Expected decline is inevitable but unexpected decline may be indicated earlier and result in some preventative action to improve your quality of life. There are a lot of long term metrics appearing in Apple health like walking stability on that front. On photos, I’m not sure what I’d do with 200,000 photos. I have 7,623 which span 4 generations and 125 years at the moment. I spend a lot of time curating these photos, adding metadata, editing and pruning garbage. They are all managed with Apple’s native Photos app which is good enough for the job (non destructive, sqlite DB underneath it, easy to back up)

acadapter 5y ago
>On photos, I’m not sure what I’d do with 200,000 photos. I have 7,623 which span 4 generations and 125 years at the moment. Some of them might be historically interesting enough to preserve on Wikimedia Commons, for example if they depict places that look very different today, or stuff that you don't see in everyday life anymore.
- hvgk 5y ago
  That’s quite possible. I have around 150 historical photos i.e. over 100 years old which depict family life in Switzerland in the 1800s. I will look into that. Thanks for the suggestion.
NoboruWataya 5y ago
Out of interest, what is the argument for reducing resolution other than space concerns? Fitness data is not really space-intensive. I recently ran a marathon and the associated GPX file is less than a megabyte; the FIT file is about 180KB. I don't really think even 30 years of regular running data would come to much. I appreciate cycling data will take more space but not that much more. My point is, it seems like once you have put in place the habits and technological solutions necessary to store fitness data, it actually takes more effort to reduce the resolution of older data. I'd rather have data and not want it than want it and not have it, but then I am an almost pathological hoarder so maybe it's just me.

KineticLensman 5y ago

(since photo management seems to be of interest) I have approx. 220k photos, of which 210k are in camera raw, 10k are ‘processed’, which typically means exposure corrected, cropped, etc. They are stored in two parallel directories (raw and processed), each of which is structured into years which are further subdivided into ‘shoot’ folders – one for each distinct upload from the camera. I use Adobe Bridge to do the uploads and to change the raw files names to support my indexing approach. The year folder names are simple (e.g. ‘2021’) and the shoot folders have names like ‘2021-010 garden birds young’ (where 010 is the sequence # of the shoot in the year). The folders thus automatically list themselves in chronological order when sorted alphabetically. A processed image might have a name like ‘20210523_010_my-IRL-name-here_0019.psd’ (so that I can immediately find the raw or processed shoot folder given a file name). I insert my name so that if I a send an image to someone I am identified as a creator (image metadata contains additional contact / copyright info). I’ll often extend the automatically generated file name to include the photo subject as I process specific images. My indexing system relies on my memory and the fact that the relatively small number of processed pictures are indicative of the contents of the wider raw set – so I can browse the relatively small number of processed shots (using Bridge) if I am looking for something thematically. I usually manage to cull obvious fails from the raw set when I do an upload before doing any processing. Storage costs being what they are I have not yet had to further cull raw or processed images to make room on my hard-drives – currently 1.7Tb of images. I have left it too late to add meaningful tags to this collection, and because I am the only librarian, I don’t have a need to help other users find particular images.

xattt 5y ago
Any suggestions for managing an advanced photo library locally with a concurrent Google Photos library to share with family?
- mceachen 5y ago
  I'd recommend using one of several mobile phone apps to sync your phone's camera/DCIM directory with a computer/NAS at home, so you have your originals. https://photostructure.com/faq/how-do-i-safely-store-files/#how-do-i-back-up-files-on-my-phone https://photostructure.com/faq/how-do-i-safely-store-files/#... I wouldn't recommend syncing from, say, your Google Photos library: even in "original quality," your files may be mucked with (it depends on your platform). Once you've got them local, there are several apps to share photos via local webserver. I wrote PhotoStructure to deduplicate and manage my 500k+ photos and videos from the past 20+ years.

Angostura 5y ago

> As the memories fade, so does interest in the photos. 10 years ago you still remember clearly. 20 years ago is another era. That's what makes them so interesting. The photos from 15 years ago with my parents still alive and my kids as toddlers are more interesting than last year's holiday.

MarkusWandel 5y ago
Fair enough. My older is just under 10 years old, so that kind of sets the "current era" event horizon. 20 years ago is relatives who aren't alive any more, parties full of people I can't even put a name to any more, trips of which the memories have faded, stuff like that.

xyzzy21 5y ago

For medical diagnosis when you get older, having that data will actually be useful in the next 20-40 years to diagnosing things that will happen to you then.

jjav 5y ago

> Maintain access 20 year old fitness data? What for? Why not? I have about 12 years worth because that's when I started tracking, no reason to ever delete it. Saving the 0.001% (actual value, I just checked) on disk space isn't a motivation. It's fun to review old data. Occasionally useful, even. Mostly for fun. > And here's the thing: As the memories fade, so does interest in the photos. 10 years ago you still remember clearly. 20 years ago is another era. Wait another 30 years and those now-50 year old photos will be treasure. I don't look much at my photos from 20 years ago, that's pretty recent so I remember it well. But photos from 50 years ago are quite the treasure. Even older ones (like when grandparents were toddlers) are invaluable. Wish there were tons more of those. But yes that's photos. Fitness data is certainly less interesting, but still fun and costs approximately zero space to keep.