4 ms·

This appears to be NEITHER your "Archive" NOR your albums in Google Photos. This is basically a collection of media you uploaded to Blogger, Hangouts, and Picas

by mholt 3y ago

This appears to be NEITHER your "Archive" NOR your albums in Google Photos. This is basically a collection of media you uploaded to Blogger, Hangouts, and Picasa Web Albums. Still, doesn't hurt to Takeout your Google Photos every ~6 months. (I am working on an app that will help you organize and view them, together with your text messages, location history, and other online-only data.)

citelao 3y ago
I would be extremely interested in a Google Takeout viewer if you ever end up releasing one. I dealt with Google Takeout, trying to export my photos to Apple Photos (when Google was planning to charge money for old Google Workspace accounts), and I found it extremely difficult to deal with the file format. The script I wrote (https://github.com/citelao/google_photos_takeout_to_apple_photos https://github.com/citelao/google_photos_takeout_to_apple_ph...) ended up being decently reliable, but there were a ton of weird mismatches between the EXIF data in Google Photos metadata and the EXIF data in the photos themselves. Although some of that wonkiness was Apple Photos, not Google. I'd love to see software that could wrangle the mess :)
- mholt 3y ago
  It's called Timelinize (might rename it?), and you can follow it here: https://twitter.com/timelinize https://twitter.com/timelinize (Click "Media" on the Twitter account to view a few screenshots for a preview. More to come!) (There's no website or project page yet because I've been busy developing.) If you want an invite to try out an early dev preview today, follow @timelinize on Twitter and tweet at it, I'll see about getting you into the Discord. Some background: Saving a local copy of my Google Photos has been a passion project of mine since ~2014 (before Google Photos even!). For years it was only focused on downloading the data using APIs -- but then we found out that Google strips location data (from your own photos!) if using the API, so I added Takeout support. The problem is there was no viewer. Well in 2019 I finally started working on a viewer. It has evolved a lot since it's a very ambitious project and there's nothing quite like it. It's not just Google Photos: it's any photos and videos. It's also for your text messages and emails. And your location history. And contact list. And chat apps. And really, any files you have. It also supports Facebook, Twitter, and Instagram account exports too. Oh, and iPhone backups. Timelinize is entity-aware, and it can map identities across data sources (with enough info, or with a manual mapping, or some optional heuristics). It's just not a photo gallery. It's basically a really detailed view of your life and online history. It's neat because I have my family pictures, my text messages between me and my wife when we were dating (and after of course), and there's different views to explore: map, timeline, conversations, gallery, and more to come (calendar, etc). We can even place non-geolocated data on a map since we can correlate timestamp and entity. So when we went on our honeymoon, I can see text messages received from friends while we were driving to a beach. It's really quite immersive and magical and I haven't seen anything quite like it. And everything is stored on your own computer, it's a GUI app and you have to have enough space to store your stuff. The data is just organized as files within a folder on disk, with a SQLite DB holding the index and the small textual items.
- chaxor 3y ago
  What is the correct tool to properly merge a large set of tar.gz files for which may have an enormous overlap of similar files, and some that have been altered just slightly? Git plus some parsing seems close in that space, as analyzing the files to create a dendrogram like tree of potential alterations to files over time by levenstein distance may be useful to approximate commit history. However, this doesn't seem to exist or be popular as a tool. There's vimdiff or meld, but they are extremely manual and tedious to the extent of being pointless to try for something like a large history of takeout tar.gz's. Throwing in the towel completely, borgfs can be helpful to reduce the amount of space they take by de-duplication on the block level, but this is a terrible solution as it doesn't really track file changes in a reasonable way, etc. It is useful to extract the files into a directory without the tar or gz, but this can also cause issues with how to appropriately organize the directory structure over the history. Any thoughts or projects that do a better job of this?
  - mholt 3y ago
    > What is the correct tool to properly merge a large set of tar.gz files for which may have an enormous overlap of similar files, and some that have been altered just slightly? Can you elaborate on this? My understanding is that they should all extract into the same target folder without issues because each archive's set of files is distinct. But maybe I'm just assuming wrongly? What exactly is your goal, too? It sounds like you are trying to find and de-duplicate visually similar images? Like what do you mean by "enormous overlap" or "altered just slightly"?
    - citelao 3y ago
      I'm not chaxor, but as far as I remember I think you're right: If I had unzipped all the takeout directories into one giant folder, there'd be no conflicts. Since I didn't do that, I had to do weird multi-pass parsing, since an album could be split across multiple ZIPs. I get a bit neurotic around backups like this, so I'd have loved some sort of virtualized filesystem that non-destructively represented all of those zips "merged together." But in retrospect, I should have just merged the directories into one folder---would have made parsing easier :) I don't recall substantial problems with duplicates. Just weird renames and EXIF data mismatches. And since I was trying to archive my data, I definitely didn't want similar photos to be deduplicated. My problems are probably different than chaxor's, though.
noman-land 3y ago
Just one correction. You should only run Takeout once and then delete all your Google accounts and never look back.
- stOneskull 3y ago
  i think give some time to make sure you've changed all your log-ins. there's bound to be accounts out there you've forgotten about and might not be able to use. maybe once you don't get any important emails for 6 months? and then there's like updating and moving all google authenticator entries to something else. things like that. anyway, don't just rush it!
  - LoganDark 3y ago
    Note to anyone who might be reading this and thinking "that's me" - if it's you, then you should move to a password manager and create a database of all your accounts. I once spent a couple days moving every account I have ever created into KeePassXC, and changing all of their passwords to unique randomly-generated ones. You only have to do this once, and given enough time (making sure you add accounts over time that you need to use but missed), you can be fairly confident that you know every single account you have, anywhere. At least any account that you'd ever need to log into. Then, you'd know every account that uses your gmail, but also you'd know the password of every account to change the email in the future without needing a password reset, if you want to delete your email early. Of course, assuming someone doesn't have a breach or something and require a reset anyway, which happens from time to time. Still good to have that passwords database. I personally replicate mine between multiple computers and also publish it to my web server on someone else's infrastructure. I will never forget another password again, because I will never need to know another password again. I will also always know what accounts I have or whether I've used a particular service before.
    - sunnybeetroot 3y ago
      One step further, buy a domain that offers email forwarding. When you sign up to websites, use this domain and have your email forwarded to whatever free email provider you choose. If you change providers you only need to change your domain forwarding email.
whynotmaybe 3y ago
Does that mean that all the images in my blog posts on blogger will disappear? So troubled to hear that a product I never used might disappear and that the impact might be massive for me... or not.
- megamike 3y ago
  it appears that way what a mess!!
predictabl3 3y ago
I have been getting those photos-cdp emails and noticed your name. I wait in earnest.
bexterw21 3y ago
[flagged]
bexterw21 3y ago
[flagged]
amai 3y ago
Does that mean blogger will also be shut down soon?

by mholt 3y ago

citelao 3y ago

I would be extremely interested in a Google Takeout viewer if you ever end up releasing one. I dealt with Google Takeout, trying to export my photos to Apple Photos (when Google was planning to charge money for old Google Workspace accounts), and I found it extremely difficult to deal with the file format. The script I wrote (https://github.com/citelao/google_photos_takeout_to_apple_photos https://github.com/citelao/google_photos_takeout_to_apple_ph...) ended up being decently reliable, but there were a ton of weird mismatches between the EXIF data in Google Photos metadata and the EXIF data in the photos themselves. Although some of that wonkiness was Apple Photos, not Google. I'd love to see software that could wrangle the mess :)

mholt 3y ago
It's called Timelinize (might rename it?), and you can follow it here: https://twitter.com/timelinize https://twitter.com/timelinize (Click "Media" on the Twitter account to view a few screenshots for a preview. More to come!) (There's no website or project page yet because I've been busy developing.) If you want an invite to try out an early dev preview today, follow @timelinize on Twitter and tweet at it, I'll see about getting you into the Discord. Some background: Saving a local copy of my Google Photos has been a passion project of mine since ~2014 (before Google Photos even!). For years it was only focused on downloading the data using APIs -- but then we found out that Google strips location data (from your own photos!) if using the API, so I added Takeout support. The problem is there was no viewer. Well in 2019 I finally started working on a viewer. It has evolved a lot since it's a very ambitious project and there's nothing quite like it. It's not just Google Photos: it's any photos and videos. It's also for your text messages and emails. And your location history. And contact list. And chat apps. And really, any files you have. It also supports Facebook, Twitter, and Instagram account exports too. Oh, and iPhone backups. Timelinize is entity-aware, and it can map identities across data sources (with enough info, or with a manual mapping, or some optional heuristics). It's just not a photo gallery. It's basically a really detailed view of your life and online history. It's neat because I have my family pictures, my text messages between me and my wife when we were dating (and after of course), and there's different views to explore: map, timeline, conversations, gallery, and more to come (calendar, etc). We can even place non-geolocated data on a map since we can correlate timestamp and entity. So when we went on our honeymoon, I can see text messages received from friends while we were driving to a beach. It's really quite immersive and magical and I haven't seen anything quite like it. And everything is stored on your own computer, it's a GUI app and you have to have enough space to store your stuff. The data is just organized as files within a folder on disk, with a SQLite DB holding the index and the small textual items.
chaxor 3y ago
What is the correct tool to properly merge a large set of tar.gz files for which may have an enormous overlap of similar files, and some that have been altered just slightly? Git plus some parsing seems close in that space, as analyzing the files to create a dendrogram like tree of potential alterations to files over time by levenstein distance may be useful to approximate commit history. However, this doesn't seem to exist or be popular as a tool. There's vimdiff or meld, but they are extremely manual and tedious to the extent of being pointless to try for something like a large history of takeout tar.gz's. Throwing in the towel completely, borgfs can be helpful to reduce the amount of space they take by de-duplication on the block level, but this is a terrible solution as it doesn't really track file changes in a reasonable way, etc. It is useful to extract the files into a directory without the tar or gz, but this can also cause issues with how to appropriately organize the directory structure over the history. Any thoughts or projects that do a better job of this?
- mholt 3y ago
  > What is the correct tool to properly merge a large set of tar.gz files for which may have an enormous overlap of similar files, and some that have been altered just slightly? Can you elaborate on this? My understanding is that they should all extract into the same target folder without issues because each archive's set of files is distinct. But maybe I'm just assuming wrongly? What exactly is your goal, too? It sounds like you are trying to find and de-duplicate visually similar images? Like what do you mean by "enormous overlap" or "altered just slightly"?
  - citelao 3y ago
    I'm not chaxor, but as far as I remember I think you're right: If I had unzipped all the takeout directories into one giant folder, there'd be no conflicts. Since I didn't do that, I had to do weird multi-pass parsing, since an album could be split across multiple ZIPs. I get a bit neurotic around backups like this, so I'd have loved some sort of virtualized filesystem that non-destructively represented all of those zips "merged together." But in retrospect, I should have just merged the directories into one folder---would have made parsing easier :) I don't recall substantial problems with duplicates. Just weird renames and EXIF data mismatches. And since I was trying to archive my data, I definitely didn't want similar photos to be deduplicated. My problems are probably different than chaxor's, though.

noman-land 3y ago

Just one correction. You should only run Takeout once and then delete all your Google accounts and never look back.

stOneskull 3y ago
i think give some time to make sure you've changed all your log-ins. there's bound to be accounts out there you've forgotten about and might not be able to use. maybe once you don't get any important emails for 6 months? and then there's like updating and moving all google authenticator entries to something else. things like that. anyway, don't just rush it!
- LoganDark 3y ago
  Note to anyone who might be reading this and thinking "that's me" - if it's you, then you should move to a password manager and create a database of all your accounts. I once spent a couple days moving every account I have ever created into KeePassXC, and changing all of their passwords to unique randomly-generated ones. You only have to do this once, and given enough time (making sure you add accounts over time that you need to use but missed), you can be fairly confident that you know every single account you have, anywhere. At least any account that you'd ever need to log into. Then, you'd know every account that uses your gmail, but also you'd know the password of every account to change the email in the future without needing a password reset, if you want to delete your email early. Of course, assuming someone doesn't have a breach or something and require a reset anyway, which happens from time to time. Still good to have that passwords database. I personally replicate mine between multiple computers and also publish it to my web server on someone else's infrastructure. I will never forget another password again, because I will never need to know another password again. I will also always know what accounts I have or whether I've used a particular service before.
  - sunnybeetroot 3y ago
    One step further, buy a domain that offers email forwarding. When you sign up to websites, use this domain and have your email forwarded to whatever free email provider you choose. If you change providers you only need to change your domain forwarding email.

whynotmaybe 3y ago

Does that mean that all the images in my blog posts on blogger will disappear? So troubled to hear that a product I never used might disappear and that the impact might be massive for me... or not.

megamike 3y ago
it appears that way what a mess!!

predictabl3 3y ago

I have been getting those photos-cdp emails and noticed your name. I wait in earnest.

bexterw21 3y ago

[flagged]

bexterw21 3y ago

[flagged]

amai 3y ago

Does that mean blogger will also be shut down soon?