11 ms·
Time to dust off my side project, Timeliner [1] which downloads your Google Photos (and other content from various services) and indexes them in a local SQLite
by mholt 6y ago
Time to dust off my side project, Timeliner [1] which downloads your Google Photos (and other content from various services) and indexes them in a local SQLite DB.
Then after backing them up locally, I won't feel bad about deleting them from the cloud later to free up space.
I've been using Timeliner for a while but need to update it. New maintainers welcomed, if you're interested!
(One major "oof" is that the Google Photos API strips geolocation data, so unfortunately coordinates are lost when using this method. There's discussion about using Takeout as a workaround, or even automating web browser interactions, but those have their own problems too.)
[1]: https://github.com/mholt/timeliner https://github.com/mholt/timeliner
- akx 6y agoAnd then you're responsible for backing that local content somewhere else... maybe the cloud?
- mittermayr 6y agoThe Google Photos interface itself isn't that big of a product (I think), if you can lose some of the sharing features, I guess it'd be quite an easy app to make that can be run off a v-server or Amazon at considerably less long-term trouble. The big big issue is the seamless auto-upload from phone to Google Photos, that's the fantastic feature that locked me in.
- majora2007 6y agoMegaUpload and Plex offer both of those things. Using Plex allows you to control the full storage and interface, then you can sync those to a cloud service or keep on a HDD yourself.
- Normille 6y agoThere's also Yandex Disc [0] which is still offering unlimited photo uploads from mobile devices. Although it wouldn't surprise me if they followed Google's lead on this and curtailed their offering as well. As they seem to pretty much mirror everything else Google does. Yandex Disc also has an AI which allows you to search for objects within your photos. [0]https://yandex.com/promo/disk/unlim https://yandex.com/promo/disk/unlim
- llimos 6y agoTrue but their AI adds a lot of value. Searching for people is invaluable (though to be fair if you have the files locally, Windows Photos does that too.) Searching by item in the photo is less useful but I can imagine people use it now and again. And the automatically created videos are a nice bonus. But yeah, not enough for me to pay them for it. If I'm going to pay someone, I'll pay someone else.
- walrus01 6y agothe long term cost of buying something like a 12TB USB3.0 external hard drive and periodically letting it synchronize from your master copies, then unplugging it and storing it at a trusted family member's house is a great deal less than paying for 12TB+ of "cloud" storage indefinitely forever.
- fouronnes3 6y agoSure, except no drive sync software is as convenient or user friendly as google photos.
- deleted 6y ago[deleted]
- fletchowns 6y agoGoogle Backup & Sync makes it very cumbersome to retain a local copy of your photos. They used to sync them to Google Drive, so you could at least use Backup & Sync to have local copies. However, they got rid of that feature some time in the last year. I had to switch to using a third party app to sync my "photo reel" from my phone to Google Drive: https://play.google.com/store/apps/details?id=com.ttxapps.drivesync&hl=en_US https://play.google.com/store/apps/details?id=com.ttxapps.dr... Then I use Google Backup & Sync to create a local copy on my desktop, scheduled SyncToy job to copy it to the NAS, and the NAS has a scheduled job to backup to Backblaze B2. In my setup, these irreplaceable photos & videos exist on my phone, Google, my desktop, my NAS, and Backblaze. If you are relying on Google Backup & Sync to restore a large amount of data after a data loss event, you are in for a world of pain and disappointment. It simply does not handle any significant amount of data well. I had to stop backing up "My Computer" with Backup & Sync and just switched to using Google Drive. The files in Google Drive seem to be the only way to "restore" a large amount of files after a data loss event. If they are backed up under "My Computer" in Google Drive and you have a decent amount of data (50GB+), you will have no way to restore these files. The web interface will simply time out when you try to download them. The Backup & Sync client won't even attempt to download these files, even if you try the dead-end workarounds suggested on the web. You won't be able to drag them into your "Google Drive" folder as a workaround either. The whole thing is atrocious. Google Backup & Sync is a fucking joke, and Google should be ashamed of themselves for releasing such a shitty product which does not actually help consumers protect against data loss. Consumers will only realize this at the point they are fucked and trying to restore their data.
- heroprotagonist 6y agoA demo or screenshots would go a long way here. The project seems like it could be very interesting, but there's too much effort required to perform an initial evaluation.
- mholt 6y agoI know, but it's not a graphical app. Contributions welcomed to lower the barrier to entry. Use it, contribute to it, or leave it, I guess!
- skybrian 6y agoAfter downloading, can you upload to a different cloud photo service? I'd love to have a way to sync them automatically, and having a non-cloud backup would be nice too.
- mholt 6y agoYou can do whatever you'd like. They're just files like any other.
- skybrian 6y agoSure, but doing it automatically (and handling things like making sure they're not duplicated) would be more convenient over cobbling together a solution.
- charlieegan3 6y agoYou might find rclone helpful. It has support for google photos.
- thinkloop 6y ago> The current google API does not allow photos to be downloaded at original resolution. This is very important if you are, for example, relying on "Google Photos" as a backup of your photos. You will not be able to use rclone to redownload original images. You could use 'google takeout' to recover the original photos as a last resort https://rclone.org/googlephotos/#limitations https://rclone.org/googlephotos/#limitations
- johnghanks 6y agowhy would anyone want this? you lose literally every benefit of Google Photos.
- mholt 6y agoHmm, good point, I guess nobody has ever lost access to their Google Account or (in the future) run out of storage space. After all, they're just photos, not important memories or anything.
- dedosk 6y agoIntegrate the takeout somewhat into www.digikam.org!
- scrollaway 6y agoMatt, did you see Promnesia? https://beepb00p.xyz/promnesia.html https://beepb00p.xyz/promnesia.html There are a lot of converging ideas out there, for "self-hosted NSA of yourself" type software which gather your own data from services you use and index it / make it more accessible to users. Would love to see something serious come out of it. I followed Timeliner early on and really like the idea, but yeah it's a bitch to get traction for this sort of thing. You need people willing to implement backends for basically everything, then maintain them.
- mholt 6y agoYep, I've also seen Perkeep and several other similar projects. They're all a bit different and I'm happy with something simple and bare-bones, it just has to be good at getting the data off.
- flashgordon 6y agoThis is awesome. What would it take to hook up the storage to say block storage (say EBS or GCS) so that we can have more durabality interms of storage instead of having to manage a NAS at home? For me there have been a whole bunch of needs (photos, videos, docs) where storage has been the major stumbling block.
- mholt 6y agoYou can use rclone to copy the files back up to any cloud you want. Timelines are just files and folders.
- fletchowns 6y agoIt's so annoying that Google strips the geolocation data. What would be the reason for this? It has been frustrating following along with the ticket that goes nowhere for years: https://issuetracker.google.com/issues/80379228 https://issuetracker.google.com/issues/80379228
- andybak 6y agoIronic as it sounds - privacy?
- irrational 6y agoIronic indeed. I can’t help but wonder if Google stores the geolocation data privately where only they can access it for their own data mining uses before removing it from the photos.
- ciarannolan 6y agoI would start with the assumption that Google is collecting and storing every single possible data point about every single thing you share with them, including this. I have zero doubt that they store this info privately for their own use.
- chrisbroadfoot 6y agoZero doubt? Wow, wish I had such confidence to make claims about things I know little about. Each photo's geolocation data is stored. Of course. It is shown in the UI, as well as indexed so you can search by location. For example, I can search for "photos of dogs in Portland" and it'll show all the pics of dogs I've taken in Portland.
- deleted 6y ago[deleted]
- ehsankia 6y ago
- jcims 6y agoI have one primary collection but I also have probably 30TB of old drives lying around with photos scattered all about. Is there something I can do to mount those drives, grab all image files and then dedupe/catalog them (at least by date)? Ideally this would work with videos as well.
- avgDev 6y agoI bought a Synology NAS mainly for images/videos/personal files and run hard drives in RAID setup. You could explore other NAS solutions like QNAP or open source.
- jcims 6y agoI've stopped and started shopping for a NAS at least five times now. One of these days i'll pull the trigger.
- alliao 6y agoSynology's photo station is slightly jarring... the whole indexing operation isn't optimized at all so if you have like 500k photos it's going to be spinning for a while.
- walterbell 6y agoThere's an ancient-but-maintained image cataloger / asset manager called NeoFinder, available on Windows and Mac. Would love to find an OSS equivalent. https://cdfinder.de/ https://cdfinder.de/
- jcims 6y agoWill take a look, thank you!
- x87678r 6y agoSame problem. The other issue is having a 2 cameras and 2 phones in the family and grouping together events. Its easier just to save them all and not try.
- fwiwm2c 6y agoWhy not just pay $30/yr to get access to all the features which Google Photos (and other Google services) offer? I know $30 is not the same everywhere but if you are in US (and assuming you are in tech given you are on HN), you'd spend that much in a dinner without thinking much. There is something about paying about online services which brings a lot of resistance in us (me included but I think I am getting over it)
- ENGNR 6y agoRealistically... if you wanted to pay $30/year for a set and forget photo album, is Google the best place to do that? Given how they often they deprecate things and how hard I've heard it is to restore a disabled account, I feel like some kind of smaller company dedicated to photo storage would be much better anyway
- ghaff 6y agoPersonally I use Flickr and am happy with them.
- Normille 6y agoI'm not sure if you're just stating a personal preference there, or hinting that Flickr are somehow morally superior to Google. But Flickr had their own 'google Photos moment' a few years back, when they were bought out by SmugMug and immediately got rid of their 'free for life' 1TB storage.
- ehsankia 6y agoPhotos is a billion user product, how many of those have they deprecated? The disabled account thing may be a real concern, but statistically it's more rare than getting into a plane crash, yet we still fly planes. Also, if anything, monetizing the product and allowing it to be self-sustaining financially makes it far less likely to be killed. I'd much rather pay for a product that brings me value, since now I'm a real customer get treated differently from a free user. Also AFAIK Google has never killed a paid service either. I would be curious to know what the competition looks like, but I don't know of any other service that lets me do "Show me photos of me and my brother at the beach" and actually return relevant results. The face/object clustering is near flawless, and even works with pets.
- jkingsman 6y agoTakeout sounds like an interesting workaround. I just do a takeout download once a month and figure that if google photos vaporizes or I lose my account I can write a python script to bake back in the JSON metadata that comes with it.
- Xorlev 6y agoExporting via Takeout will retain the geolocation tags. The API isn't the best place to export.
- Roritharr 6y agoI've stumbled upon this problem lately. Is there any software to organize the takeout dump sensibly?
- deleted 6y ago[deleted]
- BoboDupla 6y agobut the result is a mess - why can't Google just give me all my photos in a logical folder structure, but instead I get duplicate folders with .json files and pictures which are in a completely different folder.
- WalterBright 6y ago> indexes them in a local SQLite DB. I don't really understand this. I've been in computing for 45 years now. Programs come and go. I store files I want to keep in the most generic format practical. Storing a file in some program's special format is not a good plan for reading it 40 years from now. For photos, I store them as jpgs in folders named after the year. Within those folders, there might be sub-folders named with a topic, like "disneyland" or "christmas". I'll "tag" photos by selecting a name for the jpg, like "bob and sue.jpg". If more is needed, I'll just add a "notes.txt" file in the folder, with whatever text seems appropriate. I have no worries about ascii text becoming unreadable, and few worries about jpgs becoming unreadable.
- mholt 6y agoPlease don't misunderstand, with your 45 years of computing experience. :) (which I highly respect) I never said the files are stored in sqlite. They are just indexed in sqlite. The files are stored on disk like normal.
- peterwwillis 6y ago> Please don't misunderstand If there's one thing people will always do regardless of age, it's that.
- Retric 6y agoMetadata is more useful with older photos. Right now it’s obvious who’s in them, but in 40 years being able to just put names to faces can be difficult.
- WalterBright 6y agoI just note their names in the file name. If there are too many, I make a copy of the photo, bring it up in an image editor, and type the names of the people next to their heads. It works like a champ.
- ForHackernews 6y agoTBF SQLite format is also extremely unlikely to become unreadable: https://www.sqlite.org/fileformat2.html https://www.sqlite.org/fileformat2.html
- subsaharancoder 6y agoAnother alternative is Perkeep (formerly known as Camlistore) https://perkeep.org/ https://perkeep.org/ and there's even a tool to extract your pics from Google Photos https://github.com/perkeep/gphotos-cdp https://github.com/perkeep/gphotos-cdp and move them to Perkeep.
- yangikan 6y agoHow long does it take to download a terabyte of photos?
- jmonger 6y agoI extracted myself from Google Photos in 2019. Geolocation was a nightmare. I ended up using exiftool to merge the metadata from JSON available in Takeout. https://exiftool.org/forum/index.php?topic=10412.msg55391#msg55391 https://exiftool.org/forum/index.php?topic=10412.msg55391#ms...