13 ms·
Keep Your Stuff, for Life
- brudgers 6y agopast discussion, https://news.ycombinator.com/item?id=18008240 https://news.ycombinator.com/item?id=18008240
- crooked-v 6y agoIt looks like this hasn't had any updates since May 2018.
- recursive 6y agoWhy change it?
- ramzeus 6y agoThe GitHub page has lots of fairly recent updates: https://github.com/perkeep/perkeep https://github.com/perkeep/perkeep
- vaughnegut 6y agoLast commit was in March, mind you. It looks like an interesting project though
- tptacek 6y agoI imagine the author is pretty busy with Tailscale right now.
- Intermernet 6y agoAnd kids ;-)
- na85 6y agoI admire the goal of reliable long term data storage. I don't need to store tweets or 5TB videos, but my current solution is duplicate DVDs with a bunch of par2 files to hopefully ward off bit rot. I feel like there's some room for improvement.
- pmiller2 6y agoDo you store your par2 files on different discs from the data they refer to? Do you have multiple copies? Do you store the discs in a cool, dark place when you're not using them? Do you have another form of backup, preferably offsite? If you do all these things, I think that's about the best you can do with optical media.
- na85 6y agoHaha, I keep the par2s on the same disc, and then I make multiple copies of the same disc and they're in a cardboard box in my basement somewhere.
- jordanthoms 6y agoThe ink on writable optical disks fades over time... So of all the copies were written at the same time, they might all stop working at around the same time and cause problems :)
- Widdershin 6y agoYeah if you're really taking this seriously it's worth investing in an LTO-4 drive and some tapes. There might be a better medium available nowadays but if I seriously wanted to have a piece of data fifty years from now that's where I'd start.
- pmiller2 6y agoReally, any second form of backup would be a good idea. Preferably, it should be stored offsite and separate from any other backups.
- CGamesPlay 6y agoWow, it was really hard to find out what this software does. I finally found a demo at the end of an hour-long talk from 2 years ago: https://youtu.be/PlAU_da_U4s?t=2687 https://youtu.be/PlAU_da_U4s?t=2687 So it seems to have a bunch of scripts to import data into its database from a variety of sources (including cloud services), and provides a search interface to navigate that historical data. And it has a lot of under-the-hood stuff about replication, and it's entirely self-hosted.
- deleted 6y ago[deleted]
- awinter-py 6y agoyeah like a lot of OSS backup products the docs are grandiose without focusing on ease of operation / feature parity with dropbox
- mro_name 6y agono wonder looking at the funding, is it?
- TuringTest 6y agoYou don't need funding for focusing on ease of operation, it's a mindset. Either you're user-centered, or you're not.
- reallydontask 6y ago>Either you're user-centered, or you're not. It really depends on the audience for the product, targeting regular users does require more thought on the UX, etc which almost invariably means more funding.
- michaelt 6y agoI don't think the website targets the wrong audience - just the copywriter was so involved in the project they forgot to make what they're offering clear. When someone says "permanently keep your stuff, for life" do they mean some sort of pay-once eternal backup, like permanent.org? Something censorship-resistant, like Freenet? Something peer-to-peer and distributed-ledger based like Filecoin? Backing up data locked up in cloud services? Converting obscure file formats into ones with more longevity? Bypassing defunct DRM? Activism against civil forfeiture?
- Aperocky 6y agoIt's like a poor man's version of AWS Glacier
- natural219 6y agoAh, nostalgia :). It seems like with the recent wave of news about social media migrations (reddit, facebook, twitter, twitch, tiktok), people are hopefully starting to get more and more warmed up to the idea of protocolization of their social data. But most of the projects doing it are still just too immature. Solid, Perkeep, Blockstack, etc. just seem like vaporware. Seems like the only serious projects in use are Matrix, Urbit, and ActivityPub/Mastodon. But I haven't checked in with the decentralization scene in a while.
- lukecameron 6y agoI want that protocolization too, although I don't hold out much hope that the monopolies in place can be broken, outside of fairly radical regulation. To add to your list, there is also Secure Scuttlebutt [1] which has had a decent userbase over the past few years, and Planetary [2] which is a funded iOS client for it. I think in general they all suffer from the chicken-egg problem and will need some reason for enough people to switch to be able to build a userbase. There isn't really any "novel hook" like tiktok, twitter, whatsapp, instagram, snapchat, etc have had in the past. [1] https://scuttlebutt.nz/ https://scuttlebutt.nz/ [2] https://planetary.social/ https://planetary.social/
- natural219 6y agoAh, I forgot about SSB. Very unique project and interesting people working on it.
- asdkhadsj 6y agoMan, I love the idea of Scuttlebutt but I hate the developer UX. I'm writing some apps that I wanted to put on SSB but have all but given up on the idea. Something about SSB, as a dev, leaves me with a lot of questions and no idea where to even get answers from. So I'll write my app outside of SSB, hopefully in a way that's mostly compatible, and possibly with future integration. I may also toy with an SSB-like protocol myself, as the fundamentals of SSB is a work of art imo. I really enjoy what Gossip brings to the table, and how SSB focuses on human->human relationship to bring P2P to the table.
- 6y ago
- ahupp 6y agoIt sounds like this preserves your data from lots of different services. This is something I need! But I couldn't figure out what it actually supports. Suggestion: the very first paragraph should describe the specific inputs it can handle.
- dabeeeenster 6y agospent a minute looking and couldnt find the list...
- eexan 6y agoFor the most part, it's just an object storage (think Amazon S3). Content addressable (think Git): you put an object (file bytes) in, and you can get it out by its hash, that's it. There are some bits (permanodes and claims) for adding metadata to objects (filename, timestamp, geo location and other attributes, I think even arbitrary jsons) and for authentication/sharing. A few really cool bits around modularity: blob servers can be composed over network - you can transparently split your blob storage over multiple machines, databases, cloud services, set up replication, maybe encryption (unclear to me if it works or not). Importing data from different services is not really its core competency, at least not yet. It can ingest anything you can put on your file system and there are importers for a few third-party services (see https://github.com/perkeep/perkeep/tree/master/pkg/importer https://github.com/perkeep/perkeep/tree/master/pkg/importer ), but that's about it
- kemonocode 6y agoThank you so much for a description of what it actually does which the website seems to struggle so much to convey. One thing that I'm still trying to figure out is, if you do happen to know: how does it handle data deduplication (if at all)? How about redundancy and backups? I've been glancing over the docs and I do see mention of replication to another Perkeep instance but that's not quite what I'm looking for.
- eexan 6y agoDeduplication is naturally handled by content-addressable property of this object store: the address of each object is its cryptographic hash, SHA224 in Perkeep. So if you try to put a duplicate copy, you'll find that the address at which you try to put it at is already occupied by the first copy. Perkeep assumes that you never delete anything (deletion is even simply not implemented, not even for garbage collection/compaction purposes), so if you see that one copy of an object was already put, you can discard any further puts as no-ops. Then there is also some logic to chunk large objects into small pieces or "blobs". These small chunks are actually what the storage layer works with, rather than with the original unlimited-length blobs that the user uploaded. Chunking helps to space-efficiently store multiple versions of same large file (say, a large VM image) - the system only needs to store the set of unique chunks, which can be much smaller than N full but slightly-different copies of the same file. But I personally I find that it deteriorates its performance to the point of making it unusable for my use case of multi-TB multi-million-files storage of immutable media files. If chunking/snapshotting/versioning is important for your use case, I'd look more towards backup-flavored tools like restic, which share many of these storage ideas with Perkeep. Redundancy and backup is handled by configuring storage layer ("blobserver") to do it. Perkeep's blobservers are composable - you can have leaf servers storing your blobs, say, directly in a local filesystem directory, remote server over sftp, or an S3 bucket, and you can compose them using special virtual blobserver implementations into bigger and more powerful systems. One such virtual blobserver is https://github.com/perkeep/perkeep/blob/master/pkg/blobserver/replica/replica.go https://github.com/perkeep/perkeep/blob/master/pkg/blobserve... - which takes addresses of 2+ other blobservers and replicates your reads and writes to them.
- solarkraft 6y agoPerkeep is super interesting. I have been looking at it as a media database (photos, movies and such, powerful tag-based search, all downloaded on demand), but some things regarding that really hurt: - The data store messes with my files (yes, there's a FUSE mount, but eh, having to adopt a special data store always makes me feel weird since it usually comes with performance and compatiiblity implications, there are also many other block-chopping data stores, for example in IPFS). - The last time I checked there was no way to delete something. This is okay for Tweets I guess, but if I commit a 3h video I later realize to just be too large or a photo I end up not really wanting around - well, oops. I have huge respect for Brad Fitzpatrick in general, of course, and especially for creating this. Recent velocity has seemilgly been relatively low, however: https://news.ycombinator.com/item?id=22161812 https://news.ycombinator.com/item?id=22161812
- andai 6y ago> ... my time is tempered by 2.5 and 0.5 year old kids. ... I'll pick up my involvement again as kids get a bit older. > We have no plans to abandon it.
- cpach 6y agoDoes anyone know how mature Perkeep is? Is anyone using it regularly? Would love to hear if there is anyone who has experience with it.
- xafloc 6y agoI'm guessing not very: "The latest release is 0.10 ("Bellingham"), released 2018-05-02"
- eexan 6y agoVery immature, just have a look at its extensive absent documentation. The best bit that describes the state of things: "If you're a programmer or fairly technical, you can probably get it up and running and get some utility out of it". So much to show off for 7 years of development, I'm pretty skeptical of its future. But some of the ideas are pretty cool, like composable blob servers.
- eexan 6y agoI tried perkeep a while ago. While the ideas are cool, the implementation is meh: I added a single 2.7G ubuntu iso - it took 5 minutes to ingest it (on a tmpfs!), and turned it into 45k(!) little chunks, wtf is up with that? At this rate indexing my multiple terabytes of data is going to take it days and I don't even want to think how much seek time it's going to need to store its repo on a spinning HDD.
- shock 6y ago> I added a single 2.7G ubuntu iso - it took 5 minutes to ingest it (on a tmpfs!), and turned it into 45k(!) little chunks, wtf is up with that? Ingest times correlate linearly with file sizes because it needs to compute the blobref (which is a configurable hash) for all the blobs (chunks as you call them). Splitting in blobs/chunks is necessary because a stated goal of the project is to have snapshots by default when modifications are done. Doing snapshots/versioning without chunking would be very inefficient.
- eexan 6y agoReading the docs, snapshotting/versioning doesn't strike me as a major feature of perkeep. It's more important and appropriate in the domain of backup software (e.g. restic/attic/borg) and where you'd want it together with delete functionality to reclaim space. But perkeep's focus, as I understand it, is more on managing an unstructured collection of immutable things (e.g. photo archive), rather than being a tool to back up your mutable filesystem. So I'm not sure they made a good design decision to chunk the sh*t out of my files, which really kills the performance on large files and especially on spinning disks.
- deleted 6y ago[deleted]
- aabbcc1241 6y agoexporting/mirror content from clearnet social network to zeronet is also interesting. For popular content, you'll see it has many seeds, instead of likes
- darkwater 6y agoLast released version 0.10 is from May 2018. Is this project still alive? Last commit is from March 11 2020 so maybe they are just "slow" at releasing.
- PurpleRamen 6y agoLast commit was just a simple bugfix in documentation. The last juicy commits were in december 2019 and https://github.com/perkeep/perkeep/graphs/code-frequency https://github.com/perkeep/perkeep/graphs/code-frequency makes the impression that activity has slowed down in 2019 significantly. Maybe they reached the point of good enough, but the high number of open issues and mergerequests is still problematic.
- andai 6y agoThey reached the point of having kids :) https://news.ycombinator.com/item?id=22161812 https://news.ycombinator.com/item?id=22161812
- sukilot 6y agoDevelopment paused when Brad had a kid in 2018. It probably needs someone new to adopt it.
- dorcassmith 6y agoIt is important to seek english language essay writing services and english language coursework writing services since students find help when they visit English Language Writing Services. https://researchpapers247.com/english-language-writing-services/ https://researchpapers247.com/english-language-writing-servi...
- heybrandons 6y agoWatching the LinuxFest talk now, this looks really neat!
- 0xCMP 6y agoFor those curious "what does this do" and "how does this work" the presentation on the home page (Video + Slides) helps a lot.
- etskinner 6y agoHow is this better than a plain old filesystem with bitrot compensation? What does this have that my btrfs or ZFS filesystem (with parity) doesn't?
- betimsl 6y agoHow is this different from Upspin?
- mam2 6y agoThat would be nicer to be able to "delete all your stuff, for life" in our era than to keep it
- alexr243 6y agoThere's a blockchain startup called arweave which is trying to do exactly this.
- alexr243 6y agoThere's a blockchain startup called arweave which is trying to do exactly this