5 ms·

The storage requirements are tremendous, though, right? If I want to have access to everything that's been shared with me, I have to store it all. In the case

by ISL 10y ago

The storage requirements are tremendous, though, right? If I want to have access to everything that's been shared with me, I have to store it all. In the case of images, the storage burden can get large quickly.

crypt0x 10y ago
Well.. I've been on there for quite some time, granted it's been not mega active but here is a rundown of how much it took until now: there is the main sigchain database, which stores all the messages (following, posts, ....) which is now 150megs in size and there is the blobs (binary attachments like images) which is about 500megs in size. YMMV depending on how many catpictures your friend share ofc. The flipside to your remark is, that it is fully offline capable and I'm perfectly happy with that. Also: contrast it with how much space a thunderbird profile takes up.
- rattray 10y ago
  How would that change if you had, say, 5,000 friends – the fb limit, which some people do reach – who were posting multimedia content multiple times a day (which happens on fb)? Is the protocol set up in such a way as to enable easy, automatic deletion of old data from local devices, while still storing them for easy search/scroll-based access on the Pub servers?
  - deleted 10y ago
    [deleted]
- NoGravitas 10y ago
  That's going to be a bigger problem on mobile than it is on desktop (I mean, not literally those amounts, but the amounts you'd get from a busier feed).
throaway23 10y ago
The entire stackexchange and English Wikipedia dumps including all media is less than 90 Gig. Even low end cell phones have men expansion slots to 128 gig. Whatever you plan to do socially maintaining a local copy is not a storage issue. Non of the cloud ppl will tell you that though.
- stinkytaco 10y ago
  But I bet Facebook is much larger. If this were to really meet my needs (i.e. people I actually know start using it regularly), I can see this becoming an issue that needs to be solved, especially on mobile. Bitcoin ran into this issue as well, the need for a client to get the whole blockchain. I can see some solutions once cross device identity is done where I get a small amount of the network, perhaps the most recent, then it syncs up with the larger storage on my home PC later.
  - mr_spothawk 10y ago
    > But I bet Facebook is much larger. Thank goodness Facebook isn't what I want my social feed to look like... all those GIFs and garbage updates. Also, I suppose if you're linking out of the SocialApp and into the Web, that most of the content is just "messages". > then it syncs up with the larger storage on my home PC later. I can hardly wait for devices that work this way.
    - stinkytaco 10y ago
      > Thank goodness Facebook isn't what I want my social feed to look like... all those GIFs and garbage updates. Sure, but I have few enough friends as is. I know literally no one who would use this, as neat as it is. Bootstraping a social network is hard for both developers and users, but once it gets going, storage requirements would rise fast.
staltz 10y ago
There are basically two types of storage. Logs and blobs. Logs were described in the blog post, but blobs weren't. Blobs are mostly images that type of stuff, and are stored in leveldb. It can easily get to 1Gb or more. The trick is that blobs aren't sigchained, so they could be garbage collected, and that is something that we're working on. Logs can't be garbage collected, but they grow slower than blobs do, and are usually around 100 Mb or less.
- deleted 10y ago
  [deleted]
  - deleted 10y ago
    [deleted]
cookiecaper 10y ago
Yeah, I've been working on something similar to this off and on. Functionally, the mechanics need to be similar to a distributed filesystem like Tahoe-LAFS or Freenet. Content has to be entrusted to the swarm.
lj3 10y ago
Does Scuttlebutt intend to store every post forever? Or could posts 'expire' and get deleted, like on Usenet? It would be on you to save the content you wanted to have long term. Somebody could always take on the burden of capturing an archive of the whole thing in perpetuity and provide web access to it, like the archive.org does for usenet.
- JdeBP 10y ago
  Note that after the turn of the 21st century, people were not expiring non-binaries posts on Usenet. I observed in 2011 that HighWinds Media had not expired any non-binaries postings since 2006, and that Power Usenet had not expired a non-binaries posting for eight years ("3013+ days text retention" was in its advertising at the time). People effectively just turned non-binaries expiry in Usenet off, in the first few years of the 21st century. I did on my Usenet node, too. I observed then that the Usenet nodes' abilities to store posts had far outstripped the size of the non-binaries portion of a full Usenet feed, which was only a tiny proportion of the full 10TiB/day feed of the time.
  - lj3 10y ago
    You're comparing apples and oranges. Usenet was federated across beefy corporate servers. No single user had to walk around with the entire Usenet archive on their laptop.
    - JdeBP 10y ago
      It was your comparison, note. I'm simply pointing out the error in the premise of your question.
      - lj3 10y ago
        I wasn't comparing anything. I was proposing a suggestion and only mentioned usenet as a means of explanation. > I'm simply pointing out the error in the premise of your question. No, you made a non-sequitur factual post about Usenet. I see no actual error pointed out. The fact that Usenet stopped expiring non-binary posts after most of their traffic fled to other services is not a valid argument against possibly using the feature in a peer to peer distributed social network.
mstade 10y ago
I was thinking the same thing, and I don't know enough (or anything really) about this to comment to this, but my second thought was that this probably works like bittorrent, where you don't need all of a file to make sense of the individual pieces. Let's say for instance that the file you're downloading is a long text file containing a novel, but all you care about is chapter 3. Then all you need are the pieces for chapter 3 – the rest can stick around in the ether somewhere. This is harder to do with bags of bytes obviously – how do you know which bytes belong to chapter 3? – but if the pieces are self contained messages where you don't need either the previous or the next to make sense of it, then it should be trivial to link to them and the distribution could work like this. Whether it actually works like this or not I have no idea. Sounds like an interesting project anyway!
gregdeveloper 10y ago
They should break it up by time. For example your database only syncs the past month or so as needed and you can choose to request more if necessary. Someone mentioned 150MB right now. That thing is going to get massive eventually.
- cel 10y ago
  The quoted 150 MB is for all messages in one's network. On my local node there are 91773 messages, going back roughly a year and a half, taking up 147 MB - of which about 72 MB is the actual messages, and the rest are indexes. gzipped, the 72 MB of messages goes down to 29 MB.

by ISL 10y ago

crypt0x 10y ago

Well.. I've been on there for quite some time, granted it's been not mega active but here is a rundown of how much it took until now: there is the main sigchain database, which stores all the messages (following, posts, ....) which is now 150megs in size and there is the blobs (binary attachments like images) which is about 500megs in size. YMMV depending on how many catpictures your friend share ofc. The flipside to your remark is, that it is fully offline capable and I'm perfectly happy with that. Also: contrast it with how much space a thunderbird profile takes up.

rattray 10y ago
How would that change if you had, say, 5,000 friends – the fb limit, which some people do reach – who were posting multimedia content multiple times a day (which happens on fb)? Is the protocol set up in such a way as to enable easy, automatic deletion of old data from local devices, while still storing them for easy search/scroll-based access on the Pub servers?
- deleted 10y ago
  [deleted]
NoGravitas 10y ago
That's going to be a bigger problem on mobile than it is on desktop (I mean, not literally those amounts, but the amounts you'd get from a busier feed).

throaway23 10y ago

The entire stackexchange and English Wikipedia dumps including all media is less than 90 Gig. Even low end cell phones have men expansion slots to 128 gig. Whatever you plan to do socially maintaining a local copy is not a storage issue. Non of the cloud ppl will tell you that though.

stinkytaco 10y ago
But I bet Facebook is much larger. If this were to really meet my needs (i.e. people I actually know start using it regularly), I can see this becoming an issue that needs to be solved, especially on mobile. Bitcoin ran into this issue as well, the need for a client to get the whole blockchain. I can see some solutions once cross device identity is done where I get a small amount of the network, perhaps the most recent, then it syncs up with the larger storage on my home PC later.
- mr_spothawk 10y ago
  > But I bet Facebook is much larger. Thank goodness Facebook isn't what I want my social feed to look like... all those GIFs and garbage updates. Also, I suppose if you're linking out of the SocialApp and into the Web, that most of the content is just "messages". > then it syncs up with the larger storage on my home PC later. I can hardly wait for devices that work this way.
  - stinkytaco 10y ago
    > Thank goodness Facebook isn't what I want my social feed to look like... all those GIFs and garbage updates. Sure, but I have few enough friends as is. I know literally no one who would use this, as neat as it is. Bootstraping a social network is hard for both developers and users, but once it gets going, storage requirements would rise fast.

staltz 10y ago

There are basically two types of storage. Logs and blobs. Logs were described in the blog post, but blobs weren't. Blobs are mostly images that type of stuff, and are stored in leveldb. It can easily get to 1Gb or more. The trick is that blobs aren't sigchained, so they could be garbage collected, and that is something that we're working on. Logs can't be garbage collected, but they grow slower than blobs do, and are usually around 100 Mb or less.

deleted 10y ago
[deleted]
- deleted 10y ago
  [deleted]

cookiecaper 10y ago

Yeah, I've been working on something similar to this off and on. Functionally, the mechanics need to be similar to a distributed filesystem like Tahoe-LAFS or Freenet. Content has to be entrusted to the swarm.

lj3 10y ago

Does Scuttlebutt intend to store every post forever? Or could posts 'expire' and get deleted, like on Usenet? It would be on you to save the content you wanted to have long term. Somebody could always take on the burden of capturing an archive of the whole thing in perpetuity and provide web access to it, like the archive.org does for usenet.

JdeBP 10y ago
Note that after the turn of the 21st century, people were not expiring non-binaries posts on Usenet. I observed in 2011 that HighWinds Media had not expired any non-binaries postings since 2006, and that Power Usenet had not expired a non-binaries posting for eight years ("3013+ days text retention" was in its advertising at the time). People effectively just turned non-binaries expiry in Usenet off, in the first few years of the 21st century. I did on my Usenet node, too. I observed then that the Usenet nodes' abilities to store posts had far outstripped the size of the non-binaries portion of a full Usenet feed, which was only a tiny proportion of the full 10TiB/day feed of the time.
- lj3 10y ago
  You're comparing apples and oranges. Usenet was federated across beefy corporate servers. No single user had to walk around with the entire Usenet archive on their laptop.
  - JdeBP 10y ago
    It was your comparison, note. I'm simply pointing out the error in the premise of your question.
    - lj3 10y ago
      I wasn't comparing anything. I was proposing a suggestion and only mentioned usenet as a means of explanation. > I'm simply pointing out the error in the premise of your question. No, you made a non-sequitur factual post about Usenet. I see no actual error pointed out. The fact that Usenet stopped expiring non-binary posts after most of their traffic fled to other services is not a valid argument against possibly using the feature in a peer to peer distributed social network.

mstade 10y ago

I was thinking the same thing, and I don't know enough (or anything really) about this to comment to this, but my second thought was that this probably works like bittorrent, where you don't need all of a file to make sense of the individual pieces. Let's say for instance that the file you're downloading is a long text file containing a novel, but all you care about is chapter 3. Then all you need are the pieces for chapter 3 – the rest can stick around in the ether somewhere. This is harder to do with bags of bytes obviously – how do you know which bytes belong to chapter 3? – but if the pieces are self contained messages where you don't need either the previous or the next to make sense of it, then it should be trivial to link to them and the distribution could work like this. Whether it actually works like this or not I have no idea. Sounds like an interesting project anyway!

gregdeveloper 10y ago

They should break it up by time. For example your database only syncs the past month or so as needed and you can choose to request more if necessary. Someone mentioned 150MB right now. That thing is going to get massive eventually.

cel 10y ago
The quoted 150 MB is for all messages in one's network. On my local node there are 91773 messages, going back roughly a year and a half, taking up 147 MB - of which about 72 MB is the actual messages, and the rest are indexes. gzipped, the 72 MB of messages goes down to 29 MB.