3 ms·

Does Scuttlebutt intend to store every post forever? Or could posts 'expire' and get deleted, like on Usenet? It would be on you to save the content you wanted

by lj3 10y ago

Does Scuttlebutt intend to store every post forever? Or could posts 'expire' and get deleted, like on Usenet? It would be on you to save the content you wanted to have long term. Somebody could always take on the burden of capturing an archive of the whole thing in perpetuity and provide web access to it, like the archive.org does for usenet.

JdeBP 10y ago
Note that after the turn of the 21st century, people were not expiring non-binaries posts on Usenet. I observed in 2011 that HighWinds Media had not expired any non-binaries postings since 2006, and that Power Usenet had not expired a non-binaries posting for eight years ("3013+ days text retention" was in its advertising at the time). People effectively just turned non-binaries expiry in Usenet off, in the first few years of the 21st century. I did on my Usenet node, too. I observed then that the Usenet nodes' abilities to store posts had far outstripped the size of the non-binaries portion of a full Usenet feed, which was only a tiny proportion of the full 10TiB/day feed of the time.
- lj3 10y ago
  You're comparing apples and oranges. Usenet was federated across beefy corporate servers. No single user had to walk around with the entire Usenet archive on their laptop.
  - JdeBP 10y ago
    It was your comparison, note. I'm simply pointing out the error in the premise of your question.
    - lj3 10y ago
      I wasn't comparing anything. I was proposing a suggestion and only mentioned usenet as a means of explanation. > I'm simply pointing out the error in the premise of your question. No, you made a non-sequitur factual post about Usenet. I see no actual error pointed out. The fact that Usenet stopped expiring non-binary posts after most of their traffic fled to other services is not a valid argument against possibly using the feature in a peer to peer distributed social network.
      - JdeBP 10y ago
        "like on Usenet" is definitely a comparison. If you don't see an error in your premise being pointed out, then you need to put your "posts expire and get deleted, like on Usenet" right up against "people were not expiring non-binaries posts on Usenet" until the penny drops. Then you need to notice the point, already made by others as well, that the premise of ISL's question is erroneous, too. The storage requirements are not necessarily "tremendous", if one actually learns from the past. Again, your comparison to Usenet needs to involve considering how Usenet treated binaries and non-binaries very differently. (One can look to experience of the WWW for this, too, and consider the relative weights in HTTP traffic of the "images" that ISL talks about and the non-binary contents of the WWW. But your comparison to Usenet does teach the same thing.) Your and ISL's whole notion, that everything is going to get tremendously big and so everything will need to be expired, rather flies in the face of what we can see from history actually happened in systems like this, such as the one that you made your comparison to. Usenet did not expire and delete non-binaries posts. By making this comparison and then trying to pretend that it's someone else's non-sequitur you are closing your eyes to the useful lessons to actually learn from your comparison. Usenet, and the Wayback Machine, and the early WWW spiders, and Stack Exchange, and Wikipedia with all of its talk pages, and Fidonet in its later years (when hard disc sizes became large enough), all teach that in fact one can get away with keeping all of the "non-binary" stuff indefinitely, or at least for time scales on the order of decades, because that is not where the majority of the storage and transmission costs is. People have already danced this dance, several times, and making a distinction between the binary and the non-binary stuff and not fretting overmuch about the latter when one looks at the figures is generally where it ends up.
        deleted 10y ago
        [deleted]
- cel 10y ago
  The distinction of binary and non-binary posts on Usenet is paralleled by the separation of messages and blobs on Scuttlebutt. As staltz [explained](https://news.ycombinator.com/item?id=14051181 https://news.ycombinator.com/item?id=14051181), we can garbage collect ("expire") blobs, but not message logs (although a client could do so with the current APIs, it would have security/trust and UI implications, and I'm not aware of any clients doing so). We are also basically betting on the size of our message logs to generally grow slower than our individual storage capacities, and it is interesting to know that that worked for Usenet too. For blobs, we will likely develop some garbage collection or expiring approaches. Since the network is radically decentralized, each participant can choose their own retention policy. You can, in fact, delete all your blobs (`rm -rf ~/.ssb/blobs`) and assuming some peers have replicated them, your client will just fetch them again as you need them.
staltz 10y ago
Check other comments here in HN thread about "blobs" and garbage collection. But also, there is the easy possibility of just starting a new account. In fact, substack did this, we refer to his old account as "deadsubstack".

by lj3 10y ago

JdeBP 10y ago

Note that after the turn of the 21st century, people were not expiring non-binaries posts on Usenet. I observed in 2011 that HighWinds Media had not expired any non-binaries postings since 2006, and that Power Usenet had not expired a non-binaries posting for eight years ("3013+ days text retention" was in its advertising at the time). People effectively just turned non-binaries expiry in Usenet off, in the first few years of the 21st century. I did on my Usenet node, too. I observed then that the Usenet nodes' abilities to store posts had far outstripped the size of the non-binaries portion of a full Usenet feed, which was only a tiny proportion of the full 10TiB/day feed of the time.

lj3 10y ago
You're comparing apples and oranges. Usenet was federated across beefy corporate servers. No single user had to walk around with the entire Usenet archive on their laptop.
- JdeBP 10y ago
  It was your comparison, note. I'm simply pointing out the error in the premise of your question.
  - lj3 10y ago
    I wasn't comparing anything. I was proposing a suggestion and only mentioned usenet as a means of explanation. > I'm simply pointing out the error in the premise of your question. No, you made a non-sequitur factual post about Usenet. I see no actual error pointed out. The fact that Usenet stopped expiring non-binary posts after most of their traffic fled to other services is not a valid argument against possibly using the feature in a peer to peer distributed social network.
    - JdeBP 10y ago
      "like on Usenet" is definitely a comparison. If you don't see an error in your premise being pointed out, then you need to put your "posts expire and get deleted, like on Usenet" right up against "people were not expiring non-binaries posts on Usenet" until the penny drops. Then you need to notice the point, already made by others as well, that the premise of ISL's question is erroneous, too. The storage requirements are not necessarily "tremendous", if one actually learns from the past. Again, your comparison to Usenet needs to involve considering how Usenet treated binaries and non-binaries very differently. (One can look to experience of the WWW for this, too, and consider the relative weights in HTTP traffic of the "images" that ISL talks about and the non-binary contents of the WWW. But your comparison to Usenet does teach the same thing.) Your and ISL's whole notion, that everything is going to get tremendously big and so everything will need to be expired, rather flies in the face of what we can see from history actually happened in systems like this, such as the one that you made your comparison to. Usenet did not expire and delete non-binaries posts. By making this comparison and then trying to pretend that it's someone else's non-sequitur you are closing your eyes to the useful lessons to actually learn from your comparison. Usenet, and the Wayback Machine, and the early WWW spiders, and Stack Exchange, and Wikipedia with all of its talk pages, and Fidonet in its later years (when hard disc sizes became large enough), all teach that in fact one can get away with keeping all of the "non-binary" stuff indefinitely, or at least for time scales on the order of decades, because that is not where the majority of the storage and transmission costs is. People have already danced this dance, several times, and making a distinction between the binary and the non-binary stuff and not fretting overmuch about the latter when one looks at the figures is generally where it ends up.
      - deleted 10y ago
        [deleted]
cel 10y ago
The distinction of binary and non-binary posts on Usenet is paralleled by the separation of messages and blobs on Scuttlebutt. As staltz [explained](https://news.ycombinator.com/item?id=14051181 https://news.ycombinator.com/item?id=14051181), we can garbage collect ("expire") blobs, but not message logs (although a client could do so with the current APIs, it would have security/trust and UI implications, and I'm not aware of any clients doing so). We are also basically betting on the size of our message logs to generally grow slower than our individual storage capacities, and it is interesting to know that that worked for Usenet too. For blobs, we will likely develop some garbage collection or expiring approaches. Since the network is radically decentralized, each participant can choose their own retention policy. You can, in fact, delete all your blobs (`rm -rf ~/.ssb/blobs`) and assuming some peers have replicated them, your client will just fetch them again as you need them.

staltz 10y ago

Check other comments here in HN thread about "blobs" and garbage collection. But also, there is the easy possibility of just starting a new account. In fact, substack did this, we refer to his old account as "deadsubstack".