34 ms·

See the comment below https://news.ycombinator.com/item?id=15376665 https://news.ycombinator.com/item?id=15376665. It is not true that distributed systems are o

by carolc 9y ago

See the comment below https://news.ycombinator.com/item?id=15376665 https://news.ycombinator.com/item?id=15376665. It is not true that distributed systems are only good for static content or "append-only" data. "Mutable systems" can be built on top of immutable systems.

vog 9y ago
I agree with you, but your argument is deeply flawed. There's quite a far stretch between "Y can be built on top of X" and "X is good for Y". To provide an argument that might fill this gap: Most systems don't actually have a huge amount of data. Look at the data size and data growth of CRMs, special-purpose wikis, and so on: These are mostly smaller than 500 MB (excluding static content like images), and grow by less than 1MB even on a busy day. And that's the uncompressed size. Also, most systems, despite being mutable, actually want (or need) an audit trail. So these are really append-only systems which merely have a "mutable look and feel" to the user.
- carolc 9y ago
  Agreed that many CRMs etc. don't have a lot of data. And that's actually good, it makes the database size very manageable in the context of trustless, distributed networks. I'm not following the logic of the argument here though, jumping from "X is good for Y" to "...don't actually have huge amount of data", perhaps you can elaborate? With a merkelized append-only log (immutable DAG), there's always an audit trail. I agree with your point about "mutable look and feel", in a lot of use cases there's only a limited set of "writers" and updates happen infrequently. Perhaps I should rephrase my previous comment, then, as "immutable systems are good for building mutable systems on top". Does that help to provide a better counter argument?
  - vog 9y ago
    Here's my complete line of reasoning: You can build mutable systems on top of immutable (append-only) systems. But is that a good idea? Yes, it is, for systems which don't have huge amounts of (non-static) data, and/or system which need an audit-trail anyway. And these are more systems than one may initially think.
    - userpass 9y ago
      I don't really get it. Sure it's fine if one p2P app uses 3GB (1GB for the append only log, 2GB for a database with indices that can actually be queried) of data. What if you have several apps? Let's say 10. Then you need 30GB and because people only have 32GB to 64GB of storage on their phones the discussion ends right here.
      - macawfish 9y ago
        I didn't downvote you. But your data sizes are arbitrary. Why would something like a chat or email app need to hang onto that much history? Imagine a distributed "email" app that uses networks of mutually trusted peers to deliver encrypted messages ("emails") asynchronously. My device doesn't need to hang onto your emails indefinitely. It only needs to hang onto them until they've been received. This could be done via explicitly sending receipts, or probably in most cases by giving stuff simple expiration dates. The sender would have the most incentive to hang onto the original message until its been delivered. How this scales in terms of MB and GB is hugely dependent on how your application is configured, how frequently new data is emerging, the limits set by peers for how much they're willing to share, etc. But text is pretty cheap. I can't imagine storing 3 GB of yours or someone else's text emails on your phone, short term or long term. The raspberry pi plugged into the wall at your house can has much more storage anyway ;)
    - carolc 9y ago
      "Here's my complete line of reasoning: You can build mutable systems on top of immutable (append-only) systems. But is that a good idea? Yes, it is, for systems which don't have huge amounts of (non-static) data, and/or system which need an audit-trail anyway. And these are more systems than one may initially think." I disagree that immutability is a negatively defining factor here re. data size or capabilities of the database. If you look how many Big Data systems process data, you'll find that at the core of many, is an append-only log. For example: Kafka is a log (https://engineering.linkedin.com/distributed-systems/log-wha... https://engineering.linkedin.com/distributed-systems/log-wha...), and looking at Apache Samza's architecture, we can see how a log is at the core of it (https://www.confluent.io/blog/turning-the-database-inside-ou... https://www.confluent.io/blog/turning-the-database-inside-ou...). In less Big Data orientated databases, there's always a log of operations (sometimes also called a transaction log or replication log) to keep the track of changes.
  - deleted 9y ago
    [deleted]
- imtringued 9y ago
  I don't really see why a CRM needs to be decentralised. You need to host it yourself to avoid a cloud vendor going out of business but other than that what problem do you solve by decentralising it?
  - OtterCoder 9y ago
    Delivery. A cloud provides worldwide availability at the cost of trust. A distributed site can survive any one entity failing, which includes you, and it can serve from anywhere your users want it to.

by carolc 9y ago

vog 9y ago

I agree with you, but your argument is deeply flawed. There's quite a far stretch between "Y can be built on top of X" and "X is good for Y". To provide an argument that might fill this gap: Most systems don't actually have a huge amount of data. Look at the data size and data growth of CRMs, special-purpose wikis, and so on: These are mostly smaller than 500 MB (excluding static content like images), and grow by less than 1MB even on a busy day. And that's the uncompressed size. Also, most systems, despite being mutable, actually want (or need) an audit trail. So these are really append-only systems which merely have a "mutable look and feel" to the user.

carolc 9y ago
Agreed that many CRMs etc. don't have a lot of data. And that's actually good, it makes the database size very manageable in the context of trustless, distributed networks. I'm not following the logic of the argument here though, jumping from "X is good for Y" to "...don't actually have huge amount of data", perhaps you can elaborate? With a merkelized append-only log (immutable DAG), there's always an audit trail. I agree with your point about "mutable look and feel", in a lot of use cases there's only a limited set of "writers" and updates happen infrequently. Perhaps I should rephrase my previous comment, then, as "immutable systems are good for building mutable systems on top". Does that help to provide a better counter argument?
- vog 9y ago
  Here's my complete line of reasoning: You can build mutable systems on top of immutable (append-only) systems. But is that a good idea? Yes, it is, for systems which don't have huge amounts of (non-static) data, and/or system which need an audit-trail anyway. And these are more systems than one may initially think.
  - userpass 9y ago
    I don't really get it. Sure it's fine if one p2P app uses 3GB (1GB for the append only log, 2GB for a database with indices that can actually be queried) of data. What if you have several apps? Let's say 10. Then you need 30GB and because people only have 32GB to 64GB of storage on their phones the discussion ends right here.
    - macawfish 9y ago
      I didn't downvote you. But your data sizes are arbitrary. Why would something like a chat or email app need to hang onto that much history? Imagine a distributed "email" app that uses networks of mutually trusted peers to deliver encrypted messages ("emails") asynchronously. My device doesn't need to hang onto your emails indefinitely. It only needs to hang onto them until they've been received. This could be done via explicitly sending receipts, or probably in most cases by giving stuff simple expiration dates. The sender would have the most incentive to hang onto the original message until its been delivered. How this scales in terms of MB and GB is hugely dependent on how your application is configured, how frequently new data is emerging, the limits set by peers for how much they're willing to share, etc. But text is pretty cheap. I can't imagine storing 3 GB of yours or someone else's text emails on your phone, short term or long term. The raspberry pi plugged into the wall at your house can has much more storage anyway ;)
  - carolc 9y ago
    "Here's my complete line of reasoning: You can build mutable systems on top of immutable (append-only) systems. But is that a good idea? Yes, it is, for systems which don't have huge amounts of (non-static) data, and/or system which need an audit-trail anyway. And these are more systems than one may initially think." I disagree that immutability is a negatively defining factor here re. data size or capabilities of the database. If you look how many Big Data systems process data, you'll find that at the core of many, is an append-only log. For example: Kafka is a log (https://engineering.linkedin.com/distributed-systems/log-wha... https://engineering.linkedin.com/distributed-systems/log-wha...), and looking at Apache Samza's architecture, we can see how a log is at the core of it (https://www.confluent.io/blog/turning-the-database-inside-ou... https://www.confluent.io/blog/turning-the-database-inside-ou...). In less Big Data orientated databases, there's always a log of operations (sometimes also called a transaction log or replication log) to keep the track of changes.
- deleted 9y ago
  [deleted]
imtringued 9y ago
I don't really see why a CRM needs to be decentralised. You need to host it yourself to avoid a cloud vendor going out of business but other than that what problem do you solve by decentralising it?
- OtterCoder 9y ago
  Delivery. A cloud provides worldwide availability at the cost of trust. A distributed site can survive any one entity failing, which includes you, and it can serve from anywhere your users want it to.