4 ms·

I don't agree that it makes sense to "design a protocol" before building the application. You can try to imagine what you want the protocol to look like at the

I don't agree that it makes sense to "design a protocol" before building the application. You can try to imagine what you want the protocol to look like at the end of the day, but you are likely to get it wrong if you don't have an application that's actually using it. It's important to keep your eyes open and try to avoid going down blind alleys, but there's not much value in building infrastructure before it's needed. The questions raised in that post are good, and we intend to address them as they become relevant to what we have build. Discovery, namespacing and routing are areas where we are starting to feel tension on in the product, and will probably be tackled quite soon. The point about affecting interface design is good, but we look at it the other way. We want our interface design to drive the protocol design, so we want to make protocol choices as late as is reasonable. Finally, it's fantastic that people are reading and forking the code. The more people out there trying things, the better.

imaginator 16y ago
I agree - it's a fantastic thing to be forking the code. The side effect is that with each fork whatever protocol Diaspora have chosen becomes more entrenched and difficult to change. What I was trying to get at in the article was that app designers need to think about the protocol at some point since questions such as "are we relying on long polling or will the user refresh pages" start to play a question on the UI side too. Also, if a product is marketed as being distributed, some thought as to how this will work and scale become important. It's really good that Diaspora are thinking about what the user sees and programming from the user's POV. I spend a lot of time in the XMPP community and there is not enough of this focus. But a good programmer has to balance design with functionality and focus on many areas and some more thought on the protocol side would not go amiss.
- mikesofaer 16y ago
  We're very aware that we will be breaking interoperability with forks when we change the protocol, and there is a little bit of a worry that people won't keep their forks up to date and the protocol will fragment. I am optimistic that that won't happen, largely because the Diaspora product is the only part of the system actually visible to the outside world at the moment, and that gives us a pretty good way to draw a bright line between products that interoperate and products that don't. I don't think that the UI polling/refresh question affects the inter-server protocol. Your seed is not your browser, and it's seed-seed communication that's the part of the protocol that matters in the long run. Scaling is the pressure we're feeling most right now, and we're talking about how to deal with the issue of a distributed cluster with a coherent namespace. There are some interesting things about the Dynamo pattern that might be really useful to make this work. But without a running system with load and users and remote pods to connect to, it's very hard to figure out what you want. I think that were at the point where the protocol needs something of a refactor, and I will probably make a post about it on the Diaspora-dev google group soon. Maybe we'll get some good feedback and design help!
michaelchisari 16y ago
As someone who has been working on this for a very long, my advice is to write the software to be protocol agnostic. Be able to switch out protocols easily, and support multiple protocols concurrently, because we still have no idea which protocol will win out. It may be something that hasn't even been built yet.
- kenjackson 16y ago
  I agree, but this is harder than it sounds. Why? If this is your first time building such a system you don't even know what is protocol and what's not. There's an old saying that you need to build three apps on your framework. That first app shows that your framework works for something. The second app shows that its more general than just the first app. The third app shows that you weren't lucky. Curious, how was Facebook designed? Are there any post-mortem docs on it? I know that MySpace seemed really ad hoc. I wouldn't be surprised if Facebook was more ad hoc than we imagine it to have been
  - dasil003 16y ago
    How do you measure "more ad hoc"? The only software that's not ad hoc are huge enterprise projects that fail most of the time. If I had to guess I'd say that both MySpace and Facebook were completely ad hoc, but the difference was probably that Facebook kept their technical debt down so that they never accumulated an area of code that was too hairy to refactor or replace. MySpace on the other hand probably got complacent in their success and didn't worry too much as their architecture calcified into an unmaintainable mess. By the time they saw Facebook nipping at their heels their code base was past the point of no return, and the company culture made it impossible to hire the level of technical talent they would have needed to right the ship. Of course I'm completely talking out my ass here, but that's the impression given by the respective product histories.
  - jacobolus 16y ago
    Who doesn’t imagine Facebook’s early design to be “ad hoc”? It was programmed as a spaghetti mess of PHP on Zuckerberg’s tiny laptop, and only worked because the user base was inherently limited to a few thousand people. Then the spaghetti mess was grown and added to over the next year or two, and the system scaled because each college’s network could be contained in a single server that only had to support a few thousand users (these servers used to use domain names like harvard.thefacebook.com or mit.thefacebook.com, IIRC). At some point they completely rewrote it with an architecture that would scale and not waste hardware, but it definitely didn’t start out that way.

imaginator 16y ago

I agree - it's a fantastic thing to be forking the code. The side effect is that with each fork whatever protocol Diaspora have chosen becomes more entrenched and difficult to change. What I was trying to get at in the article was that app designers need to think about the protocol at some point since questions such as "are we relying on long polling or will the user refresh pages" start to play a question on the UI side too. Also, if a product is marketed as being distributed, some thought as to how this will work and scale become important. It's really good that Diaspora are thinking about what the user sees and programming from the user's POV. I spend a lot of time in the XMPP community and there is not enough of this focus. But a good programmer has to balance design with functionality and focus on many areas and some more thought on the protocol side would not go amiss.

mikesofaer 16y ago
We're very aware that we will be breaking interoperability with forks when we change the protocol, and there is a little bit of a worry that people won't keep their forks up to date and the protocol will fragment. I am optimistic that that won't happen, largely because the Diaspora product is the only part of the system actually visible to the outside world at the moment, and that gives us a pretty good way to draw a bright line between products that interoperate and products that don't. I don't think that the UI polling/refresh question affects the inter-server protocol. Your seed is not your browser, and it's seed-seed communication that's the part of the protocol that matters in the long run. Scaling is the pressure we're feeling most right now, and we're talking about how to deal with the issue of a distributed cluster with a coherent namespace. There are some interesting things about the Dynamo pattern that might be really useful to make this work. But without a running system with load and users and remote pods to connect to, it's very hard to figure out what you want. I think that were at the point where the protocol needs something of a refactor, and I will probably make a post about it on the Diaspora-dev google group soon. Maybe we'll get some good feedback and design help!

michaelchisari 16y ago

As someone who has been working on this for a very long, my advice is to write the software to be protocol agnostic. Be able to switch out protocols easily, and support multiple protocols concurrently, because we still have no idea which protocol will win out. It may be something that hasn't even been built yet.

kenjackson 16y ago
I agree, but this is harder than it sounds. Why? If this is your first time building such a system you don't even know what is protocol and what's not. There's an old saying that you need to build three apps on your framework. That first app shows that your framework works for something. The second app shows that its more general than just the first app. The third app shows that you weren't lucky. Curious, how was Facebook designed? Are there any post-mortem docs on it? I know that MySpace seemed really ad hoc. I wouldn't be surprised if Facebook was more ad hoc than we imagine it to have been
- dasil003 16y ago
  How do you measure "more ad hoc"? The only software that's not ad hoc are huge enterprise projects that fail most of the time. If I had to guess I'd say that both MySpace and Facebook were completely ad hoc, but the difference was probably that Facebook kept their technical debt down so that they never accumulated an area of code that was too hairy to refactor or replace. MySpace on the other hand probably got complacent in their success and didn't worry too much as their architecture calcified into an unmaintainable mess. By the time they saw Facebook nipping at their heels their code base was past the point of no return, and the company culture made it impossible to hire the level of technical talent they would have needed to right the ship. Of course I'm completely talking out my ass here, but that's the impression given by the respective product histories.
- jacobolus 16y ago
  Who doesn’t imagine Facebook’s early design to be “ad hoc”? It was programmed as a spaghetti mess of PHP on Zuckerberg’s tiny laptop, and only worked because the user base was inherently limited to a few thousand people. Then the spaghetti mess was grown and added to over the next year or two, and the system scaled because each college’s network could be contained in a single server that only had to support a few thousand users (these servers used to use domain names like harvard.thefacebook.com or mit.thefacebook.com, IIRC). At some point they completely rewrote it with an architecture that would scale and not waste hardware, but it definitely didn’t start out that way.