4 ms·

Open-source Twitter-scale Mastodon implementation in 10k LOC

33 points by nathanmarz 3y ago

sn0wf1re 3y ago
Discussion on the blog post for this implementation: https://news.ycombinator.com/item?id=37137110 https://news.ycombinator.com/item?id=37137110
mdaniel 3y ago
As best I can tell the Rama-awesomesauce is down in the backend modules, e.g. https://github.com/redplanetlabs/twitter-scale-mastodon/blob/f86abdb7c86e80bd57ad7f2be2b63c0431f77e4b/backend/src/main/java/com/rpl/mastodon/modules/Core.java#L350 https://github.com/redplanetlabs/twitter-scale-mastodon/blob... and the rest of the commentary I'll keep to my damn self
- samspenc 3y ago
  I'm trying to understand this as well based on the code you linked to and other comments - so the secret sauce of this implementation is the Rama library / framework / database? And that's closed source? Also looks like Rama itself is in beta right now? When will it be stable? Will it remain closed source at that point?
  - mdaniel 3y ago
    No, sorry for any confusion: since this codebase is designed to be a showcase of how awesome programming in Rama is but it has (as the author claimed) 10k lines, then I would imagine an interested reader would want to know how does one use Rama's API to do Mastodon stuff, and skip past the Spring plumbing bits that are not Rama-ized (AFAIK) Thus, I found some of the .whatever("*awesome") looking code down in the guts of the backend module but since there is no other obvious tour of how the Rama works in this codebase, I thought I would draw the reader's attention to the interesting bits As for your other product related questions, that'd be much better as a top-level comment so nathanmarz can reply to them
    - nathanmarz 3y ago
      The Spring stuff and Rama stuff is split between the api/ and backend/ folders. The backend/ folder is the more interesting part to look at.
  - nathanmarz 3y ago
    The best resource for learning about Rama is its documentation: https://redplanetlabs.com/docs/~/index.html https://redplanetlabs.com/docs/~/index.html Our blog post about our Mastodon implementation is a deep dive into how it works, and how the implementation is utilizing Rama to do some really powerful things: https://blog.redplanetlabs.com/2023/08/15/how-we-reduced-the-cost-of-building-twitter-at-twitter-scale-by-100x/ https://blog.redplanetlabs.com/2023/08/15/how-we-reduced-the...
remram 3y ago
They can just put the rest of the code in this proprietary "Rama" lib that apparently doesn't count, and make it one line of code: Rama.StartMastodon() That way you can call it "public domain" too, I doubt a single line is copyrightable. With all the real code hidden. Awesome.
- nathanmarz 3y ago
  Rama is a general purpose system. You can use it to build financial applications, note-taking apps, analytics, search systems, or pretty much anything else. The acceleration that Rama provides to building Mastodon at Twitter-scale applies similarly to other complex applications at scale.
  - remram 3y ago
    Fediverse is a general-purpose platform. My point is, they're (you're?) drawing an arbitrary line and counting a precise number. So far Rama has only been used for this one purpose, this is the kickoff marketing example, calling the system "general purpose" is a giant stretch.
    - nathanmarz 3y ago
      Mastodon isn't one use case. It's a collection of radically different use cases. The way personalized follow suggestions works is completely different than how home timelines work, which is completely different than how tracking followers works, which is completely different than how search works, and so on. I can list dozens of features that work completely differently from one another. Yet they're all expressed elegantly, scalably, and with high-performance on the exact same set of abstractions. If you peruse our blog post and documentation, you can see for yourself how general purpose Rama is. You can even try it for yourself with the build we released. https://blog.redplanetlabs.com/2023/08/15/how-we-reduced-the-cost-of-building-twitter-at-twitter-scale-by-100x/ https://blog.redplanetlabs.com/2023/08/15/how-we-reduced-the... https://redplanetlabs.com/docs/~/index.html https://redplanetlabs.com/docs/~/index.html https://redplanetlabs.com/docs/~/downloads-maven-local-dev.html https://redplanetlabs.com/docs/~/downloads-maven-local-dev.h...
npace12 3y ago
From what I can tell, this is a production-ready differential dataflow solution with basically, a DSL. Is Java supposed to be the selling point as in, "oh it's just Java so you can run it on your Java things?" Why not just make it some GRPC-thing with SDKs or even its own well-documented DSL?
- nathanmarz 3y ago
  It can be programmed with any JVM language, such as Clojure, Scala, or Kotlin. The JVM is a well-understood and widely used platform, which is why Rama is built on top of it. Being a library in a general purpose language as opposed to a custom language means you can do things like: generate dataflow code dynamically, easily use any JVM library, easily integrate with other external systems (e.g. databases) using their Java client, and easily mix regular Java code with Rama dataflow code. All of these are extremely important and should never be sacrificed. Losing any one of these massively increases complexity. Our Mastodon implementation relies heavily on being able to do these things.
  - npace12 3y ago
    Yes, I get that, and it makes sense. But given that this is a potentially very powerful, (but closed) platform, I see the JVM part as an implementation detail. If I'm going to be getting my whole team writing pstates and things, now I also have to tell them they need to write their code for the JVM. Temporal.io made a platform and wrote it in Go or whatever they use, but they released with a few SDKs for different languages. Would this just not be possible with Rama? like, does a serializing interface like grpc make Rama somehow not worth it?
    - nathanmarz 3y ago
      One way to think of Rama is as a "programmable datastore", where your application logic goes into and is colocated with your data storage. So your logic has to be written to run on the same platform on which Rama is built, which is the JVM.
PeterStuer 3y ago
Is this Rama library for which this is a demo going to be Open Source? What license?
DoesntMatter22 3y ago
This keeps launching but there is basically not much to see. I guess: why should we care?

33 points by nathanmarz 3y ago

Discussion on the blog post for this implementation: https://news.ycombinator.com/item?id=37137110 https://news.ycombinator.com/item?id=37137110

mdaniel 3y ago

As best I can tell the Rama-awesomesauce is down in the backend modules, e.g. https://github.com/redplanetlabs/twitter-scale-mastodon/blob/f86abdb7c86e80bd57ad7f2be2b63c0431f77e4b/backend/src/main/java/com/rpl/mastodon/modules/Core.java#L350 https://github.com/redplanetlabs/twitter-scale-mastodon/blob... and the rest of the commentary I'll keep to my damn self

samspenc 3y ago
I'm trying to understand this as well based on the code you linked to and other comments - so the secret sauce of this implementation is the Rama library / framework / database? And that's closed source? Also looks like Rama itself is in beta right now? When will it be stable? Will it remain closed source at that point?
- mdaniel 3y ago
  No, sorry for any confusion: since this codebase is designed to be a showcase of how awesome programming in Rama is but it has (as the author claimed) 10k lines, then I would imagine an interested reader would want to know how does one use Rama's API to do Mastodon stuff, and skip past the Spring plumbing bits that are not Rama-ized (AFAIK) Thus, I found some of the .whatever("*awesome") looking code down in the guts of the backend module but since there is no other obvious tour of how the Rama works in this codebase, I thought I would draw the reader's attention to the interesting bits As for your other product related questions, that'd be much better as a top-level comment so nathanmarz can reply to them
  - nathanmarz 3y ago
    The Spring stuff and Rama stuff is split between the api/ and backend/ folders. The backend/ folder is the more interesting part to look at.
- nathanmarz 3y ago
  The best resource for learning about Rama is its documentation: https://redplanetlabs.com/docs/~/index.html https://redplanetlabs.com/docs/~/index.html Our blog post about our Mastodon implementation is a deep dive into how it works, and how the implementation is utilizing Rama to do some really powerful things: https://blog.redplanetlabs.com/2023/08/15/how-we-reduced-the-cost-of-building-twitter-at-twitter-scale-by-100x/ https://blog.redplanetlabs.com/2023/08/15/how-we-reduced-the...

remram 3y ago

They can just put the rest of the code in this proprietary "Rama" lib that apparently doesn't count, and make it one line of code: Rama.StartMastodon() That way you can call it "public domain" too, I doubt a single line is copyrightable. With all the real code hidden. Awesome.

nathanmarz 3y ago
Rama is a general purpose system. You can use it to build financial applications, note-taking apps, analytics, search systems, or pretty much anything else. The acceleration that Rama provides to building Mastodon at Twitter-scale applies similarly to other complex applications at scale.
- remram 3y ago
  Fediverse is a general-purpose platform. My point is, they're (you're?) drawing an arbitrary line and counting a precise number. So far Rama has only been used for this one purpose, this is the kickoff marketing example, calling the system "general purpose" is a giant stretch.
  - nathanmarz 3y ago
    Mastodon isn't one use case. It's a collection of radically different use cases. The way personalized follow suggestions works is completely different than how home timelines work, which is completely different than how tracking followers works, which is completely different than how search works, and so on. I can list dozens of features that work completely differently from one another. Yet they're all expressed elegantly, scalably, and with high-performance on the exact same set of abstractions. If you peruse our blog post and documentation, you can see for yourself how general purpose Rama is. You can even try it for yourself with the build we released. https://blog.redplanetlabs.com/2023/08/15/how-we-reduced-the-cost-of-building-twitter-at-twitter-scale-by-100x/ https://blog.redplanetlabs.com/2023/08/15/how-we-reduced-the... https://redplanetlabs.com/docs/~/index.html https://redplanetlabs.com/docs/~/index.html https://redplanetlabs.com/docs/~/downloads-maven-local-dev.html https://redplanetlabs.com/docs/~/downloads-maven-local-dev.h...

npace12 3y ago

From what I can tell, this is a production-ready differential dataflow solution with basically, a DSL. Is Java supposed to be the selling point as in, "oh it's just Java so you can run it on your Java things?" Why not just make it some GRPC-thing with SDKs or even its own well-documented DSL?

nathanmarz 3y ago
It can be programmed with any JVM language, such as Clojure, Scala, or Kotlin. The JVM is a well-understood and widely used platform, which is why Rama is built on top of it. Being a library in a general purpose language as opposed to a custom language means you can do things like: generate dataflow code dynamically, easily use any JVM library, easily integrate with other external systems (e.g. databases) using their Java client, and easily mix regular Java code with Rama dataflow code. All of these are extremely important and should never be sacrificed. Losing any one of these massively increases complexity. Our Mastodon implementation relies heavily on being able to do these things.
- npace12 3y ago
  Yes, I get that, and it makes sense. But given that this is a potentially very powerful, (but closed) platform, I see the JVM part as an implementation detail. If I'm going to be getting my whole team writing pstates and things, now I also have to tell them they need to write their code for the JVM. Temporal.io made a platform and wrote it in Go or whatever they use, but they released with a few SDKs for different languages. Would this just not be possible with Rama? like, does a serializing interface like grpc make Rama somehow not worth it?
  - nathanmarz 3y ago
    One way to think of Rama is as a "programmable datastore", where your application logic goes into and is colocated with your data storage. So your logic has to be written to run on the same platform on which Rama is built, which is the JVM.

PeterStuer 3y ago

Is this Rama library for which this is a demo going to be Open Source? What license?

DoesntMatter22 3y ago

This keeps launching but there is basically not much to see. I guess: why should we care?