nathanmarz

9 ms·

91.
by nathanmarz 3y ago
Mastodon isn't one use case. It's a collection of radically different use cases. The way personalized follow suggestions works is completely different than how home timelines work, which is completely different than how tracking f
92.
by nathanmarz 3y ago
It can be programmed with any JVM language, such as Clojure, Scala, or Kotlin. The JVM is a well-understood and widely used platform, which is why Rama is built on top of it. Being a library in a general purpose language as opposed to a cus
93.
by nathanmarz 3y ago
Rama is a general purpose system. You can use it to build financial applications, note-taking apps, analytics, search systems, or pretty much anything else. The acceleration that Rama provides to building Mastodon at Twitter-scale applies s
94.
Open-source Twitter-scale Mastodon implementation in 10k LOC(github.com)
33 points by nathanmarz 3y ago|19 comments
95.
by nathanmarz 3y ago
This isn't my first time releasing a big project, so I'm used to the HN knee-jerk reactions :) Thanks for the kind words.
96.
by nathanmarz 3y ago
For now it's going to be on-prem, so each user will just have their own cluster. Things like E2E encryption are pretty easy to implement on top of Rama's existing primitives (there was a good question about this on the rama-user g
97.
by nathanmarz 3y ago
Thanks for the feedback. The post is meant to give a taste of Rama by showing what it can do for building a full application end-to-end. Next week, we'll be releasing a set of guides, tutorials, and documentation which introduces the c
98.
by nathanmarz 3y ago
A lot of detail on subindexing and everything else in Rama is coming next week when we release the docs.
99.
by nathanmarz 3y ago
Internally there actually is a new programming language, implemented using Clojure macros (so it's also Clojure). The Java dataflow API is exposing a subset of that language. We did it this way rather than expose this new language dire
100.
by nathanmarz 3y ago
Your comment is addressed in another thread by other commenters: https://news.ycombinator.com/item?id=37137653
101.
by nathanmarz 3y ago
We're storing those in-memory within the Rama modules materializing the home timelines. And the query topologies that refresh home timelines for lost partitions is colocated with that. This is dramatically simpler than operating a sepa
102.
by nathanmarz 3y ago
We are very open in the post that the core concepts are not new: Individually, none of these concepts are new. I’m sure you’ve seen them all before. You may be tempted to dismiss Rama’s programming model as just a combination of event s
103.
by nathanmarz 3y ago
That's part of designing Rama applications. Acking is only coordinated with colocated stream topologies – stream topologies consuming that depot from another module don't add any latency. Internally Rama does a lot of dynamic auto
104.
by nathanmarz 3y ago
Rama is a much broader platform than a database, so the consistency semantics you get depend on how you use it. When using Rama, you're not mutating indexes directly like you do with a database, but adding source data that then gets ma
105.
by nathanmarz 3y ago
Answered this in another comment: https://news.ycombinator.com/item?id=37138526
106.
by nathanmarz 3y ago
Yes, it's a better model for developing backends in general. Our comparison against Mastodon's official implementation demonstrates this, being at least 44% less code. It's the ability to avoid the impedance mismatches which
107.
by nathanmarz 3y ago
The load generator generates boosts/favorites for a subset of posts that are randomly picked to be "popular". However, since the rate of posts is so high even individual posts picked to be "popular" are only getting
108.
by nathanmarz 3y ago
We did notice on a less powerful machines the browser getting overwhelmed with the rate of new content (even though we're only streaming 10/s instead of the full 3.5k/s actually happening on the backend). I don't know if
109.
by nathanmarz 3y ago
We're releasing 100k words of high-quality documentation next week.
110.
by nathanmarz 3y ago
Rama codifies and integrates the concepts I described in my book, with the high level model being: indexes = function(data) and query = function(indexes). These correspond to "depots" (data) , "ETLs" (functions), "P
111.
by nathanmarz 3y ago
Rama is scalable. So as your usage grows, you add resources to keep up. Scaling a Rama module is a trivial one-line command at the terminal. Rama's built-in telemetry provides the information you need to know when it's time to sca
112.
by nathanmarz 3y ago
When you step back and consider the incredible amount of manpower and resources that have been put into these applications, it's amazing how buggy these applications are. To put it simply, they're buggy because the underlying infr
113.
by nathanmarz 3y ago
Yes. Depot appends by default don't return success until colocated streaming topologies have completed processing the data. So this is one way to coordinate the frontend with changes on the backend. Within an ETL, when the computations
114.
by nathanmarz 3y ago
What do you mean? Every post I do shows up instantly. Reloading the page from scratch can be slow due to Soapbox doing a lot of stuff asynchronously from scratch (Soapbox is the open-source Mastodon interface that we're using to serve
115.
by nathanmarz 3y ago
We're comparing just to the original consumer product, which is about the same as Mastodon is today. That's why we said "original consumer product" and not "Twitter's current consumer product". Mastodon ac
116.
by nathanmarz 3y ago
Well, we didn't start from anything as we implemented this completely from scratch. I believe Mastodon is much more widely used than those so it seemed like a better target for this.
117.
by nathanmarz 3y ago
Rama is extremely fast, as you can see for yourself by playing with our Mastodon instance.
118.
by nathanmarz 3y ago
The impedance mismatches you're used to from using databases are gone because: - You can finely tune your indexes to be exactly the optimal shape for your application (data structure). You can see this in our Mastodon implementation wi
119.
by nathanmarz 3y ago
We're not releasing one as we don't have the bandwidth right now to maintain and document another API. That said, making a Clojure wrapper around the Java API should be pretty easy.
120.
by nathanmarz 3y ago
It's one of the most widely used/known programming languages in the world. It's a Java API so any JVM language can be used (Clojure, Scala, etc.).

More ›