24 ms·

Databases and why their complexity is now unnecessary

• gwbas1c 3y ago
  I spent a lot of time reading this yesterday, and started looking at Rama's docs. I think the a database that encapsulates denormalization so that derived views (caches, aggregations) are automatic is a killer feature. But far too often awesome products and ideas fail for trivial reasons. In this case, I just can't understand how Rama fits into an application. For example: Every example is Java. Is Rama only for Java applications? Or, is there a way to expose my database as a REST API? (That doesn't require me to jump through a million hoops and become an expert in the Java ecosystem?) Can I run Rama in Azure / AWS / Google cloud / Oracle cloud? Are there pre-built docker images I can use? Or is this a library that I have to suck into a Java application and use some kind of existing runtime? (The docs mention Zookeeper, but I have very little experience with it.) IE: It's not clear where the boundary between my application (Java or not) and Rama is. Are the examples analogous to sprocs (run in the DB) or business logic (run in the application)? The documentation is also very hard. It appears the author has every concept in their head, because they know Rama inside and out, yet can't emphasize with the reader and provide simpler bits of information that convey useful concepts. There's both "too much" (mixing of explaining pstates and the depot) and "too little" (where do I host it, what is the boundary between Rama and my application?) Another thing I didn't see mentioned is tooling: every SQL database has at least one general SQL client. (MSSQL studio, Azure data studio,) that allows interacting with the database. (Viewing the schema, ad-hoc queries, ECT.) Does Rama have this, or is every query a custom application? Anyway, seems like a cool idea, but it probably needs some well-chosen customers who ask tough questions so the docs become mature.
• HackerThemAll 3y ago
  A lame marketing mumbling to persuade people to buy a specific product.
• es7 3y ago
  Did anyone else think this was satire for the first few minutes of reading it? Calling databases global state and arguing why they shouldn’t be used was ridiculous enough that I wanted to call Poe’s Law here. But it does look like the author was sincere. Event Sourcing is one of those cool things that seem great in theory but in my experience I’ve never seen it actually help teams produce good software quickly or reliably.
  • Joker_vD 3y ago
    Well, the databases are indeed mutable global state but even if you get rid of them, we would still continue to live in a single (i.e. global), mutable world of physical reality. So you have to bite that bullet somewhere, and DBMSs seem to be suited rather well for that.
• csours 3y ago
  How about this: ACID RDBMS in many cases are sugar. That is, they provide very NICE features, but those features can be implemented in other ways. In the cloud world, the sugar may not be worth the costs. I think the weak case is much stronger than the strong case - that is, you can refactor to remove RDBMS dependencies; but that moves the complexity elsewhere.
• danscan 3y ago
  I did a year long project to build a flexible engine for materialized views onto 1-10TB live event datasets, and our architecture was roughly converging toward this idea of "ship the code to where the indexes are" before we moved onto a different project I'm very compelled by Rama, but unfortunately won't adopt it due to JVM for totally irrational reasons (just don't like Java/JVM). Would love to see this architecture ported!
• pulse7 3y ago
  Could be generalized to "Everything wrong with <placeholder> and why their complexity is now unnecessary"...
• sigmonsays 3y ago
  this seems like a classic bait and switch post selling a product called rama The approach here seems drastically more complicated; for simple apps, you go for a well known master->slave setup. For complicated apps you scale (shard, cluster, etc). Pick your database appropriately
• estebarb 3y ago
  I remembers me CouchDB + incremental map reduce. Except that in CouchDB you can mutate state. Idk, but keeping all the history doesn't take a lot of space?
  • nathanmarz 3y ago
    You can configure depots to trim themselves. Depending on the application this may or may not be necessary.
• twotwotwo 3y ago
  A very simple thing about this (and many systems!) is that if your whole thing is "log writes and do the real work later", you lose read-your-writes and with it the idea that your app has a big persistent memory to play in. This doesn't only matter if you're doing balance transfers or such; "user does a thing and sees the effects in a response" is a common wish. (Of course, if you're saving data for analytics or such and really don't care, that's fine too.) When people use eventually-consistent systems in domains where they have to layer on hacks to hide some of the inconsistency, it's often because that's the best path they had out of a scaling pickle, not because that's the easiest way to build an app more generally. I guess the other big thing is, if you're going to add asynchrony, it's not obvious this is where you want to add it. If you think of ETLs, event buses, and queues as tools, there are a lot more ways to deploy them--different units of work than just rows, different backends, different amounts of asynchrony for different things (including none), etc. Why lock yourself down when you might be able to assemble something better knowing the specifics of your situation? This company's thing is riding the attention they get by making goofy claims, so I'm a bit sorry to add to that. I do wonder what happens once they're talking to actual or potential customers, where you can't bluff indefinitely.
  • vidarh 3y ago
    Note that you can sometimes "fix" the issues you mention relatively easily with one or both of these. This method isn't always appropriate, but it often is sufficient: * Have anything potentially updated by the changes users make auto-update (e.g. via websockets or server push). As long as the latency for an update to make it through the system isn't too high, this is often fine. E.g. let's say you're posting a reply, then putting a spinner in the UI to be replaced once the reply has made it "back" to the client can be fine - as long as it's quick enough. * If it's not quick enough and/or instead of doing that, it's often sufficient to "fake it": Once it's been pushed to the server, just inject the same data back into your UI as if it came from the server. As long as what you submit to the server and what you get from the server is similar enough, it's often fine. If there are fields you can't populate, then just use the first method to still have the data auto-update once they make it through the whole chain. It does have the downside that if the user refreshes it might appear to disappear unless you e.g. put it in localstorage or similar, and then it does start getting hairy, so using it to paper over long delays it not a good idea. In those cases, there is the option of updating a server-side cache, but then you're starting to do a lot of work to avoid having consistent state and might want to consider if you should instead just bite the bullet and ensure you do have consistent state. Of course this does not mean its always appropriate, or sufficient, to use this style anyway, and I absolutely agree that there are plenty of instances where not being able to immediately read your writes is a problem, but sometimes these kinds of tweaks can be very helpful. I agree the claims made are a bit goofy, though, and I think I'd frankly mostly prefer to roll my own over dealing with their API, but the architectural style can be quite useful if its not overdone.
• w10-1 3y ago
  I was in favor of doubling the complexity by prefixing RDB with event logs, for retrospective QA/analysis and prospective client segregation. Databases now are a snapshot of the data modeling and usage at a particular point in application lifecycle. We manage to migrate data as it evolves, but you can't go back in time. Why go back? In our case, our interpretation of events (as we stuffed data into the DB) was hiding the data we actually needed to discover problems with our (bioinformatics and factory) workflow - the difference between expected and actual output that results from e.g., bad batches of reagent or a broken picker tip. We only stored e.g., the expected blend of reagents because that's all we needed for planning. That meant we had no way to recover the actions leading to that blend for purposes of retrospective quality analysis. So my proposal was to log all actions, derive models (of plate state) as usual for purpose of present applications, but still be able to run data analysis on the log to do QA when results were problematic. Ha ha! They said, but still :) Event prefixing might also help in the now/later design trade-off. Typically we design around requirements now, and make some accommodation for later if it's not too costly. Using an event log up front might work for future-proofing. It also permits "incompatible" schema to co-exist for different clients, as legacy applications read the legacy downstream DB, while new ones read the upcoming DB. For a bio service provider, old clients validate a given version, and they don't want the new model or software, while new clients want the new stuff you're building for them. You end up maintaining different DB models and infrastructure -- yuck! But with event sourcing, you can at least isolate the deltas, so e.g., HIPAA controls and auditing live in the event layer, and don't apply to the in-memory bits. TBH, a pitch like Rama's would play better in concert with existing DB's, to incrementally migrate the workflows that would benefit from it. Managers are often happy to let IT entrepreneurs experiment if it keeps them happy and away from messing with business-critical functions. YMMV...
  • ebiester 3y ago
    This was a very similar conversation that the team at my last job had. Our data were customer-facing so it was a slightly different problem, but the question of "our application cares about what the present state is, but our analytics cares about trends and point-in-time" is a more universal problem. We used Debezium and Change Data Capture + Outbox patterns in some cases, and solely outbox patterns in others. (See https://debezium.io/blog/2020/02/10/event-sourcing-vs-cdc/ https://debezium.io/blog/2020/02/10/event-sourcing-vs-cdc/ ) - You then can use the change events to do the analysis. That solves your "log all actions" problem. It's still a lot of work to put together the analysis (a tool like DBT helps) but it is a solved problem with enough labor at that point and doesn't disrupt the actual application. If I was starting from scratch on an application today that I thought might have analytics needs in the future, I would get this early on even if I didn't need it yet.
• thaanpaa 3y ago
  Disk drives are also large global mutable states. So is RAM at the operating system level. The article conflates the concept of data storage with best programming practices. Sure, you should not change the global state throughout your app because it becomes impossible to manage. The database is actually an answer to how to do it transactionally and centrally without messing up your data.
• chrisjc 3y ago
  I haven't read through all of the documentation and while I actually love Java, I'm surprised that there isn't some kind of declarative language (DDL but more than just the "data" in Data Description Language) even if that means relying on non-standard SQL objects/conventions. CREATE OR REPLACE MODULE MY_MOD ... CREATE OR REPLACE PSTATE MY_MOD.LOCATION_UPDATE (USER_ID NUMBER, LOC... CREATE PACKAGE MY_PACKAGE USING MY_MOD DEPLOY OR REDEPLOY MY_PACKAGE TASKS = 64 THREADS=16 ... Perhaps the same could be said for DML (Data Manipulation Language). I can imagine most DML operations (insert/update/delete/merge) could be used, while event-source occurs behind the scenes with the caller being none the wiser. Might there be an expressive way to define the serialization of parts of the DML (columns) down to the underlying PState. After all, if the materialized version of the PStates is based on expressions to the underlying data, then surely the reverse expression would be enough to understand how to mutate said underlying data. Or at least a way for Rama to derive the respective event-sourcing processes and handle it behind the scenes? Serialization/deserialization could also defined in SQL-like expressions as part of the schema/module. I say all of this while being acutely aware that there is undoubtedly as many people out there that dislike SQL as there are that dislike Java, or maybe more. I really like this: > Every backend that’s ever been built has been an instance of this model, though not formulated explicitly like this. Usually different tools are used for the different components of this model: data, function(data), indexes, and function(indexes).
• tpl 3y ago
  What sort of cost increase can I expect using something like this?
  • nathanmarz 3y ago
    Our license fee captures a small percentage of the cost savings you get from reducing development cost so much.
• keeganpoppen 3y ago
  this post is actually extremely based
• lambda_garden 3y ago
  indexes = function(data) query = function(indexes) How does this model a classic banking app where you need to guarantee that transfers between accounts are atomic?
  • nathanmarz 3y ago
    This is the bank transfer example we show in rama-demo-gallery. There's both Java and Clojure versions of that example. https://github.com/redplanetlabs/rama-demo-gallery https://github.com/redplanetlabs/rama-demo-gallery The atomic bank transfer is done as part of function(data). The data record contains fromAccountId, toAccountId, and amount. The function applies the transfer if it's valid (fromAccountId has at least that amount of funds), and no-ops otherwise.
    • lambda_garden 3y ago
      Thanks for sharing this, very interesting. It seems that you cannot, in one database call, make a transaction? You would need to push your transfer to the depot, then poll until it has been accepted. If you do not poll, then your transaction may be written but as a no-op. Does this increase latency for such applications?
      • nathanmarz 3y ago
        This example uses microbatching for the processing, so the latency will be ~200 millis. You don't need to poll and could set this up with a reactive PState query to know when the transaction is done. 200 millis is an acceptable latency for a task that needs to have strong cross-partition atomicity. Note that depot appends can be done with "acking" so as not to return success until all colocated stream ETLs have finished processing the record. Stream ETLs also take only a handful of millis to complete. This is how you would coordinate a front-end with many tasks like you typically do with databases today (e.g. registering an account, updating a profile, adding something to a shopping cart, making a friend request, etc.). This example uses microbatching because getting cross-partition transactionality with streaming is quite a bit harder, as streaming has either at-least once or at-most once semantics depending on how you configure it. Microbatching is always exactly-once semantics regardless of any failures that happen during processing.
• _xnmw 3y ago
  As a simple soloprenuer full stack dev who's never worked on an application serving more than a few thousand users, I can understand and relate to all of the problems written about here (some very compelling arguments), found myself nodding most of the way through, but I simply don't understand the proposed solution. Even the Hello World example from the docs flew over my head. And I've been programming apps in production for 15 years, and I like Java. This needs a simple pluggable adaptor for some popular frameworks (Django, Laravel, or Ruby on Rails) and then I can begin to have an idea how this actually be used in my project.
• jrockway 3y ago
  A few years ago I tried writing an application (something like Status Hero for internal use) with a non-traditional database. I used Badger, which is just a transactional k/v store, and stored each "row" as protobuf value and an ID number key. (message { int id = 1 }, query by type + ID, store anything with interface { GetId() int }.) I had additional messages for indexes and per-message-type IDs. (I like auto-incrementing IDs, sue me.) A typical transaction would read indexes, retrieve rows, manipulate them, save the rows, save the indexes, and commit. The purity in my mind before I wrote the application was impressive; this is all a relational database is doing under the hood (it has some bytes and some schema to tell it what the bytes mean, just like protos). But it was actually a ton of work that distracted me from writing the app. The code to handle all the machinery wasn't particularly large or anything, but the app also wasn't particularly large. I would basically say, it wasn't worth it. I should have just used Postgres. The one ray of sunshine was how easy it is to ship a copy of the database to S3; the app just backed itself up every hour, which is a better experience than I've had with Postgres (where the cloud provider deletes your backups when you delete the instance... so you have to do your own crazy thing instead). The article is on-point about managing the lifecycle of data. Database migrations are a stressful part of every deployment. The feature I want is to store a schema ID number in every row and teach the database how to run a v1 query against a v2 piece of data. Then you can migrate the data while the v1 app is running, then update the app to make v2 queries, then delete the v1 compatibility shim. If you store blobs in a K/V store, you can do this yourself. If you use a relational model, it's harder. You basically take the app down that knows v1 of your schema, upgrade all the rows to v2, and deploy the app that understands v2. The "upgrade all the rows to v2" step results in your app being unavailable. (The compromise I've seen, and used, which is horrible is "just let the app fail certain requests while the database is being migrated, and then have a giant mess to clean up when the migration fails". Tests lower the risk of a giant mess, and selective queries result in fewer requests that can't be handled by the being-migrated database, so in general people don't realize what a giant risk they're taking. But it can all go very wrong and you should be horrified when you do this.)
• keeganpoppen 3y ago
  this comment section has gotta be in the absolute upper echelons of non-RTFA i have seen on HN in a long time. even for HN. i acknowledge my own bias, though: i’ve been an admirer of nathan marz’s work from afar for years now, and basically trust him implicitly. but… wow. what fraction of the comments even engage with the substance of the article in any way? it’s not like they didn’t put their money where their mouth(s) is/are: they feel strongly enough about the problem that they built an entire goddamn “dont call it a database” (and business) around it. i’ve always been pretty sympathetic to code-/application- driven indexing, storage, etc.— it just seems intuitively more correct to me, if done appropriately. the biggest “feature” of databases, afaict, is that most people dont trust themselves to do this appropriately xD. and they distrust themselves in this regard so thoroughly that they deny the mere possibility of it being useful. some weird form of learned helplessness. you can keep cramming all of your variously-shaped blocks into tuple-shaped holes if you want, but it seems awfully closed-minded to deny the possibility of a better model on principle. what principle? the lindy effect?
• fipar 3y ago
  I can't wrap my head around the way this solves the global mutable state problem. First, here's what I do understand about databases and global state: compared to programming variables, I don't think databases are shared, mutable global state. Instead, I see them as private variables that can be changed through set/get methods (e.g., with SQL statements if on such a DB). So I agree shared, global state is dangerous (I'm not sure I'd call it harmful) and the reason I like databases is that I assume a DB, being specialized at managing data, will do a better job at protecting the integrity of that global state than I'd do myself from my program. With luck, there may even be a jepsen test of the DB I'm using that lets me know how good the DB is at doing this job. In this post there's an example of a question we'd ask Rama: “What is Alice’s current location?” How's that answered without global state? Because of the mention of event sourcing, I'd guess there's some component that knows where Alice was when the system was started, and keeps a record of events every time she changes her place. If Alice were the LOGO turtle, this component would keep a log with entries such as "Left 90 degrees" or "Forward 10 steps". If I want to know where Alice is now, I just need to access this log and replay everything and that'd be my answer. Now, I'm certain my understanding here must be wrong, at least on the implementation side, because this wouldn't be able to scale to the mastodon demo mentioned in the post, which makes me very curious: how does Rama solve the problem of letting me know where Alice is without giving me access to her state?
  • nathanmarz 3y ago
    Rama PStates are globally readable but not globally writable. They are only writable from the topology that declares them. All code writing to PStates is thereby always in the exact same program. Additionally, since PStates are not the source of truth – the depots (event logs) are – mistakes can be corrected via recompute from the source of truth. "Alice's current location" would be done in Rama like this: * Have a depot that receives new locations for people. The appended records would have three keys: userId, location, and timestamp. The depot is partitioned by userId. * Have an ETL with a PState called $$currentLocation that's a map from userId to location. * The ETL consumes the depot and update the PState as new data comes in.
    • 3y ago
• strangattractor 3y ago
  Seems similar datomic. https://www.datomic.com/benefits.html https://www.datomic.com/benefits.html
  • dustingetz 3y ago
    Rama is a meta-database that contains the generalized set of primitives that can be arranged to represent Datomic in principle. Which is just one of many possible arrangements.
• bccdee 3y ago
  Reminds me a lot of "Turning the Database Inside-Out"[1], but I think Red Planet Labs is overstating their point a little. TtDIO is a lot more careful with its argument, and it doesn't claim to have some sort of silver bullet to sell me. [1]: https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/ https://www.confluent.io/blog/turning-the-database-inside-ou...
• saberience 3y ago
  Such a poorly written article doesn't encourage me to use a brand new and untrusted database; if you can't write a clear article, why would I trust your database code? This is a thinly veiled ad for Rama but the explanation for why it's so much "better" isn't clear and doesn't make much sense. I strongly urge the author to work with some who is a clear and concise technical writer to help with articles such as these.
• jcrawfordor 3y ago
  I feel like I went into this from a position of genuine interest, I'm always on the lookout for significant developments in backend architecture. But when I hit the sentence "This can completely correct any sort of human error," I actually laughed out loud. Either the author is overconfident or they have had surprisingly little exposure to humans. More concretely, it seems to completely disregard the possibility of invalid/improper/inconsistent events being introduced by the writer... the way that things go wrong. And I don't see any justification for disregarding this possibility, it's just sort of waved away. That means waving way most of the actual complexity I see in this design, of having to construct your PState data models from your actual, problematic event history. Anyone that's worked with ETLs over a large volume of data will have spent many hours on this fight. I think the concept is interesting, but the revolutionary zeal of this introduction seems unjustified. It's so confident in the superiority of Rama that I have a hard time believing any of the claims. I would like to see a much more balanced compare/contrast of Rama to a more conventional approach, and particularly I would like to see that for a much more complex application than a Twitter clone, which is probably just about the best possible case for demonstrating this architecture.
  • bunderbunder 3y ago
    Indeed. The single most expensive human error I ever made in my life, I made in a system that was architected in exactly this way that the author claims eliminates the possibility of such errors. Yes, it is true that it was very easy to fix the problem and restore the system to working order. But, by the time we detected and resolved the defect, the system had already been operating out of spec for some time and the damage had already been done. This article seems to be ignorant of its own context. If ensuring that the database is internally consistent is really all it takes, then we all would have stopped worrying about it back in 1983 when Reuter and Harder originally formalized ACID.
  • Affric 3y ago
    Everything about databases is terrible and there is no problem any of their restrictions solve. Everything about Rama’s model is revolutionary, dramatic, very this, extremely that. The hyperbolic nature of every claim combined with the very thin evidence makes it hard to take any of it seriously. It’s like speaking to a proselytiser for a cult.
• avereveard 3y ago
  Eh materializing data upon mutation can bring you some gains if your product does like one thing and needs to do it very fast. But as soon as you get complex transactions with things that need to be updated in a atomic write or you want to add a new feature that needs data organized in a different way then you're in trouble. Also deeply unsatisfied of "just slap an index on it" that was lightly trow around on the part about building an application. The index is a global state, it was just moved one step further down the layer.
  • ColonelPhantom 3y ago
    > it was just moved one step further down the layer And therefore, crucially, you don't need to manage it yourself anymore. The only thing you need to do is tell the system what to index (which is code, not state/data).
    • avereveard 3y ago
      But I don't manage databases either, management is a poor differentiation
      • ColonelPhantom 3y ago
        You do manage the state of your database. When you do an UPDATE, you explicitly change the state. The statefulness of an index, however, is merely an implementation detail. A database without indices would be in the same state, and behave the same (except much slower). This means that the state is not important for the semantics of the datastore, unlike with a regular database.
        • avereveard 3y ago
          when I do an update I don't do a global state transition tho.
• ecshafer 3y ago
  I don't see how you can claim this is proved by a "twitter scale mastodon client" unless you are actually running a 40m daily user website. Simulating a real environment, and the accompanying code and infra changes, real users, network usage, etc is impossible.
  • mnming 3y ago
    They did perform a Twitter scale simulation, check out: https://blog.redplanetlabs.com/2023/08/15/how-we-reduced-the-cost-of-building-twitter-at-twitter-scale-by-100x/ https://blog.redplanetlabs.com/2023/08/15/how-we-reduced-the...
• kgeist 3y ago
  >The solution is to treat these two concepts separately. One subsystem should be used for representing the source of truth, and another should be used for materializing any number of indexed stores off of that source of truth. Once again, this is event sourcing plus materialized views. At work we decouple the read model from the write model: the write model ("source of truth") consists of traditional relational domain models with invariants/costraints and all (which, I think, is not difficult to reason about for most devs who are already used to ORM's), and almost every command also produces an event which is published to the shared domain event queue(s). The read model(s) are constructed by workers consuming events and building views however they fit (and they can be rebuilt, too). For example, we have a service which manages users ("source of truth" service), and another service is just a view service (to show a complex UI) which builds its own read model/index based on the events of the user service (and other services). Without it, we'd have tons of joins or slow cross-service API calls. Technically we can replay events (in fact, we accidentally once did it due to a bug in our platform code when we started replaying ALL events for the last 3 years) but I don't think we ever really needed it. Sometimes we need to rebuild views due to bugs, but we usually do it programmatically in an ad hoc manner (special scripts, or a SQL migration). I don't know how our architecture is properly called (I never heard anyone call it "event sourcing"). It's just good old MySQL + RabbitMQ and a bit of glue on top (although not super-trivial to do properly I admit: things like transactional outboxes, at least once delivery guarantee, eventual consistency, maintaining correct event processing order, event data batching, DB management, what to do if an event handler crashes? etc.) So I wonder, what we're missing without Rama with this setup, what problems it solves and how (from the list above) provided we already have our battle-tested setup and it's language-agnostic (we have producers/consumers both in PHP and Go) while Rama seems to be more geared towards Java.
  • lifeisstillgood 3y ago
    So does the “command” (say update customer address) perform the SQL and then some RDBMS trigger send the event onto rabbitMQ or is it an ORM that sends SQL and posts to rabbitMQ? Plus where do you store events, in what format? Tell me more please :-) What you are missing is a cool name for the whole ecosystem
• cryptonector 3y ago
  Event sourcing (+ materialized views and indices) != abandon your RDBMS. You can have both. Though you might find that traditional RDBMSes don't optimize well enough in the event sourcing (+ materialized views and indices) model.
• dcow 3y ago
  I get weird looks when I tell people we ran for 3.5 years on an s3api in front of bucket storage. It scaled to meet our needs and was especially appropriate for our app’s storage profile. And now that the startup doesn’t exist I’m glad that I never wasted time messing with “real” DBs. There’s definitely an industry bias toward using DBs.
• bob1029 3y ago
  > No single data model can support all use cases. In theory, there is no domain (or finite set of domains) that cannot be accurately modeled using tuples of things and their relations. Practically speaking, the scope of a given database/schema is generally restricted to one business or problem area, but even this doesn't matter as long as the types aren't aliasing inappropriately. You could put a web retailer and an insurance company in the same schema and it would totally work if you are careful with naming things. Putting everything into exactly one database is a superpower. The #1 reason I push for this is to avoid the need to conduct distributed transactions across multiple datastores. If all business happens in one transactional system, your semantics are dramatically simplified.
  • baq 3y ago
    This this this. > Putting everything into exactly one database is a superpower. Especially this. $1M big iron DB server is much cheaper than a redundant array of inexpensive databases when people come to actually use the data; be it developers, analysts or leadership, everybody saves time, except perhaps a few DBAs.
    • riku_iki 3y ago
      > $1M big iron DB server is much cheaper than a redundant array of inexpensive databases but you need some redundancy to avoid service downtime if that server/rack/datacenter will have a failure.
    • bunderbunder 3y ago
      It can be, but I've discovered that in tech it's common to assume that your employees' time costs $0, for some reason. I couldn't even begin to count the number of times I've had to walk a manager back from deciding that we should spend a person-month of engineering time to develop an in-house solution that lets us avoid a $500/year licensing fee. I see this a lot in RDBMSes. I've used plenty of 'em, and I have nothing against projects like PostgreSQL; they're amazing, especially for the price. But computers cost money, and maintenance time costs money, and development time costs money, and I know from experience that I can squeeze a lot more performance out of the same hardware using MSSQL than I can PostgreSQL, and I'll have an easier time doing it, too. I can even point to some of the specific design and feature differences that lead to the discrepancy. It's to the point where, whenever I see an article talking about the limitations of relational databases, I automatically mentally insert a "that you can get for free" into the sentence.
• shay_ker 3y ago
  What's an ELI5 of Rama? I found the docs confusing as well: https://redplanetlabs.com/docs/~/index.html https://redplanetlabs.com/docs/~/index.html Please no buzzwords like "paradigm shift" or "platform". If diagrams are necessary, I'd love to read a post that explains clearer.
  • pi-e-sigma 3y ago
    Seems like another attempt at No-SQL. "But this time it's different!"
    • fulafel 3y ago
      That's how things typically take off, not on the first attempt. Depends on what's different this time. (Though NoSQL has outlived its usefulness as a concept IMO, it's just too loose to be useful beyond its early use for "something like CouchDB/Mongo", which this is clearly not)
    • nineteen999 3y ago
      Negative SQL?
      • deleted 3y ago
        [deleted]
    • archarios 3y ago
      This is completely different from no-sql. Much more than just a different database..
      • pi-e-sigma 3y ago
        How exactly is it different from No-SQL? No schema? Check. No consistency? Check. (eventually consistent) Key-value store? Check. (because it's using ZooKeeper under the hood) Promising amazing results and freeing you from the chains of the SQL? CHECK!
• davedx 3y ago
  > The better approach, as we’ll get to later in this post, is event sourcing plus materialized views. Right, so the solution is more complexity? Of course it is. Sigh
  • crowcroft 3y ago
    Complexity can't be destroyed, only moved around. For a great number of tasks this solution offers a bad set of trade offs. But, at a certain level of scale everything is a data engineering problem, and sometimes this is the (relatively) simple solution when viewed in the context of the entire system. 'Just use mySQL/SQLite/Postgres' is great advice until it isn't.
    • aranchelk 3y ago
      Every problem has an amount of essential complexity, which as you said can’t be done away with. Imperfect engineering often adds unnecessary (accidental) complexity. I think it would a mistake to approach a domain and assume nothing can be made simpler or more straightforward.
      • crowcroft 3y ago
        I agree 100%
  • saberience 3y ago
    In all the companies I've worked at "event sourcing plus materialized views." has only ever caused intense confusion and resulted in more bugs and more downtime. The simpler solutions of either using MySQL or PostGres OR Redis/DynamoDB, all have worked better. I genuinely believe we would all be better off if Martin Fowler's original article about Event Sourcing had never been written. IMO, it's a bad idea in 99% of cases.
    • tetha 3y ago
      I've worked with people who did work with successfully employed event sources architectures. One thing there is: These were massive engineering departments in very large companies. Think of several dozen data processing teams, each very much their own small company, all working on different domains with event and stream consumers and utilizing the possibility of restarting their stream consumers some time ago. Impressive Kafka-Clusters ingesting thousands and thousands of events per second and keeping quite the backlog around to enable this refeeding, outages and catching back up and such. At their scale and complexity, I can see benefits. However, at that scale, Kafka is your database. And you end up with a different color of the same maintenance work you'd have with your central database. Data ends up not being written, data ends up being written incorrectly, incorrectly written data now ends up with transitive errors. At times, those guys ended up having to drop a significant timespan of data, filter our the true input events, pull in the true events and start refeeding from there - except then there was a thundering herd effect, and then they had to start slowly bringing up load... great fun for the management teams of the persistence. Note that I'm not necessarily saying e.g. Postgres is the solution to everything. However, a decently tuned and sized, competently run Postgres cluster allows you to delay larger architectural decisions for a long time with run-of-the-mill libraries and setup. Forever in a nonzero amount of projects.
• intrasight 3y ago
  "Everything wrong blogs that do not support reader view" Why does anyone create a blog like that?
• morsecodist 3y ago
  This is more an advertisement for a type of database than a statement that they are unnecessary. From what I can tell in the article it seems their differentiator is Event Sourcing and having arbitrary complex index builders on top of the events. It seems similar to EventStoreDB[1]. I have always been interested by the concept of an event sourcing database with projections and I want to build one eventually so it is interesting to see how they have approached the problem. Also they mention on their site: > Rama is programmed entirely with a Java API – no custom languages or DSLs. It makes sense why they have gone this route if they want a "Turing-complete dataflow API" but this can be a major barrier to adoption. This is a big challenge with implementing these databases in my opinion because you want to allow for any logic to build out your indexes/projections/views but then you are stuck between a new complicated DSL or using a particular language. 1: https://developers.eventstore.com/server/v23.10/#getting-started https://developers.eventstore.com/server/v23.10/#getting-sta...
  • cryptonector 3y ago
    The API could be LinkQ-like, so it'd still be SQL-ish, just in a query builder sort of way.
  • vintermann 3y ago
    Well, it is an advertisement, but much of what it says is undoubtedly true. It's true that databases are effectively global mutable state in practice, with all the headaches that gives, and that event sourcing and materialized views is a damn appealing solution to it. However, the jump from that to saying you have it is a bit like the jump from talking about the messiah and saying that he's sitting in the next room just now. We investigated event sourcing and materialized views as the backbone of a (new) big business application five years ago. The problem was that there were so few products back then and they were so untested (unless my mind is playing tricks on me, we did look at EventStoreDB, as well as Kafka streams), and despite being open source they were also usually heavily tied to one commercial provider. No one had anything like experience with it. I couldn't convince my colleagues to go for this, much less my bosses. Hell, I could barely convince myself. Philosophically, I believe it's sound, but how much has changed in those years?
  • 3y ago
• cmrdporcupine 3y ago
  Data models are restrictive That's kind of the point. Model your data. Think about it. Don't (mis)treat your database as a "persistence layer" -- it's not. It's a knowledge base. The "restriction" in the relational model is making you think about knowledge, facts, data, and then structure them in a way that is then more universal and less restrictive for the future. Relations are very expressive and done right is far more flexible than the others named there. That was Codd's entire point: https://www.seas.upenn.edu/~zives/03f/cis550/codd.pdf https://www.seas.upenn.edu/~zives/03f/cis550/codd.pdf "Future users of large data banks must be protected from having to know how the data is organized in the machine (the internal representation) ..." and then goes on to explain how the predicate-logic based relational data model is a more universal and flexible model that protects users/developers from the static impositions of tree-structured/network structure models. All the other stuff in this article is getting stuck in the technical minutiae of how SQL RDBMSs are implemented (author seems obsessed with indexes). But that's somewhat beside the point. A purely relational database that jettisons SQL doesn't have to have the limitations the author is poking at. It's so frustrating we're still going over this stuff decades later. This was a painful read. People developing databases should already be schooled in this stuff.
  • vidarh 3y ago
    In their chosen example, and many others that involves external sources, it's not really "your" data, though. You have no control over the structure, and you then have the choice of deciding whether you want to throw away what you can't shoehorn into your current schema, or store it and transform it as best you can and be prepared to re-transform it as your system evolves. Really, they're not avoiding modelling the data. They're modelling it in their "P-stores" whether they want to admit that's just another data model or not. It's clearly not data models they object to, but typical database DDL coupled with a desired to not throw away data that doesn't conform right now. Depending on your application domain, not throwing away data that doesn't conform to your current data model can be good or bad, but it's not an accident that they picked Twitter/Mastodon where it's easy to justify not rejecting non-conforming documents. I agree with you that this doesn't require ditching relational models, though. For that matter not even SQL. You can even fairly easily build this kind of architecture without e.g. ever leaving Postgres (replicated pair for their "depots"; triggers + postgres foreign data wrappers to materialize views on index servers and/or to do requests across index servers).
• brianmcc 3y ago
  We do go in circles/cycles quite a lot as an industry. I wonder if the trend is back towards SQL, right now, too many teams been burned by Event Sourcing when they just needed a decent SQL DB? Just idle conjecture....
• kevsim 3y ago
  Is this Rama solution similar to the kind of thing you can get with Kafka with KTables? If so I'd be curious how they've solved making it in anyway operational less complex to manage then a database. It's been a few years since I've run Kafka but it used to kind of be a beast.
• ram_rar 3y ago
  Even after reading this doc [1], I am not clear on who is the target audience and what are you trying to solve? It would be helpful to take a real world example and translate how easy /efficient it would be to do this via RAMA. [1] https://redplanetlabs.com/docs/~/why-use-rama.html#gsc.tab=0 https://redplanetlabs.com/docs/~/why-use-rama.html#gsc.tab=0
  • nathanmarz 3y ago
    We've released many examples. The first is our Twitter-scale Mastodon implementation, which is 100x less code than Twitter wrote to build the equivalent at scale (just the consumer product). It's also more than 40% less code than Mastodon's official implementation (which isn't scalable). https://github.com/redplanetlabs/twitter-scale-mastodon https://github.com/redplanetlabs/twitter-scale-mastodon The rama-demo-gallery repo also contains many short, self-contained, thoroughly commented examples of applying Rama towards very different use cases. These include user profile management, time-series analytics, atomic and fault-tolerant bank transfers, and more. https://github.com/redplanetlabs/rama-demo-gallery https://github.com/redplanetlabs/rama-demo-gallery
    • deleted 3y ago
      [deleted]
    • CyberDildonics 3y ago
      How do you know if it's "twitter-scale" if it's a mastodon implementation?
      • nathanmarz 3y ago
        Mastodon is extremely similar to the consumer Twitter product, and "Twitter-scale" is a known number that we tested well beyond. Among other things, we verified our timeline delivery latency was at least as good as Twitter's number ad that it scaled linearly.
  • deleted 3y ago
    [deleted]
• 0xbadcafebee 3y ago
  strapping in for the clickbait blog post... "Global mutable state is harmful" - well... yes, that's totally correct. "The better approach [..] is event sourcing plus materialized views." .....errr... that's one approach. we probably shouldn't hitch all our ponies to one post. "Data models are restrictive" - well, yes, but that's not necessarily a bad thing, it's just "a thing". "If you can specify your indexes in terms of the simpler primitive of data structures, then your datastore can express any data model. Additionally, it can express infinite more by composing data structures in different ways" - perhaps the reader can see where this is a bad idea? by allowing infinite data structures, we now have infinite complexity. great. so rather than 4 restrictive data models, we'll have 10,000. "There’s a fundamental tension between being a source of truth versus being an indexed store that answers queries quickly. The traditional RDBMS architecture conflates these two concepts into the same datastore." - well, the problem with looking at it this way is, there is no truth. if you give any system enough time to operate, grow and change, eventually the information that was "the truth" eventually receives information back from something that was "indexing" the truth. "truth" is relative. "The solution is to treat these two concepts separately. One subsystem should be used for representing the source of truth, and another should be used for materializing any number of indexed stores off of that source of truth." this will fail eventually when your source of truth isn't as truthy as you'd like it to be. "The restrictiveness of database schemas forces you to twist your application to fit the database in undesirable ways." - it's a tool. it's not going to do everything you want, exactly as you want. the tradeoff is that it does one thing really specifically and well. "The a la carte model exists because the software industry has operated without a cohesive model for constructing end-to-end application backends." - but right there you're conceding that there has to be a "backend" and "frontend" to software design. your models are restrictive because your paradigms are. "When you use tooling that is built under a truly cohesive model, the complexities of the a la carte model melt away, the opportunity for abstraction, automation, and reuse skyrockets, and the cost of software development drastically decreases." - but actually it's the opposite: a "cohesive model" just means "really opinionated". a-la-carte is actually a significant improvement over cohesion when it is simple and loosely-coupled. there will always be necessary complexity, but it can be managed easier when individual components maintain their own cohesion, and outside of those components, maintain an extremely simple, easy interface. that is what makes for more composable systems that are easier to think about, not cohesion between all of the components! "A cohesive model for building application backends" - some really good thoughts in the article, but ultimately "cohesion" between system components is not going to win out over individual components that maintain their cohesion and join via loosely-coupled interfaces. if you don't believe me, look at the whole Internet.
• the_duke 3y ago
  Every time I tried to use event sourcing I have regretted it, outside of some narrow and focused use cases. In theory ES is brilliant and offers a lot of great functionality like replaying history to find bugs, going back to any arbitrary point in history, being able to restore just from the event log, diverse and use case tailored projections, scalability, ... In practice it increases the complexity to the point were it's a pointless chore. Problems: * the need for events, aggregates and projections increases the boilerplate tremendously. You end up with lots of types and related code representing the same thing. Adding a single field can lead to a 200+ LOC diff * a simple thing like having a unique index becomes a complex architectural decision and problem ... do you have an in-memory aggregate? That doesn't scale. Do you use a projection with an external database? well, how do you keep that change ACID? etc * you need to keep support for old event versions forever, and either need code to cast older event versions into newer ones, or have a event log rewrite flow that removes old events before you can remove them from code * if you have bugs in you can end up needing fixup events / event types that only exist to clean up , and as above, you have to keep that around for a long time * similarly, bugs in projection code can mess up the target databases and require cumbersome cleanup / rebuilding the whole projection * regulation like GDPR requires deleting user data, but often you can't / don't want to just delete everything, so you need an anonimizing rewrite flow. it can also become quite hard to figure out where the data actually is * the majority of use cases will make little to no use of the actual benefits A lot of the above could be fixed with proper tooling. A powerful ES database that handles event schemas, schema migrations, projections, indexes, etc, maybe with a declarative system that also allows providing custom code where necessary. I'll take a look at Rama I guess.
• deleted 3y ago
  [deleted]
• LispSporks22 3y ago
  What’s micro batch streaming?
  • nathanmarz 3y ago
    It's a way of doing stream processing that sacrifices a little bit of update latency for higher throughput and exactly-once processing semantics. Regular streaming has single-digit milli update latency, while microbatching has at least a few hundred millis update latency. By "exactly-once processing semantics", I mean that regardless of how many failures are on the cluster (e.g. nodes losing power, network partitions), the updates into all partitions of all indexes will be as if there were no failures at all. Pure streaming, on the other hand, has either "at-least once" or "at-most once" semantics.
• skywhopper 3y ago
  This is marketing spiel masquerading as a bad take. Rama may or may not be cool tech, but the idea that they are anywhere close to being able to get rid of structured database systems for complex systems is absolutely laughable to the point that it makes me uninterested in learning more about the tech. Please tone down the hyperbole if you want serious attention.
• jakozaur 3y ago
  My biggest problem with databases is that they are very hard to evolve. They accumulate a history of decisions and are in a suboptimal state. Legacy is widespread in enterprises. Oracle is still milking $ 50B+ annually, and the databases are the primary driver of why you need Oracle and why they can upsell you other products after a compliance audit. The schema changes are hard (e.g. try to normalize/denormalize data), production is the only environment when things go wrong, in-place changes with untested revert options are default, etc.
  • emn13 3y ago
    Clearly, there's some truth to this perspective. On the other hand, which alternative actually makes this easier? If your schema is non-enforced, and/or your data (and its schema) are in loosely coupled distributed silos, and/or as in the articles case indexes are essentially little programs... Then broad scale schema evolution goes from being difficult to often being effectively entirely infeasible. It's possible that in turn forces users to adapt and split their changes into smaller chunks, but that's pretty speculative and they might simply fail entirely in the attempt. Which data stores are more easily evolved than a relational DB with a fairly restrictive schema?
    • jakozaur 3y ago
      I agree with you that schema has tremendous value. Let's store JSON everywhere and infer it on read, backfires in a big way. Even MongoDB added tooling to enforce a schema. I just wish there was a better tooling to evolve schema. Blue/green deployments, schema as code (no manual up and down procedures), good backward compatibility story, shadow testing in production, etc. The database still feels like a monolith and did not get the benefits of the microservice revolution.
• manicennui 3y ago
  A lot of these same problems were solved in a similar way with Datomic and xtdb. https://www.datomic.com/ https://www.datomic.com/ https://xtdb.com/ https://xtdb.com/
• marcosdumay 3y ago
  Well, I guess it's official. The belief on software architecture dogma is so strong that we can consider it a church. I imagine that will bring great tax benefits to programing schools.
• big_whack 3y ago
  A lot of the commenters seem like database fans instinctively jumping to defend databases. The post is talking about contexts where you are dealing with petabytes of data. Building processing systems for petabytes has a separate set of problems from what most people have experienced. Having a single Postgres for your startup is probably fine, that's not the point here. There is no option to just "put it all in a database". You need to compose a number of different systems. You use your individual databases as indexes, not as primary storage, and the primary storage is probably S3. The post is interesting and the author has been working on this stuff for a while. He wrote Apache Storm and used to promote some of these concepts as the "Lambda architecture" though I haven't seen that term in a while.
  • nathanmarz 3y ago
    I think the complexity gets really crazy at high scale, but the complexity caused by databases is still significant at low scale as well. For example, needing to use an ORM and dealing with all the ways that can leak is pure complexity caused by not being able to index your domain model directly.
    • chasd00 3y ago
      I've never known anyone to _need_ to use an ORM, it's always been out of convenience.
    • big_whack 3y ago
      I feel like there are a couple different points - * The immutability, lambda architecture points I agree with. I think the separation of the immutable log from the views is important. Databases are frequently used in ways that go against these principles. * I am not sold that being unable to express the domain model correctly is really a fair criticism of databases. Most businesses in my experience have a domain that is modeled pretty well in a relational DB. I haven't seen a better general solution yet, though I haven't checked out Rama. At the low end of the scale, there are a lot of companies (or projects) for which the entire dataset fits in a single managed Postgres instance, without any DBA or scalability needs. They still suffer from complexity due to mutable state, but the architectural separation of source of truth vs "views" can be implemented inside the one database, using an append only table and materialized views. There are some kinds of data that are poorly modeled this way (e.g images) but many kinds that work well. So I don't really view the architectural ideas as repudiating databases in general, more as repudiating a mutable approach to data management.
• continuational 3y ago
  Does it have ACID transactions? Does the indexes have read after write guarantees?
  • nathanmarz 3y ago
    Yes to both. Our bank transfer example in rama-demo-gallery is a good example of this. https://github.com/redplanetlabs/rama-demo-gallery https://github.com/redplanetlabs/rama-demo-gallery
    • ARandomerDude 3y ago
      > (:use [com.rpl.rama] Who hurt you?
      • beders 3y ago
        it's fine for example purposes IMHO.
    • continuational 3y ago
      That's very good to hear! I can't say I understand the example. It seems to talk a lot about implementation details compared to the naive SQL: create table account ( account_id bigint primary key, balance bigint not null default 0, check (balance >= 0) ); begin; update account set balance = balance - $amount where account_id = $from_id; update account set balance = balance + $amount where account_id = $to_id; commit;
      • nathanmarz 3y ago
        The rama-demo-gallery example is distributed, and it's also tracking incoming/outgoing transfers for each user.
• kaba0 3y ago
  > However, storing data normalized can increase the work to perform queries by requiring more joins. Oftentimes, that extra work is so much you’re forced to denormalize the database to improve performance. Databases have materialized views though, that solves this problem.
  • Kinrany 3y ago
    Materialized views are not a full solution without updates. Some databases can update incrementally, but not the popular ones.
• pgaddict 3y ago
  Did I miss something, or does that post completely omit concepts like concurrency, isolation, constraints and such? And are they really suggesting "query topologies" (which seem very non-declarative and essentially making query planning/optimization responsibility of the person writing them) are a superior developer environment?
  • nathanmarz 3y ago
    This stuff is all covered thoroughly in our docs. This is a post about the complexity aspect of backend development and how those are addressed with Rama. It's not a thorough explanation of every aspect of Rama, since that would be extremely long. If you dig into Rama, you'll see that it's properties and guarantees are very strong. And yes, Rama's queries are a massively superior approach. The need for complex query planners is a result of limitations in how you index data, oftentimes from the tension between normalization and denormalization. With Rama, it's easy to robustly materialize multiple views that are already in the shape needed for queries. Check out our tutorial for a gentle introduction to Rama's concepts https://redplanetlabs.com/docs/~/tutorial1.html https://redplanetlabs.com/docs/~/tutorial1.html
    • pgaddict 3y ago
      I did check the docs. According to the search, there are like 5 references to "consistency", 4 of which are talking about how traditional databases do that poorly and the 5th one seems to suggest to use "depot partitioner" which seems very much like sharding with per-shard consistency. For "isolation" there are t2 references, for "transaction" 1, none of which explains anything. And I'm sorry, you won't convince me something defined in Java is superior to declarative SQL. There's a lot of problems with SQL, no doubt about it.
• russdpale 3y ago
  Seems like a bunch of buzzwords and such. I've been working with databases for years for one of the largest companies in the world and no one has ever said "topology" before. Any time I would save with this is wasted on learning java and this framework. There isn't anything wrong with databases.
  • nathanmarz 3y ago
    Our production-ready, Twitter-scale Mastodon implementation in 100x less code than Twitter wrote to build the equivalent feature-set (just the consumer product) begs to differ that it's "a bunch of buzzwords". https://github.com/redplanetlabs/twitter-scale-mastodon https://github.com/redplanetlabs/twitter-scale-mastodon
    • bawolff 3y ago
      Is it actually powering a site the scale of twitter? If no, then there is no reason to think its twitter scale.
      • lpapez 3y ago
        Exactly this. Filling some kind of a database with terabytes of data is not the difficult bit, serving it is... To claim that something static is "Twitter-scale" is just misleading IMO.
        • nchammas 3y ago
          They _did_ serve the data. From TFA: > To demonstrate its scale, we operated the instance with 100M bots posting 3,500 times per second at 403 average fanout. The linked post about their demo [1] has more details about the challenges involved, including rendering home timelines. [1]: https://blog.redplanetlabs.com/2023/08/15/how-we-reduced-the-cost-of-building-twitter-at-twitter-scale-by-100x/ https://blog.redplanetlabs.com/2023/08/15/how-we-reduced-the...
          • 3y ago
• specialist 3y ago
  https://blog.redplanetlabs.com/2024/01/09/everything-wrong-with-databases-and-why-their-complexity-is-now-unnecessary/#Restrictive_schemas https://blog.redplanetlabs.com/2024/01/09/everything-wrong-w... > It’s common to instead use adapter libraries that map a domain representation to a database representation, such as ORMs. However, such an abstraction frequently leaks and causes issues. ... FWIW, I'm creating a tool (strategy) that is neither an ORM or an abstraction layer (eg JOOQ) or template-based (eg myBatis). Just type safe adapters for normal SQL statements. Will be announcing an alpha release "Any Week Now". If anyone has an idea for how to monetize yet another database client library, I'm all ears. I just need to eat, pay rent, and buy dog kibble.
  • drbawb 3y ago
    >If anyone has an idea for how to monetize yet another database client library, I'm all ears. I just need to eat, pay rent, and buy dog kibble FWIW jOOQ's model worked great from a consumer/end-user point of view. I got to learn the whole library with no real strings attached, and then pay to adapt it/support it on Big Fucking Databases(TM) as projects required it. Their site and documentation is also quite pleasant. In a way it feels like double-piggybacking: Microsoft, Oracle, et al. spend a lot on mindshare to make sure technical leads choose those respective stacks. Then the army of developers/consultants, required to get the promised ROI out of those stacks, inevitably have to go looking for tools. Be there to sell them the shovels. It helps if they are already familiar with it. (Free OSS adapters / "community" editions, etc., or even just very generous "evaluation" versions with no real effort to police use in a lab environment.) Not sure how successful jOOQ has been financially, but considering they've been around for many years at this point, I have to imagine it's worked out well enough to pay for the lights and kibble? The only other "dev tool" company I've enjoyed working with _more_ has been JetBrains. (The recent AI plugin not-withstanding.)
    • specialist 3y ago
      Thanks. I'll chew on your feedback. The domain knowledge embodied in JOOQ is kind of daunting. Truly huge. Of the current SQL client API tools, mine is most like sqlc. Mine's major improvement, IMHO of course; but concept and workflow are comparable. What do you think of paying to use such a tool for builds? eg Deployed to a CI/CD pipeline. Free to use for sql-fiddle, personal, desktop, FOSS, etc. But once a project starts to use a tool for "real work", that's proof of value. Right? Part of my just wants to want to do shareware. I published shareware in the early 1990s. Since I have the biz acumen of a sea cucumber, I just did the work and let people decide. It was great gig while it lasted. I guess today's equiv would be some kind of patreon. My reluctance there is the PR/promotion/social media part. I have trouble even imagining what that'd look like.
• jmull 3y ago
  Pretty interesting once you read past the marketing push. I mostly like the approach, but there are a lot of questions/issues that spring to mind (not that some of them don't already have answers, but I didn't read everything). I'll list some of them: * I'm pretty sure restrictive schemas are a feature not a bug, but I suppose you can add your own in your ETL "microbatch streaming" implementation (if I'm reading this right, this is where you transform the events/data that have been recorded to the indexed form your app wants to query). So you could, e.g., filter out any data with invalid schema, and/or record and error about the invalid data, etc. A pain, though, for it to be a separate thing to implement. * I'm not that excited to have my data source and objects/entities be Java. * The Rama business model and sustainability story seem like big question marks that would have to have strong, long-lasting answers/guarantees before anyone should invest too much in this. This is pretty different and sits at a fundamental level of abstraction. If you built on this for years (or decades) and then something happened you could be in serious trouble. * Hosting/deployment/resources-needed is unclear (to me, anyway) * Quibble on "Data models are restrictive": common databases are pretty flexible these days, supporting different models well. * I'm thinking a lot of apps won't get too much value from keeping their events around forever, so that becomes a kind of anchor around the neck, a cost that apps using Rama have to pay whether they really want it or not. I have questions about how that scales over time. E.g., say my has depot has 20B events and I want to add an index to a p-state or a new value to an enum... do I need to ETL 20 billion events to do routine changes/additions? And obviously schema changes get a lot more complicated than that. I get that you could have granular pstates but then I start worrying about the distributed nature of this. I guess you would generally do migrations by creating new pstates with the new structure, take as much time as you need to populate them, then cut over as gradually as you need, and then retire the old pstates on whatever timeline you want.... But that's a lot of work you want to avoid doing routinely, I'd think. I'm starting to think of more things, but I better stop (my build finished long ago!)
  • nathanmarz 3y ago
    * By "restrictive schemas" I mean being forced to represent your data storage in non-optimal ways – like not being able to have nested objects in a first-class way. Schemas themselves are extremely important, and they should be as tight as possible. * Rama's JVM-based, so the entire ecosystem is available to you. You can represent data as primitive types, Java objects, Protobuf, Clojure records, etc. * You deploy and manage your own Rama clusters. The number of nodes / instance types depends on the app, but Rama doesn't use more resources than traditional architectures combining multiple tools. * Some databases support multiple very specific data models (e.g. Redis). I don't consider that flexible compared to Rama, which allows for arbitrary combinations of arbitrarily sized data structures of arbitrary partitioning. * Depots (the "event sourcing" part of Rama) can be optionally trimmed. So you can configure it to only keep the last 2M entries per partition, for example. Some applications need this, while others don't. * If you're adding a new PState, it's up to you how far back in the depot to start. For example, you could say "start from events appended after a specific timestamp" or "start from 10M records ago on each partition". * We have a first-class PState migrations feature coming very soon. These migrations are lazy, so there's no downime. Basically you can specify a migration function at any level of your PStates, and the migration functions are run on read. In the background, it migrates iterates over the PState to migrate every value on disk (throttled so as not to use too many resources).
• nojvek 3y ago
  Read the tutorial https://redplanetlabs.com/docs/~/tutorial1.html#gsc.tab=0 https://redplanetlabs.com/docs/~/tutorial1.html#gsc.tab=0 This is quite complex compared to setting up Postgres or mysql and sending some sql over some port. I’m not sure I get what they are selling.
• benlivengood 3y ago
  I didn't notice a mention of transactions in the article, nor of constraints. It's all fine to claim that you can compose arbitrary event source domains together and query them but IMHO the biggest power of RDBMS are transactions and constraints for data integrity. Maybe Rama comes with amazing composability features that ensure cross-domain constraints, but I would be really surprised if they can maintain globally consistent real-time transactions. I've worked on huge ETL pipelines with materialized views (Photon, Ubiq, Mesa) and the business logic in Ubiq to materialize the view updates for Mesa was immense. None of it was transactional; everything was for aggregate statistics and so it worked well. Ads-DB and Payments used Spanner for very good reasons.
  • nathanmarz 3y ago
    Rama supports very strong transactions actually. Besides being able to do arbitrary atomic updates across multiple indexes on a single partition, you can also do distributed transactions. Our bank transfer example in rama-demo-gallery demonstrates this. https://github.com/redplanetlabs/rama-demo-gallery https://github.com/redplanetlabs/rama-demo-gallery
    • photon_lines 3y ago
      Sorry, but can you elaborate on what type of transaction isolation-levels your solution supports (i.e. read committed, repeatable read, serializable, read uncommitted)?
      • nathanmarz 3y ago
        Those concepts are relevant to databases like RDBMS's where you have multiple writes happening concurrently to the same index. Rama doesn't work like that – you get parallelism by having multiple partitions, writes on a single partition are strictly ordered, and Rama gets performance by batching the flushing of writes. For a foreign query in Rama, you only ever get back committed data. So that would be analogous to "read committed". For ETL code, an event has atomic access to all indexes on its task thread. Any data written to those indexes before the event would have "repeatable read" consistency. Any writes done by the event before it finishes have read-after-write consistency, which would be analogous to "read uncommitted". This is correct during ETL execution because no acknowledgement of completion has been given back to clients yet, and uncommitted writes are never visible to client queries.
• qaq 3y ago
  Most SQL RDBMS are a materialised view over a transaction log.
• fifticon 3y ago
  I work in a shop with about 6 years of event-sourcing experience (as in, our production has run on eventsourcing since 2017). My view is, that 'humans are not mature enough for eventsourcing'. For eventsourcing to work sanely, it must be used responsibly. Reality is, that people make mistakes, and eventsourcing HURTS whenever your developers don't act maturely on the common history you have built. For us, it has meant a bungee jump of "move ALL the things to eventsourcing", followed by a long slow painful 'move everything that doesn't NEED eventsourcing out of eventsourcing again, into relational database, and only keep the relevant eventsourcing parts in the actual eventsource db'. The main consequences for us have been 'consume a huge/expensive amount of resources' to do what we already did earlier, with vastly fewer resources, at the benefit of having some things easier to do, and a lot of other things suddenly complex. In particular, it was not a 'costless abstraction', instead it forced us to always consider the consequences for our eventsourcing.
  • JoeAltmaier 3y ago
    Been a joke since porting to new systems has existed: "Well, now it's bigger and slower, but at least it was more expensive!"
  • winrid 3y ago
    I know of teams that spent over a year trying to build a generic import system on streams, at multiple companies, and in both cases they should have just stood up a simple service on boxes with a lot of ram and auto scaling. Load CSV into memory. Done.
• register 3y ago
  Pure B...t. The title is deceiving and should be instead something along the lines of: How to architect an application at Mastodon scale without relying on databases. Also I would be very interested in seeing the actual technology rather than reading sensational claims about the unparalled level of scalability it supports. What does it provide in order to recover from failure and exceptions and to guarantee consistency of state? Relational databases are and will always be necessary as they provide a convenient model for querying, aggregating, joining and reporting data. Much of the value in a database lies in how it supports extracting value from business information rather what extreme scalability features it supports. Try to create a decent business report from events and then we can speak again.
  • vidarh 3y ago
    I mean, AcitivityPub is pretty much the ideal case for this architecture. ActivityPub/the underlying ActivityStream consists of activities that manipulates entities. The activities are in principle immutable, even if the entities they may create are not, and you can largely naively serialize them into a log, and partition them naively by actor or by server and actor or any number of schemes, to put them into the initial "depot". From there, materializing indexes is trivial, and replicating that is also fairly trivial. You "just" need a bunch of workers processing the depot logs into these "P-stores" If your requirements fits that, then recovery from failures "just" requires having replicas of the depots, which is "easy" as you just stream the logs elsewhere, and archive them to, say, a blob store, combined with the ability to reset the "cursors" of the workers populating the P-stores. It's an architecture that works very well when you want to index and query large sets of data where the order either doesn't matter (much) or you can order "well enough" from the data itself, so you can stream to multiple "depots" without worrying about assigning a global order. Such as ActivityPub. E.g. it doesn't really matter for Mastodon if a reply appears a second before the post its a reply to, because the key linking them together is there and you need to be able to gracefully handle the case where you never get the original or isn't even allowed to fetch it. So I don't doubt they can achieve this, and a platform that makes that easy would be nice. Their problem is that 1) very few people have data problems that are large enough that it isn't easier to "just" buy/rent a larger database for the depots, 2) or even that have a dataset large enough that they'd need a secondary set of servers to handle the materialization of views (or need the views to actually be materialized in the first place), 3) and even if they do, running workers to populate a secondary set isn't hard. It's also trivial to use tools your devs are already familiar with for the depots and "P-stores". E.g. Postgres works just fine for both until you have an architecture far larger than most people ever need to deal with, but there are many other options too. Once you start running into challenges with that, the obvious, well-known method is to stream into append-only logs, cut the logs regularly, run the indexers on the new log elements into indexed segments, and zipper merge the log elements and do streaming compression of the values (e.g. arithmetic coding is a common, trivial method) + a skip list. Since the point of these "P-stores" is to reduce the datasets to something where queries are low-complexity, you can generally assume you won't need to support much fancy re-ordering and grouping etc., mostly cheap filtering, coupled with streaming merges of query results from multiple P-store partitions. This is the "do it yourself" solution, which you can find "packaged up" in any number of search solutions, like Elasticsearch, Lucene, Sphinx etc., but it's also not all that hard to build from scratch for a specific use case. This method has been used for e.g. search engines "forever". First time any system I worked on actually needed it was ca 2006, and we first used Sphinx, then built a custom one (single developer effort; needed only because Sphinx at the time had only a fraction of the features it has today) Basically, they're building a distributed database that just has very opinionated ideas about the type of problems you should be solving and how. If your problem fits in that it's very possible they can provide something "turn key" that is more cohesive and better packaged up than picking and choosing your own components for the depot and indexes and writing your own indexing workers, but as you say, to most people the extreme scalability isn't the issue and a regular database is enough. Addressing the Twitter and Mastodon cases is kind of a warning here, because both Twitter and Mastodon (the software) started out with a tremendously naive architecture in this respect, so it's a low hanging fruit if you want to show off impressive-looking improvements. In terms of reporting, this architecture isn't a problem, because you'd not be working from events, but from the "P-stores" and nothing stops you from ensuring that data is trivially queryable with something reasonable. E.g. either using something like Postgres for the P-stores themselves, or just streaming the data into whatever you want to do reporting from. But again, it doesn't need to replace the database.
• winrid 3y ago
  I get what they're trying to do but I'm not sure this [0] syntax is the answer. [0] https://github.com/redplanetlabs/twitter-scale-mastodon/blob/master/backend/src/main/java/com/rpl/mastodon/modules/Notifications.java#L70 https://github.com/redplanetlabs/twitter-scale-mastodon/blob...
  • runeks 3y ago
    Agreed. Their landing page[1] says > Rama is programmed entirely with a Java API – no custom languages or DSLs. But this sure looks like an embedded DSL with stringly typed identifiers. [1] https://redplanetlabs.com/ https://redplanetlabs.com/
  • EzraM 3y ago
    The Clojure code reads better.
  • davedx 3y ago
    Yeah, this reminds me of the one time I had to try and read some Akka/Scala code. Good luck debugging that!
• kopos 3y ago
  The comments here are needlessly pessimistic and dismissive of a new data flow paradigm. In fact, this looks like the best NoSQL experience there is. SQL while is a standard now, had to prove itself many times over and also was a result of a massive push by few big tech backers. Rama still looks like it needs some starter examples - that is all. From what i could gather reading the documentation over few weeks... Rama is an engine supporting Stored Procedure over NoSQL systems. That point alone is worth a million bucks. I hope it lives up to the promise. Now back to my coding :D
  • nathanmarz 3y ago
    FYI, in case you're unaware the rama-demo-gallery repo has a bunch of short, self-contained, thoroughly commented examples of applying Rama towards different use cases https://github.com/redplanetlabs/rama-demo-gallery https://github.com/redplanetlabs/rama-demo-gallery
  • dathinab 3y ago
    I think people are more put of by the exact way/formulation the blog starts out with then the content it is about.
    • kopos 3y ago
      That seems fair. Instead of saying “this is what you’re doing wrong” better formulate “here’s how to super empower your skills”
• phartenfeller 3y ago
  I have been working as a database consultant for a few years. I am, of course, in my bubble, but there are a few things I really don't enjoy reading. > No single data model can support all use cases. This is a major reason why so many different databases exist with differing data models. So it’s common for companies to use multiple databases in order to handle their varying use cases. I hate that this is a common way of communicating this nowadays. Relational has been the mother of all data models for decades. In my opinion, you need a good reason to use something different. And this is also not an XOR. In the relational world, you can do K/V tables, store and query documents, and use graph functions for some DBs. And relational has so many safety tools to enforce data quality (e.g. ref. integrity, constraints, transactions, and unique keys). Data quality is always important in the long run. > Every programmer using relational databases eventually runs into the normalization versus denormalization problem. [...] Oftentimes, that extra work is so much you’re forced to denormalize the database to improve performance. I was never forced to denormalize something. Almost always, poor SQL queries are a problem. I guess this can be true for web hyperscalers, but these are exceptions.
  • automatic6131 3y ago
    The modern relational database is a near-miracle of human engineering. In the software field, it's probably only surpassed by the operating system, and maybe (or maybe not) the web browser.
    • collyw 3y ago
      Choose boring technology.
    • btilly 3y ago
      I would hold SQLite up against anything. Likely the second most widely deployed software ever. https://www.sqlite.org/mostdeployed.html https://www.sqlite.org/mostdeployed.html The best tested piece of software that I know of. https://www.sqlite.org/testing.html https://www.sqlite.org/testing.html People call it a toy database. Some toy.
• coldtea 3y ago
  "We announced Rama on August 15th with the tagline “the 100x development platform”." Alternative to database company peddling their ware with misinformed rant.
• MagicMoonlight 3y ago
  If you gut all the features like persistence and rolling back errors then you can definitely make things less complex. But then someone wants to access their email and it turns out the server restarted so it’s gone.